Method for diagnosing and segmenting positive lymph nodes in CT image based on multi-task learning

Abstract

The invention is suitable for the technical field of CT image processing, and provides a diagnosis and segmentation method for a positive lymph node in a CT image based on multi-task learning, and the method comprises the steps: carrying out the preprocessing of a CT image sequence, enhancing the data, constructing a multi-task deep learning model, extracting features through an encoder, and executing a classification task through a classification module. And segmenting the positive lymph node by using a segmentation module in combination with the classification result. According to the method, the classification accuracy (up to 91.1%) and the segmented DICE coefficient (up to 0.602) are both excellent, the method is superior to a single 2D segmentation model, and compared with a 3D segmentation model, the method keeps similar segmentation performance and is shorter in reasoning time. According to the method, classification and segmentation tasks for the positive lymph nodes in the CT image can be completed at the same time, the workload of doctors is relieved, the diagnosis and treatment efficiency of the doctors is improved, and the blank that positive lymph nodes are judged and sketched at the same time in the head and neck CT image of a nasopharyngeal carcinoma patient is filled.

Description

Technical Field

[0001] The present invention belongs to the technical field of CT image processing, and in particular to a diagnosis and segmentation method of positive lymph nodes in CT images based on multi-task learning. Background Art

[0002] As a common disease among head and neck malignant tumors, nasopharyngeal carcinoma has a high morbidity and mortality rate worldwide. In the imaging diagnosis of nasopharyngeal carcinoma patients, there are some specific characteristics, among which enlarged lymph nodes are the most important sign, usually with a diameter of more than 1 cm. In addition to size, the morphology of lymph nodes may also be abnormal, such as irregular shapes or blurred outlines. Furthermore, changes in the internal density of lymph nodes are also an important imaging diagnostic indicator, which is often related to pathological changes in the lymph nodes. Accurately judging and outlining positive lymph nodes in head and neck medical images is crucial for doctors' subsequent diagnosis and treatment.

[0003] In actual medical practice, computed tomography (CT) is the most commonly used imaging type in the diagnosis and treatment of nasopharyngeal carcinoma patients. Head and neck CT scans can help doctors detect and evaluate lesions such as lymphadenopathy, tumors, infections, and trauma, thereby improving the accuracy of diagnosis and treatment effectiveness. However, due to the variable shapes and sizes of head and neck lymph nodes, similar to the surrounding tissue structure, and unclear contour boundaries, the diagnosis of lymph nodes is very time-consuming and labor-intensive. Traditionally, the target area judgment and delineation of lymph nodes are performed manually by clinicians. For each head and neck CT sequence, the number of slices is usually around 150. Clinicians first need to determine whether there are positive lymph nodes in each CT slice. If so, they use professional software to delineate their contours. Studies have shown that the average segmentation time for a single head and neck cancer case is 2.7 hours. As the treatment process continues to advance, this process often needs to be repeated many times. In addition, the accuracy of segmentation is highly dependent on the knowledge, experience, and preferences of clinicians. There is often a certain deviation between different doctors. A standard judgment and delineation usually requires multiple doctors to complete together.

[0004] To solve this problem, deep learning has been applied in the target delineation and segmentation of positive lymph nodes in head and neck CT. Some studies have used 2D or 3D deep learning models to segment positive lymph nodes, such as 2D U-Net, 3DU-Net, etc. The basic ideas of these models are similar: multi-layer convolution operations are used in the encoder part to extract image features from local to global in sequence, and pooling operations are used to reduce the feature size at the same time. The features are then gradually restored to the original size through transposed convolution or interpolation in the decoder part, and the segmentation results are finally obtained. The difference is that 2D models usually use a single CT slice as input, while 3D models require multiple CT slices stacked as input. However, these models also have some problems. The 2D model ignores the hidden information between slices, resulting in low segmentation accuracy. Although the 3D model takes into account the information between slices, it introduces a large amount of 3D calculations, which requires higher computing resources and longer training and inference time.

[0005] In view of the above problems, the present invention proposes a diagnosis and segmentation method of positive lymph nodes in CT images based on multi-task learning. Summary of the invention

[0006] The purpose of the present invention is to provide a diagnosis and segmentation method of positive lymph nodes in CT images based on multi-task learning, aiming to solve the problems raised in the above background technology.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] The diagnosis and segmentation method of positive lymph nodes in CT images based on multi-task learning includes the following steps:

[0009] Step 1: Preprocessing: preprocess the acquired CT image sequence and add data enhancement operation;

[0010] Step 2: Build a multi-task deep learning model, which includes an encoder, a classification module, and a segmentation module;

[0011] Step 3, feature extraction: The encoder is used for feature extraction. The encoder uses a multi-layer convolutional neural network to extract local features of the image and gradually gather them together. At the same time, the pooling operation is used to reduce the feature size, and finally the required features are obtained;

[0012] Step 4: Classification task: The classification task is performed using a classification module. The classification module takes the features extracted in step 3 as input and obtains the classification result of each feature through calculation.

[0013] Output 0 means that the classification module believes that the CT slice corresponding to the feature has no positive lymph nodes, and output 1 means that the classification module believes that the CT slice corresponding to the feature has positive lymph nodes;

[0014] Step 5, segmentation task: The segmentation task is performed using the segmentation module. The segmentation module takes the features extracted in step 3 as input, obtains the segmentation result of each feature by restoring the feature size, and then removes useless segmentation results through the classification results, and finally obtains the segmentation result of the CT slice with only positive lymph nodes.

[0015] Furthermore, the preprocessing operation includes using an additional U-Net model to remove all background interference, and then clipping the intensity in the range of [0, 300HU] to retain the lymph node HU value, and normalizing the lymph node HU value to the range of [0, 1.0].

[0016] Furthermore, the data enhancement operation includes random rotation, random upside down flipping, and random left-right flipping.

[0017] Furthermore, the encoder is composed of 4 layers of stacked down modules, each of which includes two convolutions, batch normalization, and activation function operations, and finally completes the downsampling operation through a maximum pooling layer; after being processed by the encoder, each CT slice obtains a corresponding feature vector.

[0018] Furthermore, the classification module is composed of multiple 1×1 convolutional layers and fully connected layers, each 1×1 convolutional layer is used to reduce the dimension of the feature, and finally the feature is flattened and mapped to the output category through the fully connected layer to obtain the classification result.

[0019] Furthermore, the segmentation module consists of a fusion module and four up modules; the fusion module includes two SEblocks, 1×1 convolution, batch normalization, and activation function operations; SE block suppresses unimportant channels in the data and assigns low weights, and encourages important channels in the data and assigns high weights through the two operations of squeezing and excitation; each up module includes two convolutions, batch normalization, and activation function operations, and finally up-samples through transposed convolution; the input of each up module consists of two parts, one is the output of the previous layer of the network, and the other is the output of the corresponding down module for the i-th slice.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention performs well in classification accuracy (up to 91.1%) and segmentation DICE coefficient (up to 0.602), which is better than a single 2D segmentation model. Compared with a 3D segmentation model, the present invention has a shorter reasoning time while maintaining similar segmentation performance. In addition, the method proposed by the present invention can simultaneously complete the classification and segmentation tasks for positive lymph nodes in head and neck CT images, reducing the workload of doctors, improving the efficiency of doctors' diagnosis and treatment, and filling the gap in the simultaneous judgment and delineation of positive lymph nodes in head and neck CT images of nasopharyngeal carcinoma patients, providing strong support for clinical diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Flow chart of the method of the present invention.

[0023] Figure 2 It is a framework flow chart of the method of the present invention.

[0024] Figure 3 This is a schematic diagram of the structure of the down module.

[0025] Figure 4 This is a schematic diagram of the structure of the up module.

[0026] Figure 5 This is a schematic diagram of the structure of the fusion module. DETAILED DESCRIPTION

[0027] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below, but it should not be construed as limiting the applicable scope of the present invention.

[0028] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0029] An embodiment of the present invention provides a method for diagnosing and segmenting positive lymph nodes in CT images based on multi-task learning, and its flow chart is as follows: Figure 1 As shown, the framework flow chart is as follows Figure 2 The method comprises the following steps:

[0030] Step 1: Preprocessing: The acquired CT image sequence is preprocessed to adapt to the input of the deep neural network model, and data enhancement operations are added to improve the generalization ability of the model;

[0031] The original CT image size is 512×512 pixels. Since the black background occupies most of the space, and the base of the CT device will also appear white after imaging, these will affect the segmentation effect of the model. In addition, due to factors such as the patient's posture, the position of the patient's head and neck in the original CT image is not fixed, and it is difficult to remove the background part using a fixed value cropping operation. Therefore, an additional U-Net model is first used to remove all background interference, and then the intensity is sheared in the range of [0, 300HU] to retain the lymph node HU value, and these values ​​are normalized to the range of [0, 1.0], and data enhancement operations such as random rotation, random upside down flipping, and random left and right flipping are added.

[0032] Step 2: Build a multi-task deep learning model, which includes an encoder, a classification module, and a segmentation module;

[0033] Step 3, feature extraction: The encoder is used for feature extraction. The encoder uses a multi-layer convolutional neural network to extract local features of the image and gradually gather them together. At the same time, the pooling operation is used to reduce the feature size, and finally the required features are obtained;

[0034] The encoder consists of 4 layers of stacked down modules (see Figure 3 ), each down module contains two convolutions, batch normalization, and activation function operations, and finally the down-sampling operation is completed through the maximum pooling layer; after being processed by the encoder, each CT slice obtains a corresponding feature vector.

[0035] Step 4, classification task: The classification task is performed using a classification module. The classification module uses the features extracted in step 3 as input and obtains the classification result of each feature through calculation. Output 0 means that the classification module believes that the CT slice corresponding to the feature has no positive lymph nodes, and output 1 means that the classification module believes that the CT slice corresponding to the feature has positive lymph nodes.

[0036] The classification module consists of multiple 1×1 convolutional layers and fully connected layers. Each 1×1 convolutional layer is used to reduce the dimension of the feature. Finally, the feature is flattened and mapped to the output category through the fully connected layer to obtain the classification result.

[0037] Step 5, segmentation task: The segmentation task is performed using a segmentation module. The segmentation module takes the features extracted in step 3 as input, obtains the segmentation result of each feature by restoring the feature size, and then removes useless segmentation results through classification results, and finally obtains the segmentation result of the CT slice with only positive lymph nodes;

[0038] The segmentation module consists of a fusion module (see Figure 5 ) and 4 up modules (see Figure 4). The fusion module contains two SE blocks, 1×1 convolution, batch normalization, and activation function operations; SE block suppresses unimportant channels in the data and assigns them lower weights through squeezing and excitation operations, and encourages important channels in the data and assigns them higher weights. The structure of the up module is similar to that of the down module, except that the up module is finally upsampled by transposed convolution. The input of each up module consists of two parts, one is the output of the previous layer of the network, and the other is the output of the corresponding down module for the i-th slice, which is consistent with the jump connection of the original U-Net.

[0039] Example 1. The experimental data of this example come from 92 nasopharyngeal carcinoma patients who visited the Radiotherapy Department of the First Hospital of Jilin University from January 1, 2020 to December 31, 2023. All patients underwent computed tomography examinations in the supine position according to the doctor's instructions, and were accurately judged and outlined by experienced clinicians. In order to verify the performance of the model, the data set was divided into 71 training sets and 18 test sets. Three different models were compared in the experiment: the multi-task deep learning model proposed in the present invention, a separate 2D segmentation model, and a separate 3D segmentation model. These models were compared in terms of execution time and related task indicators, and the specific results are shown in Table 1:

[0040] Table 1 Comparison results

[0041]

[0042] It can be seen from the data results in Table 1 that the classification accuracy of the multi-task deep learning model proposed in the present invention can reach 91.1%, the DICE coefficient of segmentation can reach 0.602, and the average time for a single patient in the whole process is 102s. In contrast, the DICE coefficient of the 2D segmentation model that only performs the segmentation task is only 0.537, and the average time for a single patient is 13.2s. Although its execution efficiency is higher than that of the present invention, its segmentation index is significantly lower than that of the present invention, and the classification task is not completed. Similarly, the DICE coefficient of the 3D segmentation model that only performs the segmentation task is 0.610, and the average time for a single patient is 244s. The segmentation index is basically the same as that of the present invention, but the time consumption is more than twice that of the present invention, and the classification task is also unable to be completed.

[0043] In summary, the present invention fills the gap in simultaneous positive lymph node determination and delineation in head and neck CT images of nasopharyngeal carcinoma patients, not only improving the accuracy of segmentation and classification, but also greatly reducing the workload of doctors and providing a certain auxiliary effect for doctors.

[0044] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for diagnosing and segmenting positive lymph nodes in CT images based on multi-task learning, characterized in that: The following steps are involved: Step 1: Preprocessing: preprocess the acquired CT image sequence and add data enhancement operation; Step 2: Build a multi-task deep learning model, which includes an encoder, a classification module, and a segmentation module; Step 3, feature extraction: The encoder is used for feature extraction. The encoder uses a multi-layer convolutional neural network to extract local features of the image and gradually gather them together. At the same time, the pooling operation is used to reduce the feature size, and finally the required features are obtained; Step 4: Classification task: The classification task is performed using a classification module. The classification module takes the features extracted in step 3 as input and obtains the classification result of each feature through calculation. Output 0 means that the classification module believes that the CT slice corresponding to the feature has no positive lymph nodes, and output 1 means that the classification module believes that the CT slice corresponding to the feature has positive lymph nodes; Step 5, segmentation task: The segmentation task is performed using the segmentation module. The segmentation module takes the features extracted in step 3 as input, obtains the segmentation result of each feature by restoring the feature size, and then removes useless segmentation results through the classification results, and finally obtains the segmentation result of the CT slice with only positive lymph nodes.

2. The method for diagnosing and segmenting positive lymph nodes in CT images based on multi-task learning according to claim 1, characterized in that: The preprocessing operation includes using an additional U-Net model to remove all background interference, then clipping the intensity in the range of [0, 300HU] to retain the lymph node HU values, and normalizing the lymph node HU values ​​to the range of [0, 1.0].

3. The method for diagnosing and segmenting positive lymph nodes in CT images based on multi-task learning according to claim 1, characterized in that: The data enhancement operations include random rotation, random upside down flipping, and random left-right flipping.

4. The method for diagnosing and segmenting positive lymph nodes in CT images based on multi-task learning according to claim 1, characterized in that: The encoder consists of 4 layers of stacked down modules, each of which includes two convolutions, batch normalization, and activation function operations, and finally completes the downsampling operation through the maximum pooling layer; after being processed by the encoder, each CT slice obtains a corresponding feature vector.

5. The method for diagnosing and segmenting positive lymph nodes in CT images based on multi-task learning according to claim 1, characterized in that: The classification module consists of multiple 1×1 convolutional layers and fully connected layers. Each 1×1 convolutional layer is used to reduce the dimension of the features. Finally, the features are flattened and mapped to the output category through the fully connected layer to obtain the classification result.

6. The method for diagnosing and segmenting positive lymph nodes in CT images based on multi-task learning according to claim 4, characterized in that: The segmentation module consists of a fusion module and four up modules; the fusion module includes two SE blocks, 1×1 convolution, batch normalization, and activation function operations; the SE block suppresses unimportant channels in the data and assigns low weights, and encourages important channels in the data and assigns high weights through the two operations of squeezing and excitation; each up module includes two convolutions, batch normalization, and activation function operations, and finally up-samples through transposed convolution; the input of each up module consists of two parts, one is the output of the previous layer of the network, and the other is the output of the corresponding down module for the i-th slice.

Images