A thyroid papillary carcinoma pathological image classification method and system based on multi-task learning

Abstract

The application discloses a thyroid papillary carcinoma pathological image classification method based on multi-task learning, realizes multi-task learning on high cell type tumor cell rotation target detection, cell nucleus target detection and cell nucleus classification, and further takes the abnormal evaluation of the high cell type tumor cell detection and the abnormal evaluation of the cell nucleus detection as two auxiliary tasks. The application effectively overcomes the problem of insufficient single task information utilization, improves the accuracy and efficiency of the classification and / or statistics of the thyroid papillary carcinoma pathological image through the information mutual assistance among the multi-tasks.

Description

Technical Field

[0001] This invention discloses a method and system for classifying pathological images of papillary thyroid carcinoma based on multi-task learning, belonging to the technical field of intelligent learning classification of pathological images. Background Technology

[0002] Papillary thyroid carcinoma is the most common malignant tumor of the thyroid gland, accounting for more than half of all thyroid malignancies, and it affects a very wide age range. The vast majority of papillary thyroid carcinomas have a good prognosis, with slow tumor growth, a low rate of invasion and metastasis, and a 10-year survival rate greater than 90%. However, high-risk subtypes of papillary thyroid carcinoma have a poor prognosis, with rapid tumor proliferation, strong invasiveness and destructive power, a high tendency to metastasize early, and shorter survival times. Therefore, accurate diagnosis of high-risk subtypes of papillary thyroid carcinoma is crucial for subsequent treatment and prognosis.

[0003] Among the high-risk subtypes of papillary thyroid carcinoma, the high-cellular type accounts for the largest proportion, approximately 10% of all papillary thyroid carcinomas. High-cellular tumor cells, in addition to exhibiting typical nuclear features of papillary thyroid carcinoma, such as ground-glass nuclei, nuclear grooves, irregular nuclear membranes, intranuclear pseudo-inclusions, and an increased nuclear-cytoplasmic ratio, should have a cell height that is three times or more than the width, and these cells should constitute 30% or more of the total tumor cells. Diagnosis is based on the latest diagnostic criteria for head and neck tumors (5th edition of the WHO). Furthermore, high-cellular tumor cells have more abundant eosinophilic cytoplasm, and necrosis and mitotic figures are visible. Fine-needle aspiration cytology (FNAC) is a minimally invasive, convenient, and rapid diagnostic technique that can assist clinicians in determining subsequent treatment plans. Currently, thyroid cell diagnosis mainly relies on manual slide reading. The large volume of cytology smears, the high number of cells, and the diverse morphological changes, coupled with the requirement for cytological diagnostic reports to be issued within 24 hours, create significant time pressure and place immense stress on pathologists. The fatigue caused by prolonged high-intensity observation and the uneven quality of diagnostic skills, especially in the diagnosis of high-cell papillary thyroid carcinoma, can easily lead to strong subjectivity, large estimation errors, and poor diagnostic consistency, which seriously affects the accuracy of diagnosis.

[0004] Therefore, there is an urgent need in the industry for a method to classify pathological images of papillary thyroid carcinoma in order to identify target cells, especially for the identification and statistical analysis of high-cell types. The identification criteria for high-cell types are: cell aspect ratio ≥3 and high-cell percentage ≥30%. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention discloses a method for classifying pathological images of papillary thyroid carcinoma based on multi-task learning.

[0006] The present invention also discloses a classification system for implementing the above method. Invention Overview:

[0008] A multi-task learning-based method for classifying pathological images of papillary thyroid carcinoma is proposed. This method performs multi-task learning for the detection of rotational targets (high-cellularity tumor cells), nuclear targets, and nuclear classification. Furthermore, it incorporates anomaly assessment for high-cellularity tumor cell detection and anomaly assessment for nuclear detection as two auxiliary tasks. This invention effectively overcomes the problem of insufficient information utilization in single-task scenarios by leveraging information exchange between multiple tasks to improve the accuracy and efficiency of classification and / or statistical work on papillary thyroid carcinoma pathological images.

[0009] The detailed technical solution of this invention is as follows:

[0010] A method for classifying pathological images of papillary thyroid carcinoma based on multi-task learning, characterized by comprising:

[0011] 1) Collect pathological images separately: pathological images of high-cell papillary thyroid carcinoma and pathological images of papillary thyroid carcinoma other than high-cell type; preferably, select the above pathological images as JPG format cytopathological images magnified 400 times;

[0012] 2) Normalize the acquired pathological images: Staining normalization operation to reduce the influence of human factors in the staining process on staining depth, etc.

[0013] 3) Manually annotate the pathological images from step 2):

[0014] The nuclei of papillary thyroid carcinoma cells were labeled using bounding boxes; the high-cell thyroid papillary carcinoma cells were labeled using rotated bounding boxes; other papillary thyroid carcinoma cells were labeled using bounding boxes; all labeled papillary thyroid carcinoma cells were independently separated, and rotation and flipping operations at different angles were used to expand the dataset.

[0015] 4) Construct a pathological image classification model:

[0016] The network structure includes RetinaNet, which comprises a ResNet50 Backbone module. Within the ResNet50 Backbone module, Conv1, C2, C3, C4, and C5 layers are sequentially arranged along the data flow direction. Layers C3, C4, and C5 output target detection feature maps for large, medium, and small targets, respectively, and these maps are then connected to subsequent multi-task branch head modules. The data flow includes the acquired image dataset and its corresponding labels. The image dataset is input to the RetinaNet network structure, as shown below. Figure 2The feature maps are processed through the Conv1 layer, then sequentially through layers C2, C3, C4, and C5. The features obtained from layers C3, C4, and C5 at different scales are then extracted to obtain three target detection feature maps of 20*20*256, 40*40*256, and 80*80*256 respectively, targeting large, medium, and small targets. These maps are then input into three sets of multi-task branch heads. To fully utilize information from the cell nucleus and cytoplasm, this invention designs a multi-task branch head, which is an improvement on the classification and detection heads of the RetinaNet network structure. The detection branch is expanded into a cell nucleus detection branch and a high-cell detection branch. Since cell nucleus detection only utilizes cell nucleus information, while high-cell detection utilizes both cell nucleus and cytoplasm information, this approach is more efficient. To improve the quality of information, the RetinaNet network structure detection branch was expanded into two parallel branches, and two additional anomaly detection branches were added to evaluate whether the detection of papillary thyroid carcinoma nuclei and high-cell thyroid carcinoma nuclei were abnormal. This provides additional constraints for the nucleus and high-cell detection branches. The network output includes five items: assessment of abnormality in papillary thyroid carcinoma nuclei (tumor cell nucleus detection anomaly assessment), detection of papillary thyroid carcinoma nuclei (nucleus detection branch), classification of papillary thyroid carcinoma nuclei (nucleus classification branch), detection of high-cell thyroid carcinoma nuclei (high-cell detection branch), and assessment of abnormality in high-cell thyroid carcinoma nuclei (high-cell tumor cell detection anomaly assessment).

[0017] 5) Constructing a recognition network:

[0018] The single-group multi-task branch header includes:

[0019] Assessment of abnormalities in papillary thyroid carcinoma nuclei (assessment of abnormalities in tumor cell nuclei), branch of papillary thyroid carcinoma nuclei detection (nucleus detection branch), branch of classification of papillary thyroid carcinoma nuclei (nucleus classification branch), branch of detection of high-cytosis papillary thyroid carcinoma (high-cytosis detection branch), assessment of abnormalities in detection of high-cytosis papillary thyroid carcinoma (assessment of abnormalities in detection of high-cytosis tumor cells).

[0020] According to a preferred embodiment of the present invention, step 5) further includes:

[0021] The target detection feature map is used as input. The data obtained after processing by the thyroid papillary carcinoma cell nucleus detection branch is fused with the target detection feature map by the ROIAlign module and then output as the final result of the thyroid papillary carcinoma cell nucleus abnormality assessment after three layers of convolution.

[0022] The target detection feature map is used as input. After processing by the high-cell papillary thyroid carcinoma detection branch, the resulting data is fused with the target detection feature map by the ROIAlign module and then subjected to three layers of convolution to finally output the abnormal assessment result of high-cell papillary thyroid carcinoma detection.

[0023] According to a preferred embodiment of the present invention, the abnormal assessment results of papillary thyroid carcinoma nuclei and the abnormal assessment results of high-cellularity papillary thyroid carcinoma are the similarity between the label and the prediction. Users can pre-set a threshold for this similarity based on the application scenario, thereby guiding users to automatically identify whether the corresponding detection results are abnormal.

[0024] The classification method also includes two auxiliary tasks: assessment of abnormal tumor cell nucleus detection and assessment of abnormal high-cytoplasmic tumor cell detection; wherein, the assessment of abnormal tumor cell nucleus detection is used to combine the output results of the tumor cell nucleus detection box to determine the abnormality of tumor cell nucleus detection in the detected area;

[0025] The assessment of abnormal detection of high-cellularity tumor cells is used to combine the output results of the high-cellularity tumor cell rotation detection box to determine the abnormality of high-cellularity tumor cell detection in the detected area.

[0026] According to a preferred embodiment of the present invention, the classification method further includes: automatically counting the number of high-cell papillary thyroid cancer cells identified by classification, and calculating the proportion of high-cell papillary thyroid cancer cells to the total number of papillary thyroid cancer cells.

[0027] The classification system implementing the above method includes: a dataset module, a manual annotation module, a pathological image classification model, and a recognition network module.

[0028] The dataset module is used to implement the contents recorded in steps 1) and 2).

[0029] The manual annotation module is used to implement the content recorded in step 3);

[0030] The pathological image classification model is used to implement the content recorded in step 4);

[0031] The identification network module is used to implement the content recorded in step 5).

[0032] Beneficial technical effects of the present invention:

[0033] This invention proposes a screening and detection method for high-cellularity thyroid papillary carcinoma that combines tumor cell nucleus detection and high-cellularity tumor cell rotation detection based on multi-task learning. By automatically extracting inter-task cooperative features through multi-task learning, it overcomes the shortcomings of traditional deep learning, which only provides diagnostic results and cannot show the diagnostic basis, and cannot make full use of the relationships between different tasks.

[0034] The technical advantage of data training in the classification method described in this invention is that it can simultaneously achieve cell counting and high-cell count through cell nucleus detection and high-cell detection, directly obtaining diagnostic results. The technical advantage of multi-task input is that learning across multiple tasks can automatically extract mutually beneficial features between tasks through neural networks. For example, the cell nucleus detection result is always within the high-cell detection result. By inputting multiple tasks with strong positional correlation and simultaneously learning, multiple tasks can be achieved simultaneously. Information interaction between tasks is mainly through common feature layers, such as... Figure 2 The network employs three feature layers of varying sizes. These feature layers are input into the multi-task branch of this invention to complete the constraint learning for a single task. Because the network is trained end-to-end as a whole, it eventually learns feature layers that are applicable to multiple tasks simultaneously.

[0035] This invention also utilizes anomaly assessment of detection results to reduce negative impacts between tasks. This is because anomaly assessment corresponds to the two target detection tasks: tumor cell nucleus detection and high-cytoplasm tumor cell detection. This effectively separates and constrains the two sub-tasks at the target level. Since the main difference between tumor cell nuclei and high-cytoplasm tumor cells lies in the presence or absence of intact cytoplasm, anomaly assessment at the target level helps eliminate negative impacts between tasks. Attached Figure Description

[0036] Figure 1 This is a flowchart of the classification method described in this invention;

[0037] Figure 2 This is the network structure diagram corresponding to step 4) in this invention;

[0038] Figure 3 This is the network structure diagram corresponding to step 5) in this embodiment of the invention;

[0039] Figure 4 This is a schematic diagram showing the selected thyroid papillary carcinoma cells identified in an embodiment of the present invention. Detailed Implementation

[0040] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0041] Example 1

[0042] like Figure 1 As shown.

[0043] A method for classifying pathological images of papillary thyroid carcinoma based on multi-task learning, characterized by comprising:

[0044] 1) Collect pathological images separately: pathological images of high-cell papillary thyroid carcinoma and pathological images of papillary thyroid carcinoma other than high-cell type; preferably, select the above pathological images as JPG format cytopathological images magnified 400 times;

[0045] 2) Normalize the acquired pathological images: Staining normalization operation to reduce the influence of human factors in the staining process on staining depth, etc.

[0046] 3) Manually annotate the pathological images from step 2): such as... Figure 4 As shown,

[0047] The nuclei of papillary thyroid carcinoma cells were labeled using bounding boxes; the high-cell thyroid papillary carcinoma cells were labeled using rotated bounding boxes; other papillary thyroid carcinoma cells were labeled using bounding boxes; all labeled papillary thyroid carcinoma cells were independently separated, and rotation and flipping operations at different angles were used to expand the dataset.

[0048] 4) Construct a pathological image classification model: such as Figure 2 As shown,

[0049] The network structure includes RetinaNet, which comprises a ResNet50 Backbone module. Within the ResNet50 Backbone module, Conv1, C2, C3, C4, and C5 layers are sequentially arranged along the data flow direction. Layers C3, C4, and C5 output target detection feature maps for large, medium, and small targets, respectively, and these maps are then connected to subsequent multi-task branch head modules. The data flow includes the acquired image dataset and its corresponding labels. The image dataset is input to the RetinaNet network structure, as shown below. Figure 2The feature maps are processed through the Conv1 layer, then sequentially through layers C2, C3, C4, and C5. The features obtained from layers C3, C4, and C5 at different scales are then extracted to obtain three target detection feature maps of 20*20*256, 40*40*256, and 80*80*256 respectively, targeting large, medium, and small targets. These maps are then input into three sets of multi-task branch heads. To fully utilize information from the cell nucleus and cytoplasm, this invention designs a multi-task branch head, which is an improvement on the classification and detection heads of the RetinaNet network structure. The detection branch is expanded into a cell nucleus detection branch and a high-cell detection branch. Since cell nucleus detection only utilizes cell nucleus information, while high-cell detection utilizes both cell nucleus and cytoplasm information, this approach is more efficient. To improve the quality of information, the RetinaNet network structure detection branch was expanded into two parallel branches, and two additional anomaly detection branches were added to evaluate whether the detection of papillary thyroid carcinoma nuclei and high-cell thyroid carcinoma nuclei were abnormal. This provides additional constraints for the nucleus and high-cell detection branches. The network output includes five items: assessment of abnormality in papillary thyroid carcinoma nuclei (tumor cell nucleus detection anomaly assessment), detection of papillary thyroid carcinoma nuclei (nucleus detection branch), classification of papillary thyroid carcinoma nuclei (nucleus classification branch), detection of high-cell thyroid carcinoma nuclei (high-cell detection branch), and assessment of abnormality in high-cell thyroid carcinoma nuclei (high-cell tumor cell detection anomaly assessment).

[0050] 5) Construct a recognition network: such as Figure 3 As shown,

[0051] The single-group multi-task branch header includes:

[0052] Assessment of abnormalities in papillary thyroid carcinoma nuclei (assessment of abnormalities in tumor cell nuclei), branch of papillary thyroid carcinoma nuclei detection (nucleus detection branch), branch of classification of papillary thyroid carcinoma nuclei (nucleus classification branch), branch of detection of high-cytosis papillary thyroid carcinoma (high-cytosis detection branch), assessment of abnormalities in detection of high-cytosis papillary thyroid carcinoma (assessment of abnormalities in detection of high-cytosis tumor cells).

[0053] Example 2

[0054] The classification method described in Example 1 further includes, in step 5), the following:

[0055] The target detection feature map is used as input. The data obtained after processing by the thyroid papillary carcinoma cell nucleus detection branch is fused with the target detection feature map by the ROIAlign module and then output as the final result of the thyroid papillary carcinoma cell nucleus abnormality assessment after three layers of convolution.

[0056] The target detection feature map is used as input. After processing by the high-cell papillary thyroid carcinoma detection branch, the resulting data is fused with the target detection feature map by the ROIAlign module and then subjected to three layers of convolution to finally output the abnormal assessment result of high-cell papillary thyroid carcinoma detection.

[0057] The abnormal assessment results for papillary thyroid carcinoma nuclei and high-cytosis papillary thyroid carcinoma are the similarity between the labels and predictions. Users can pre-set the similarity threshold according to their application scenarios, thereby guiding users to automatically identify whether the corresponding detection results are abnormal.

[0058] The classification method also includes two auxiliary tasks: assessment of abnormal tumor cell nucleus detection and assessment of abnormal high-cytoplasmic tumor cell detection; wherein, the assessment of abnormal tumor cell nucleus detection is used to combine the output results of the tumor cell nucleus detection box to determine the abnormality of tumor cell nucleus detection in the detected area;

[0059] The assessment of abnormal detection of high-cellularity tumor cells is used to combine the output results of the high-cellularity tumor cell rotation detection box to determine the abnormality of high-cellularity tumor cell detection in the detected area.

[0060] Example 3

[0061] The classification method described in Example 1 further includes: automatically counting the number of tall cell papillary thyroid cancer cells identified by classification, and calculating the proportion of tall cell papillary thyroid cancer cells to the total number of papillary thyroid cancer cells.

[0062] Example 4

[0063] The classification system implementing the method described in Examples 1, 2, and 3 includes: a dataset module, a manual annotation module, a pathological image classification model, and a recognition network module.

[0064] The dataset module is used to implement the contents recorded in steps 1) and 2).

[0065] The manual annotation module is used to implement the content recorded in step 3);

[0066] The pathological image classification model is used to implement the content recorded in step 4);

[0067] The identification network module is used to implement the content recorded in step 5).

Claims

1. A method for classifying thyroid papillary carcinoma pathological images based on multi-task learning, characterized in that, Comprise: 1) Collect pathological images respectively: high cell type papillary thyroid carcinoma pathological images, papillary thyroid carcinoma pathological images other than high cell type; 2) Normalize the collected pathological images: staining normalization operation; 3) Manually annotate the pathological images of step 2): Annotate the thyroid papillary carcinoma cell nucleus with a rectangular box; Annotate the high cell type thyroid papillary carcinoma cell with a rotating rectangular box; Annotate the thyroid papillary carcinoma cell other than high cell type with a rectangular box; Independently split all the annotated thyroid papillary carcinoma cells, and use different angles of rotation and flipping to expand the data set; 4) Construct a pathological image classification model: The network structure of RetinaNet includes a ResNet50 Backbone module, which includes Conv1 layer, C2 layer, C3 layer, C4 layer and C5 layer in sequence along the data flow direction, respectively outputs target detection feature maps for large targets, medium targets and small targets through C3 layer, C4 layer and C5 layer respectively, and is connected with subsequent multi-task branch head module; 5) Construct an identification network: The multi-task branch head module comprises: Thyroid papillary carcinoma cell nucleus detection abnormality evaluation, thyroid papillary carcinoma cell nucleus detection branch, thyroid papillary carcinoma cell nucleus classification branch, high cell type thyroid papillary carcinoma cell detection branch, high cell type thyroid papillary carcinoma cell detection abnormality evaluation.

2. The thyroid papillary carcinoma pathological image classification method based on multi-task learning according to claim 1, characterized in that, In step 5), it also includes: The target detection feature map is taken as input, the data obtained after the thyroid papillary carcinoma cell nucleus detection branch processing is fused with the target detection feature map through ROIAlign module, and finally the thyroid papillary carcinoma cell nucleus detection abnormality evaluation result is output after three layers of convolution; The target detection feature map is taken as input, the data obtained after the high cell type thyroid papillary carcinoma cell detection branch processing is fused with the target detection feature map through ROIAlign module, and finally the high cell type thyroid papillary carcinoma cell detection abnormality evaluation result is output after three layers of convolution.

3. The thyroid papillary carcinoma pathological image classification method based on multi-task learning according to claim 2, characterized in that, The thyroid papillary carcinoma cell nucleus detection abnormality evaluation result and the high cell type thyroid papillary carcinoma cell detection abnormality evaluation result are the similarity between the label and the prediction.

4. The thyroid papillary carcinoma pathological image classification method based on multi-task learning according to claim 1, characterized in that, The classification method further comprises: automatically counting the high cell type thyroid papillary carcinoma cells identified by classification, and calculating the proportion of high cell type thyroid papillary carcinoma cells in total thyroid papillary carcinoma cells.

5. A classification system implementing the method of any one of claims 1-4, comprising: Dataset module, artificial annotation module, pathological image classification model and identification network module: The dataset module is used to realize the contents recorded in steps 1) and 2); The artificial annotation module is used to realize the contents recorded in step 3); The pathological image classification model is used to realize the contents recorded in step 4); The identification network module is used to realize the contents recorded in step 5).

Images