Thyroid pathology image auxiliary analysis system

Abstract

The invention belongs to the technical field of medical image analysis, and particularly relates to a thyroid cytopathology image auxiliary analysis system. The system comprises an input module used for inputting a thyroid cell pathology slice image; the cell detection module is used for detecting target cells through a neural network model; the semantic classification module is used for carrying out semantic classification on the target cells to obtain classification tags; and the knowledge graph searching module is used for searching in the knowledge graph according to the classification labels and carrying out text interpretation on the obtained diagnosis result. The system disclosed by the invention can be used for carrying out high-accuracy auxiliary diagnosis on papillary thyroid carcinoma. And meanwhile, a diagnosis basis and reference can be provided for a diagnosis result according to a knowledge graph, so that the AI auxiliary pathological diagnosis has credibility. Therefore, the method has a good application prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image analysis, and in particular relates to a thyroid pathology image auxiliary analysis system. Background Art

[0002] In recent years, the incidence of various malignant tumors has increased year by year worldwide, among which the incidence of thyroid malignant tumors has increased significantly, and the detection rate of thyroid nodules in the physical examination population is 24.4%-33.22%. Fine Needle Aspiration Cytology (FNAC) under the guidance of thyroid ultrasound is recognized as the most specific and accurate diagnostic method for thyroid nodules. Due to its advantages of less trauma, economy and convenience, it has gradually become the main preoperative examination method for thyroid nodules in clinical practice.

[0003] In the current field of pathology, pathological diagnosis has begun to be deeply integrated with artificial intelligence (AI) technology. People hope to use AI's powerful learning and computing capabilities to reduce a lot of repetitive work and subjective judgments of pathologists. Previous studies and most pathological AI use convolutional neural networks (CNNs) based on deep learning for image analysis. Many scholars have also reported on this in the field of thyroid FNAC. CNN is very good at image processing, but the disadvantage of CNN technology is that the "black box" effect makes it difficult to summarize the laws of observation. This is an obvious disadvantage of the "second generation of AI", which makes it difficult for pathologists and patients to fully trust the judgments made by AI during pathological diagnosis. Therefore, it is necessary to solve the problem that AI diagnostic results are unexplainable. To this end, how to select an appropriate model algorithm to build a thyroid pathology image-assisted analysis system with good interpretability and high predictive performance is still an urgent problem to be solved in this field.

[0004] GNN is a graph data structure, and GNN technology is the technology of neural network algorithms that run directly on graph data structures. Knowledge graph is a new concept proposed by Google in 2012. It is essentially a knowledge base of semantic networks, which can also be understood as a multi-relational graph composed of nodes and edges. However, there are no reports on the application of GNN and knowledge graph in thyroid pathology image analysis. Summary of the invention

[0005] In view of the problems in the prior art, the present invention provides a thyroid pathology image-assisted analysis system.

[0006] A thyroid pathology image-assisted analysis system, comprising:

[0007] An input module, used for inputting thyroid cell pathology slice images;

[0008] A cell detection module is used to detect target cells through a neural network model;

[0009] The semantic classification module is used to semantically classify the target cells and obtain classification labels;

[0010] The knowledge graph search module is used to search in the knowledge graph according to classification labels and provide textual interpretation of the obtained diagnostic results.

[0011] Preferably, the neural network model is selected from the Cascade-R-CNN model.

[0012] Preferably, the target cells are thyroid papillary carcinoma cells, multinuclear macrophages, normal thyroid cells or other background cells.

[0013] Preferably, the semantic classification is implemented using GINet, a graph reasoning method based on semantic features.

[0014] Preferably, the classification labels include at least one of the following labels: ground glass nucleus, nuclear groove, intranuclear pseudoinclusion, multinucleated macrophage.

[0015] Preferably, the method for constructing the knowledge graph includes:

[0016] Step 1, search for literature related to thyroid pathology images;

[0017] Step 2: Extract entities and relations from the document;

[0018] Step 3: Perform knowledge fusion to solve entity disambiguation and coreference disambiguation problems of equivalent instances, equivalent classes / subclasses, and equivalent attributes / subattributes;

[0019] Step 4: Use Neo4j graph database for storage.

[0020] Preferably, in step 2, the spaCy library of pytorch is used for entity extraction and relationship extraction.

[0021] Preferably, in step 4, the storage structure is composed of four elements: labels, nodes, relationships, and attributes.

[0022] The present invention also provides a computer-readable storage medium, on which is stored: a computer program for implementing the above-mentioned thyroid pathology image-assisted analysis system.

[0023] The present invention constructs a system based on a neural network model, a graph neural network and a knowledge graph, and further optimizes the algorithm selection of each model. Specifically, the neural network model divides the thyroid pathology image into small images including target cells; further, based on the graph neural network, it can not only accurately identify the target cells, but also, compared with other AI-assisted pathological diagnosis studies, it can also classify the karyotype of the identified tumor cells through GINet, and then use the knowledge graph to interpret the classification results in text. It can be seen that the system of the present invention can not only make a diagnosis of the diagnostic category of thyroid cell pathology papillary thyroid carcinoma (PTC), but also provide a diagnostic basis and references, so that AI-assisted pathological diagnosis has credibility. Therefore, the present invention has a good application prospect.

[0024] Obviously, according to the above contents of the present invention, in accordance with common technical knowledge and customary means in the art, without departing from the above basic technical ideas of the present invention, other various forms of modification, replacement or change may be made.

[0025] The above contents of the present invention are further described in detail below through specific implementation methods in the form of embodiments. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following examples. All technologies realized based on the above contents of the present invention belong to the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the loss function curve of the Casdade-R-CNN model in Example 1;

[0027] Figure 2 is the result of the semantic loss weight ablation experiment in Example 1;

[0028] Figure 3 It is the Train / loss curve of the semantic classification model in Example 1. DETAILED DESCRIPTION

[0029] It should be noted that the algorithms of data collection, transmission, storage and processing steps not specifically described in the embodiments, as well as the hardware structure, circuit connection, etc. not specifically described can all be implemented through the contents disclosed in the prior art.

[0030] Example 1 Thyroid Pathology Image-Assisted Analysis System

[0031] This embodiment provides a thyroid pathology image-assisted analysis system, including:

[0032] An input module, used for inputting thyroid cell pathology slice images;

[0033] The cell detection module is used to detect target cells using the Cascade-R-CNN model;

[0034] The semantic classification module uses GINet, a graph reasoning method based on semantic features, to semantically classify the target cells and obtain classification labels;

[0035] The knowledge graph search module is used to search in the knowledge graph according to classification labels and provide textual interpretation of the obtained diagnostic results.

[0036] The target cells are thyroid papillary carcinoma cells, multinuclear macrophages, normal thyroid cells or other background cells. The classification labels include at least one of the following labels: ground glass nucleus, nuclear groove, intranuclear pseudoinclusion, multinuclear macrophage.

[0037] The training of the above model and the construction of the knowledge graph are carried out in the following steps:

[0038] Step A, cell nucleus annotation: According to the classification of the Bthesda Reporting System for Thyroid Cytopathology, the nuclear characteristics of papillary thyroid carcinoma mainly include: nuclear enlargement, nuclear crowding, nuclear overlap, nuclear atypia, ground glass nucleus, nuclear grooves, and nuclear pseudoinclusions; background components include: multinuclear macrophages. In the Qupath3.0 software, add ground glass nucleus, nuclear grooves, nuclear pseudoinclusions, and multinuclear macrophage category labels to annotate the cell nuclei with corresponding characteristics, and set the "Tumour" label to annotate tumor cells that only have nuclear enlargement or nuclear atypia.

[0039] Step B, literature annotation: Enter keywords such as "thyroid", "papillary carcinoma", "cytology", "pathology" in the search engine and use "and" or "or" to search for literature. Literature inclusion criteria: The literature content must contain direct or indirect descriptions of the pathological morphology of papillary carcinoma cells, such as nuclear crowding, nuclear overlap, nuclear enlargement, nuclear pleomorphism, ground glass nuclei, nuclear grooves, nuclear pseudoinclusions, background cells, multinucleated macrophages, etc. Set semantic types include: tumor cells, nuclear crowding, nuclear overlap, nuclear enlargement, nuclear pleomorphism, ground glass nuclei, nuclear grooves, nuclear pseudoinclusions, background cells, multinucleated macrophages; semantic relationships include: cell nuclear characteristics, arrangement, belonging to. After annotation according to the defined semantic type, associate the annotated text content;

[0040] Step C, the computer reads the digital slice image file that has been annotated and cytologically classified, and trains the Cascade-R-CNN model and the graph reasoning method GINet;

[0041] In this example, the labeled training set data is imported into the Cascade-R-CNN model for training. We obtain mAPs of 0.89, 0.73 and 0.87, 0.69 for the training set and validation set when IoU is 0.5 and 0.75, respectively (Table 1). The test loss (test-loss) and detection loss (train-loss) tend to decrease with the increase of iterations ( Figure 1 ) The curve indicates that the model training effect is good.

[0042] Table 1 Casdade-R-CNN model training set and validation set mAP values

[0043]

[0044] * Note: mAP@.5 / .75 is the average mAP value when IoU=0.5 / 0.75.

[0045] The training set is imported into the GINet model for image classification training, and the model is evaluated using mIoU and Dice. In order to study the necessity and effectiveness of semantic loss, the weight hyperparameter λ = {0.2, 0.4, 0.6, 0.8, 1.0} is set for training according to the general standard ( Figure 2 ), the results show that when λ is 0.2, the best training results are mIoU 0.5614 and Dice 0.6470. In addition, as the number of iterations increases during the training process, Train loss continues to decline, indicating that the model training effect is good ( Figure 3 ).

[0046] Step D, construction of knowledge graph: data mainly comes from relevant literature, and pytorch's spaCy library is used for entity extraction and relationship extraction. After knowledge acquisition, knowledge fusion is performed to solve the entity disambiguation and coreference disambiguation problems of equivalent instances, equivalent classes / subclasses, and equivalent attributes / subattributes. Finally, Neo4j graph database is used for storage, and the storage structure consists of four elements: labels, nodes, relationships, and attributes.

[0047] The trained Cascade-R-CNN model, GINet model and the constructed knowledge graph information were imported into the CDSS software. The internal link module of the CDSS software jointly analyzed the data of each step, diagnosed it as PTC and was able to determine the relevant semantic type and literature description, with a compliance rate of 88.84%.

[0048] The technical solution of the present invention is further illustrated by experiments below.

[0049] Experimental Example 1 Optimal selection of target cell detection model

[0050] 1. Experimental Methods

[0051] From the data collected from the hospital, nearly 1,000 thyroid cell samples were labeled through pre-labeling of the segmentation model and manual inspection and correction, and divided into training and test sets in a ratio of 8:2.

[0052] The above datasets were used to train the Cascade-R-CNN model and other comparison models. During the training, except for the variables of the model and pre- and post-processing, other variables remained unchanged, and random factors such as the model parameter initialization scheme were kept unchanged.

[0053] 2. Experimental Results

[0054] The performance of the Cascade-R-CNN model and other comparison models are shown in Table 2.

[0055]

[0056] From the data in the above table, it can be seen that among various existing neural network models, the Cascade-R-CNN model has the best prediction performance for the target cell detection task of the present invention.

[0057] Through the above embodiments and experimental examples, it can be seen that the present invention provides a system that combines a target cell detection model, a target cell semantic classification model and a knowledge graph, which can perform high-accuracy auxiliary diagnosis of papillary thyroid carcinoma. At the same time, the diagnosis results can provide diagnostic basis and references based on the knowledge graph, making AI-assisted pathological diagnosis credible. Therefore, the present invention has a good application prospect.

Claims

1. A thyroid pathology image-assisted analysis system, characterized in that: include: An input module, used for inputting thyroid cell pathology slice images; A cell detection module is used to detect target cells through a neural network model; The semantic classification module is used to semantically classify the target cells and obtain classification labels; The knowledge graph search module is used to search in the knowledge graph according to classification labels and provide textual interpretation of the obtained diagnostic results.

2. The thyroid pathology image-assisted analysis system according to claim 1, characterized in that: The neural network model is selected from the Cascade-R-CNN model.

3. The thyroid pathology image-assisted analysis system according to claim 1, characterized in that: The target cells are thyroid papillary cancer cells, multinuclear macrophages, normal thyroid cells or other background cells.

4. The thyroid pathology image-assisted analysis system according to claim 1, characterized in that: The semantic classification is implemented using GINet, a graph reasoning method based on semantic features.

5. The thyroid pathology image-assisted analysis system according to claim 1, characterized in that: The classification labels include at least one of the following labels: ground glass nucleus, nuclear groove, intranuclear pseudoinclusion, and multinucleated macrophage.

6. The thyroid pathology image-assisted analysis system according to claim 1, characterized in that: The method for constructing the knowledge graph includes: Step 1, search for literature related to thyroid pathology images; Step 2: Extract entities and relations from the document; Step 3: Perform knowledge fusion to solve entity disambiguation and coreference disambiguation problems of equivalent instances, equivalent classes / subclasses, and equivalent attributes / subattributes; Step 4: Use Neo4j graph database for storage.

7. The thyroid pathology image-assisted analysis system according to claim 6, characterized in that: In step 2, pytorch's spaCy library is used for entity extraction and relationship extraction.

8. The thyroid pathology image-assisted analysis system according to claim 6, characterized in that: In step 4, the storage structure consists of four elements: labels, nodes, relationships, and attributes.

9. A computer-readable storage medium, characterized in that: Stored thereon is: a computer program for implementing the thyroid pathology image-assisted analysis system according to any one of claims 1-8.

Images