An automatic annotation method and system suitable for multiple types of pathological images

Automatic labeling of multi-type pathological images is performed through deep learning models and cluster analysis algorithms, and the doctor review module is introduced, which solves the accuracy and workload of automatic labeling of multi-type pathological images in the existing technology, achieving efficient and accurate Mark the results.

CN113793307BActive Publication Date: 2024-11-08SHANGHAI PAIYING MEDICAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110965979.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-23
Publication Date
2024-11-08
Estimated Expiration
2041-08-23

AI Technical Summary

Technical Problem

The prior art is difficult to realize automatic labeling of multiple types of pathological images, and the automatic labeling system lacks the doctor's review function, resulting in labeling accuracy and workload problems.

Method used

An automatic labeling method for multi-type pathological images is designed, using deep learning models to classify images and label tumor areas, and a cluster analysis algorithm is used to generate XML vector annotations, and a doctor review module is introduced to ensure the accuracy of the labeling.

Benefits of technology

It realizes efficient automatic labeling of multiple types of pathological images, reduces the workload of pathologists, and ensures the accuracy of labeling results through doctor review, adapting to the actual hospital process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113793307B_ABST
    Figure CN113793307B_ABST
Patent Text Reader

Abstract

The present invention provides an automatic annotation method and system applicable to multiple types of pathological images, including classifying and identifying uploaded pathological images through a trained deep learning model to obtain the classification results of the pathological images, the position coordinates of each classification, and the corresponding probabilities of each classification, and automatically annotating the part identified as the tumor area to generate an annotation result, and displaying the annotation result of the tumor area in the pathological image. If the pathologist believes that the annotation result is wrong, the annotation result is subjected to operations including adding annotations, modifying annotations, and deleting annotations and then re-annotating. The pathologist gives an audit result and a diagnosis opinion for the annotation result. The present invention provides an XML vector annotation of the pathological image based on an automatic annotation algorithm for pathological images based on cluster analysis, designs a doctor review function, ensures the accuracy of automatic annotation, and combines it with the actual process of the hospital to alleviate the workload of the pathologist.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of medical image diagnosis, and in particular to an automatic annotation method and system suitable for multiple types of pathological images. Background Art

[0002] In recent years, with the continuous development of artificial intelligence and deep learning technology, computer-assisted diagnosis and treatment has become the focus of medical imaging research. The tissue pathology image-assisted diagnosis method based on deep learning can speed up the diagnostic efficiency of pathologists, alleviate the scarcity of pathologists, the long training cycle, and the general overload of pathology departments. Pathology image diagnosis based on deep learning requires the use of a large number of pathology data sets, but the size of the full-slice image is huge, and the annotation work is very time-consuming, resulting in fewer pixel-level annotated data sets in the field of pathology images. Most of the existing annotation systems are manually annotated by pathologists, which is time-consuming and labor-intensive, and is prone to mislabeling and missing labels. However, the tissue pathology-assisted diagnosis method based on deep learning often has low recognition accuracy, no XML vector annotation is given, and it cannot be used as a deep learning data set, and one system only corresponds to one disease. Problems. Therefore, the present invention designs an automatic annotation system suitable for multiple types of pathological images, realizes automatic annotation of tumor areas in multiple types of pathological images, and includes a pathology expert review module, so that it can reduce the workload of pathology experts while ensuring the accuracy of annotation.

[0003] In recent years, many scholars have conducted research on the recognition of pathological images. For example, Sun Y et al. proposed a system based on deep learning that can automatically identify tumor areas in histopathological images. Diao S et al. proposed a fully automatic cancer region recognition framework for computer-aided diagnosis of pathological WSI based on deep convolutional neural networks. Wang X et al. proposed a weakly supervised learning method to solve the classification problem of full-slice lung cancer images, and can also effectively predict the probability of cancer and detect suspected cancer areas. However, none of the above methods provide XML vector map annotations and cannot provide data sets for deep learning.

[0004] At present, there are also some related technologies to solve the above problems. For example, Chinese patent CN105608319B provides a method for annotating digital pathological images, but this method requires the user to manually select the location of the annotation point and the type of annotation graphic. Chinese patent CN110659692A realizes automatic annotation of pathological images, but it does not include a doctor review function and cannot guarantee the accuracy of automatic annotation. Chinese patent CN110826560A provides a method for annotating pathological images of esophageal cancer, but the invention only corresponds to a method for annotating one disease. Summary of the invention

[0005] The present invention proposes an automatic annotation method and system applicable to multiple types of pathological images to solve the defects of the prior art mentioned above.

[0006] In one aspect, the present invention provides an automatic annotation method applicable to multiple types of pathological images, the method comprising the following steps:

[0007] S1: The pathologist uploads the pathological image, edits the relevant information of the uploaded pathological image, and checks, adds, modifies, and deletes the uploaded pathological image information, the disease type corresponding to the uploaded pathological image, and the patient information corresponding to the uploaded pathological image;

[0008] S2: According to the different disease types displayed in the uploaded pathological image, the uploaded pathological image is sent to the trained deep learning model corresponding to the disease type for identification, and the classification result of the uploaded pathological image, the position coordinates of each classification and the probability corresponding to each classification are obtained, and the part identified as the tumor area is automatically annotated to generate an annotation result, and the annotation result of the tumor area is displayed in the pathological image, and the annotation result is returned to the pathology expert;

[0009] S3: The pathologist reviews the annotation results. If the pathologist believes that the annotation results are incorrect, the pathologist performs operations including adding annotations, modifying annotations, and deleting annotations and then re-annotating the annotation results. The pathologist gives a review result and a diagnosis opinion based on the annotation results.

[0010] The above method automatically labels normal and pathological multi-system type pathological images in binary classification, sets the number of data sets in each category to be basically the same, and proposes an automatic labeling algorithm for pathological images based on cluster analysis, gives XML vector map labeling, designs a doctor review function, ensures the accuracy of automatic labeling, and combines it with the actual process of the hospital to alleviate the workload of pathologists.

[0011] In a specific embodiment, according to the different disease types displayed in the uploaded pathological images, the uploaded pathological images are sent to the trained deep learning model corresponding to the disease type for identification, the classification results of the uploaded pathological images, the position coordinates of each classification and the probability corresponding to each classification are obtained, and the part identified as the tumor area is automatically annotated to generate the annotation results, which specifically includes:

[0012] Using a density-based clustering algorithm, the blocks of the pathological image are classified and identified using a trained deep learning model to obtain classification results and position coordinates of each classification, clustering is performed based on the classification results and the position coordinates of each classification, and the probability mean, standard deviation and variance of each classification are calculated;

[0013] When the probability mean, standard deviation, and variance of a classification are greater than a certain threshold, the classification is retained; otherwise, the classification is deleted.

[0014] In a specific embodiment, the labeling result of displaying the tumor area in the pathological image includes:

[0015] The edge points of each classification are obtained by a convex hull algorithm, and the annotation results of the tumor area in the pathological image are displayed in the form of a polygonal frame according to the edge points.

[0016] In a specific embodiment, the method further comprises S4:

[0017] Manage the information of pathologists, query, add, modify and delete the information of pathologists, including name, account number, password, mobile phone number, hospital and department.

[0018] In a specific embodiment, the method further comprises S5:

[0019] View the pathological image information and its corresponding patient information, and download the annotation results of the reviewed pathological images.

[0020] In a specific embodiment, the labeling result of displaying the tumor area in the pathological image further includes:

[0021] The OpenSeadragon image control is used to support the DZI format image display technology, realize the basic display function of pathological images, and freely scale and move according to the needs of users. At the same time, the annotation results are scaled and moved synchronously with the needs of users. The pathological images displayed in this way can be freely scaled and moved according to the needs of users, and the zoom multiple can reach 60 times, which is convenient for annotation experts to carefully view the image information in the pathological images.

[0022] In a specific embodiment, before the uploaded pathological image is sent to the trained deep learning model corresponding to the disease type for identification, the uploaded pathological image is segmented into several small cell tissue area images using an image mask method, and then the several small cell tissue area images cut out are input into the trained deep learning model for classification and identification.

[0023] In a specific embodiment, the annotation result includes XML vector graphics annotation, specifically including location information of the annotation area and classification information of the lesion type.

[0024] In a specific embodiment, the operations of adding annotations, modifying annotations, and deleting annotations and then re-annotating the annotations in the annotation results specifically include:

[0025] Create a new annotation, delete an annotation, modify the annotation shape, modify the annotation type, and save the annotation on the annotation result;

[0026] The newly created annotation includes adding a regional annotation outline to the tumor region that is missed in S2;

[0027] The deleting of the annotations includes deleting the incorrect annotations in S2 on the annotation list, and simultaneously deleting the incorrect annotations in S2 on the annotation display area;

[0028] Modifying the annotation includes modifying the area outline of the annotation in the display area of ​​the original annotation, and also includes modifying the classification information of the annotation in the annotation list;

[0029] The saving of annotations includes submitting and saving all current annotation information to a database.

[0030] In a specific embodiment, the annotation result of the tumor area displayed in the pathological image is drawn based on drawing forms including pencil drawing, polygon drawing, rectangle drawing and circle drawing, and is drawn according to the movement position of the mouse based on paper.js technology.

[0031] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer processor, the above method is implemented.

[0032] According to a third aspect of the present invention, an automatic annotation system applicable to multiple types of pathological images is proposed, the system comprising:

[0033] Pathological image management module: configured for pathological experts to upload pathological images, edit relevant information of uploaded pathological images, and view, add, modify, and delete the uploaded pathological image information, the disease type corresponding to the uploaded pathological image, and the patient information corresponding to the uploaded pathological image;

[0034] Pathological image automatic annotation module: configured to send the uploaded pathological image to a trained deep learning model corresponding to the disease type for identification according to the different disease types displayed in the uploaded pathological image, obtain the classification result of the uploaded pathological image, the position coordinates of each classification and the probability corresponding to each classification, and automatically annotate the part identified as the tumor area to generate an annotation result, and display the annotation result of the tumor area in the pathological image, and return the annotation result to the pathology expert;

[0035] Pathology image annotation review module: configured for pathology experts to review the annotation results. If they believe that the annotation results are incorrect, they will perform operations on the annotation results, including adding annotations, modifying annotations, and deleting annotations and then re-annotating. The pathology experts will give review results and diagnostic opinions based on the annotation results.

[0036] The present invention uses a trained deep learning model to identify the uploaded pathological images to obtain the classification results of the pathological images, the position coordinates of each classification, and the corresponding probabilities of each classification, and automatically annotates the part identified as the tumor area to generate an annotation result, and displays the annotation result of the tumor area in the pathological image, and returns the annotation result to the pathology expert. If the pathology expert believes that the annotation result is wrong, the annotation result is subjected to operations including adding annotations, modifying annotations, and deleting annotations and then re-annotating. The pathology expert gives the review result and diagnosis opinion for the annotation result. The present invention performs binary automatic annotation of normal and lesion multi-system type pathological images, sets the number of each type of data set to be basically consistent, and proposes an automatic annotation algorithm for pathological images based on cluster analysis, provides XML vector map annotation, designs a doctor review function, ensures the accuracy of automatic annotation, and combines it with the actual process of the hospital, alleviating the workload of pathology experts. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and together with the description are used to explain the principles of the present invention. Other embodiments and many expected advantages of the embodiments will be readily appreciated as they become better understood by reference to the following detailed description. Other features, objects and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings:

[0038] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0039] Figure 2 is a flow chart of an automatic annotation method applicable to multiple types of pathological images according to an embodiment of the present invention;

[0040] Figure 3 It is a block diagram of the overall modules of a system suitable for automatic annotation of multiple types of pathological images according to a specific embodiment of the present invention;

[0041] Figure 4 It is a flow chart of an expert module applicable to automatic annotation of multiple types of pathological images according to a specific embodiment of the present invention;

[0042] Figure 5is a framework diagram of an automatic annotation system applicable to multiple types of pathological images according to an embodiment of the present invention;

[0043] Figure 6 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0044] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0045] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] Figure 1 An exemplary system architecture 100 to which an automatic annotation method for multiple types of pathological images according to an embodiment of the present application can be applied is shown.

[0047] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0048] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various applications can be installed on terminal devices 101, 102, 103, such as data processing applications, data visualization applications, web browser applications, etc.

[0049] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitation is made here.

[0050] The server 105 may be a server that provides various services, such as a background information processing server that supports pathological images displayed on the terminal devices 101, 102, and 103. The background information processing server may process the acquired classification results and generate processing results (such as XML vector graphics annotation).

[0051] It should be noted that the method provided in the embodiment of the present application can be executed by the server 105, and can also be executed by the terminal devices 101, 102, 103. The corresponding device is generally set in the server 105, and can also be set in the terminal devices 101, 102, 103.

[0052] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0053] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0054] According to an embodiment of the present invention, an automatic annotation method applicable to multiple types of pathological images is provided. Figure 2 FIG. 4 is a flowchart of an automatic annotation method for multi-type pathological images according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0055] S201: The pathology expert uploads a pathology image, edits the relevant information of the uploaded pathology image, and checks, adds, modifies, and deletes the uploaded pathology image information, the disease type corresponding to the uploaded pathology image, and the patient information corresponding to the uploaded pathology image.

[0056] S202: According to the different disease types displayed in the uploaded pathological image, the uploaded pathological image is sent to the trained deep learning model corresponding to the disease type for identification, and the classification result of the uploaded pathological image, the position coordinates of each classification and the corresponding probability of each classification are obtained, and the part identified as the tumor area is automatically annotated to generate an annotation result, and the annotation result of the tumor area is displayed in the pathological image, and the annotation result is returned to the pathology expert.

[0057] In a specific embodiment, according to the different disease types displayed in the uploaded pathological images, the uploaded pathological images are sent to the trained deep learning model corresponding to the disease type for identification, the classification results of the uploaded pathological images, the position coordinates of each classification and the probability corresponding to each classification are obtained, and the part identified as the tumor area is automatically annotated to generate the annotation results, which specifically includes:

[0058] Using a density-based clustering algorithm, the blocks of the pathological image are identified using a trained deep learning model to obtain classification results and position coordinates of each classification, clustering is performed based on the classification results and the position coordinates of each classification, and the probability mean, standard deviation and variance of each classification are calculated;

[0059] When the probability mean, standard deviation, and variance of a classification are greater than a certain threshold, the classification is retained; otherwise, the classification is deleted.

[0060] In a specific embodiment, the labeling result of displaying the tumor area in the pathological image includes:

[0061] The edge points of each classification are obtained by a convex hull algorithm, and the annotation results of the tumor area in the pathological image are displayed in the form of a polygonal frame according to the edge points.

[0062] In a specific embodiment, the labeling result of displaying the tumor area in the pathological image further includes:

[0063] The OpenSeadragon image control is used to support the DZI format image display technology, realize the basic display function of the pathological image, and freely scale and move according to the needs of the user. At the same time, the annotation results are scaled and moved synchronously with the needs of the user.

[0064] In a specific embodiment, before the uploaded pathological image is sent to the trained deep learning model corresponding to the disease type for identification, the uploaded pathological image is segmented into several small cell tissue area images using an image mask method, and then the several small cell tissue area images cut out are input into the trained deep learning model for classification and identification.

[0065] In a specific embodiment, the annotation result includes XML vector graphics annotation, specifically including location information of the annotation area and classification information of the lesion type.

[0066] In a specific embodiment, the annotation result of the tumor area displayed in the pathological image is drawn based on drawing forms including pencil drawing, polygon drawing, rectangle drawing and circle drawing, and is drawn according to the movement position of the mouse based on paper.js technology.

[0067] S203: The pathologist reviews the annotation results. If the pathologist believes that the annotation results are incorrect, the pathologist performs operations including adding annotations, modifying annotations, and deleting annotations and then re-annotating the annotation results. The pathologist gives a review result and a diagnosis opinion on the annotation results.

[0068] In a specific embodiment, the operations of adding annotations, modifying annotations, and deleting annotations and then re-annotating the annotations in the annotation results specifically include:

[0069] Create a new annotation, delete an annotation, modify the annotation shape, modify the annotation type, and save the annotation on the annotation result;

[0070] The newly creating annotation includes adding a regional annotation outline to the tumor region that is missed in S202;

[0071] The deleting of the annotations includes deleting the incorrect annotations in S202 on the annotation list, and simultaneously deleting the incorrect annotations in S202 on the annotation display area;

[0072] Modifying the annotation includes modifying the area outline of the annotation in the display area of ​​the original annotation, and also includes modifying the classification information of the annotation in the annotation list;

[0073] The saving of annotations includes submitting and saving all current annotation information to a database.

[0074] In a specific embodiment, the method further includes S204:

[0075] Manage the information of pathologists, query, add, modify and delete the information of pathologists, including name, account number, password, mobile phone number, hospital and department.

[0076] In a specific embodiment, the method further includes S205:

[0077] View the pathological image information and its corresponding patient information, and download the annotation results of the reviewed pathological images.

[0078] Figure 3 The overall module block diagram of the system applicable to automatic annotation of multiple types of pathological images according to a specific embodiment of the present invention is shown below. Figure 3 The illustrated system is used to introduce an embodiment of the present invention, thereby further illustrating the present invention.

[0079] It consists of two modules: expert module and administrator module;

[0080] Figure 4 A flowchart of an expert module applicable to automatic annotation of multiple types of pathological images according to a specific embodiment of the present invention is shown;

[0081] The expert module is used by experts to upload and manage pathological images and patient information, view and review the tumor areas automatically annotated by AI in pathological images, and add, modify, delete and save the annotated results;

[0082] The administrator module is used for the administrator to manage expert information, view pathological image information and patient information, and download annotation files reviewed by experts;

[0083] Furthermore, the expert module includes an image list display for displaying pathological image information and its corresponding patient information.

[0084] Furthermore, the expert module includes a pathology image uploading function, and the expert uploads the pathology image to the backend server for storage.

[0085] Furthermore, the expert module includes pathological image information management, and experts can add patient information corresponding to the pathological image, as well as the disease type corresponding to the pathological image, for subsequent automatic annotation of multiple types of pathological images.

[0086] Furthermore, the expert module includes image display, which uses OpenSeadragon image controls to support DZI format image display technology and realizes basic display functions of pathological images. Images can be freely scaled and moved according to user needs, and the scaling factor can reach sixty times, making it convenient for annotation experts to carefully view the image information in pathological images.

[0087] Furthermore, the expert module includes automatic annotation of pathological images, which mainly includes the following steps:

[0088] (1) Input a pathological image and use the image mask method to slide and cut it to obtain several 512*512 cell tissue regions;

[0089] (2) inputting the cut cell tissue area image into the designed convolutional neural network recognition model for recognition, and identifying the relevant coordinates of the image and its probability of canceration;

[0090] (3) using a density-based clustering algorithm and clustering the coordinates of the small images judged as positive;

[0091] (4) After clustering, calculate the average probability, standard deviation, and variance of each cluster, set relevant thresholds, filter the clusters, and delete clusters that do not meet the requirements;

[0092] (5) Use the Graham-Scan convex hull algorithm to obtain the edge points of each cluster, and connect the edge points one by one to obtain the annotation of the tumor area in the pathological image.

[0093] Furthermore, the expert module includes annotation display and operation, that is, on the basis of the display of the pathological image, the corresponding annotation display is guaranteed. The annotation information includes the location information of the annotation area and the classification information of the lesion type. During the image zooming and moving process, the annotation information will also be zoomed and moved accordingly. At the same time, the annotation operation includes creating new annotations, deleting annotations, modifying annotation shapes, modifying annotation types, saving annotations and other operations. Creating new annotations can add regional annotation outlines to the tumor area that AI missed. This system provides four drawing forms (pencil drawing, polygon drawing, rectangle drawing and circle drawing). The implementation of the annotation tool is mainly based on the movement position of the mouse, and the drawing is mainly supported by paper.js technology. Deleting annotations allows experts to delete incorrect AI annotations in the annotation list, and correspondingly delete them in the annotation display area; modifying annotations can modify the outline of the annotation area in the original annotation display area, and can also modify the classification information of the annotation in the annotation list; saving annotations is to submit all current annotation information to the database.

[0094] Furthermore, the expert module includes writing and submitting review opinions and saving them to a database.

[0095] Furthermore, the administrator module includes an expert management module for adding, deleting, and modifying expert account passwords and personal information.

[0096] Furthermore, the administrator module includes an image management module for viewing the image number, image type and patient information corresponding to the pathological image. The image management module also includes a function for downloading the annotated files reviewed by experts.

[0097] Furthermore, the B / S architecture is adopted, which is suitable for running on low-end and medium-end computers. The automatic annotation module and expert review module are introduced, which not only reduces the workload of experts, but also makes the annotation results more accurate.

[0098] Figure 5 The framework diagram of an automatic annotation system applicable to multiple types of pathological images according to an embodiment of the present invention is shown. The system comprises a pathological image management module 501 , a pathological image automatic annotation module 502 , and a pathological image annotation review module 503 .

[0099] In a specific embodiment, the pathological image management module 501 is configured to allow a pathological expert to upload a pathological image, edit relevant information of an uploaded pathological image, and view, add, modify, and delete the uploaded pathological image information, the disease type corresponding to the uploaded pathological image, and the patient information corresponding to the uploaded pathological image;

[0100] The pathological image automatic annotation module 502 is configured to send the uploaded pathological image to a trained deep learning model corresponding to the disease type for identification according to the different disease types displayed in the uploaded pathological image, obtain the classification result of the uploaded pathological image, the position coordinates of each classification and the probability corresponding to each classification, and automatically annotate the part identified as the tumor area to generate an annotation result, and display the annotation result of the tumor area in the pathological image, and return the annotation result to the pathology expert;

[0101] The pathology image annotation review module 503 is configured for the pathology expert to review the annotation results. If the annotation results are considered to be incorrect, the pathology expert will perform operations including adding annotations, modifying annotations, and deleting annotations and then re-annotating the annotation results. The pathology expert will give an audit result and diagnosis opinion based on the annotation results.

[0102] This system provides XML vector annotations of pathological images based on an automatic annotation algorithm for pathological images based on cluster analysis, designs a doctor review function to ensure the accuracy of automatic annotation, and combines it with the actual process of the hospital to alleviate the workload of pathology experts.

[0103] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 suitable for implementing an electronic device of an embodiment of the present application. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0104] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0105] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0106] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0107] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The modules involved in the embodiments described in the present application may be implemented by software or hardware. The units described may also be arranged in a processor, and the names of these units do not constitute limitations on the units themselves in certain circumstances.

[0110] An embodiment of the present invention further relates to a computer-readable storage medium having a computer program stored thereon, which implements the method described above when the computer program is executed by a computer processor. The computer program includes program code for executing the method shown in the flowchart. It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable medium or any combination of the above two.

[0111] The present invention provides an automatic annotation method and system applicable to multiple types of pathological images, including classifying and identifying uploaded pathological images through a trained deep learning model to obtain the classification results of the pathological images, the position coordinates of each classification, and the corresponding probabilities of each classification, and automatically annotating the part identified as the tumor area to generate an annotation result, and displaying the annotation result of the tumor area in the pathological image. If the pathologist believes that the annotation result is wrong, the annotation result is subjected to operations including adding annotations, modifying annotations, and deleting annotations and then re-annotating. The pathologist gives an audit result and a diagnosis opinion for the annotation result. The present invention provides an XML vector annotation of the pathological image based on an automatic annotation algorithm for pathological images based on cluster analysis, designs a doctor review function, ensures the accuracy of automatic annotation, and combines it with the actual process of the hospital to alleviate the workload of the pathologist.

[0112] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. An automatic annotation method suitable for multi-type pathological images, characterized in that: The following steps are involved: S1: The pathologist uploads the pathological image, edits the relevant information of the uploaded pathological image, and checks, adds, modifies, and deletes the uploaded pathological image information, the disease type corresponding to the uploaded pathological image, and the patient information corresponding to the uploaded pathological image; S2: According to the different disease types displayed in the uploaded pathological images, the uploaded pathological images are sent to the trained deep learning model corresponding to the disease types for identification, and the classification results of the uploaded pathological images, the position coordinates of each category and the corresponding probabilities of each category are obtained. Through the density-based clustering algorithm, the blocks of the pathological images are classified and identified using the trained deep learning model to obtain the classification results and the position coordinates of each category. Clustering is performed according to the classification results and the position coordinates of each category, and the probability mean, standard deviation and variance of each category are calculated; when the probability mean, standard deviation and variance of a category are greater than a certain threshold, the category is retained, otherwise, the category is deleted; and the part identified as the tumor area is automatically annotated to generate an annotation result, and the annotation result of the tumor area is displayed in the pathological image, and the annotation result is returned to the pathology expert; S3: The pathologist reviews the annotation results. If the pathologist believes that the annotation results are incorrect, the pathologist performs operations including adding annotations, modifying annotations, and deleting annotations and then re-annotating the annotation results. The pathologist gives a review result and a diagnosis opinion based on the annotation results.

2. The method according to claim 1, characterized in that: The marking result of displaying the tumor area in the pathological image includes: The edge points of each classification are obtained by a convex hull algorithm, and the annotation results of the tumor area in the pathological image are displayed in the form of a polygonal frame according to the edge points.

3. The method according to claim 1, characterized in that The method further comprises S4: Manage the information of pathologists, query, add, modify and delete the information of pathologists, including name, account number, password, mobile phone number, hospital and department.

4. The method according to claim 1, characterized in that: The method further comprises S5: View the pathological image information and its corresponding patient information, and download the annotation results of the reviewed pathological images.

5. The method according to claim 1, characterized in that The marking result of displaying the tumor area in the pathological image also includes: The OpenSead ragon image control is used to support the DZI format image display technology, realize the basic display function of pathological images, and freely scale and move according to the needs of users. At the same time, the annotation results are scaled and moved synchronously with the needs of users.

6. The method according to claim 1, characterized in that Before the uploaded pathological image is sent to the trained deep learning model corresponding to the disease type for identification, the uploaded pathological image is segmented into several small cell tissue area images using an image mask method, and then the several small cell tissue area images cut out are input into the trained deep learning model for classification and identification.

7. The method according to claim 1, characterized in that The annotation result includes XML vector graphics annotation, specifically including location information of the annotation area and classification information of the lesion type.

8. The method according to claim 1, characterized in that The operations of adding annotations, modifying annotations, and deleting annotations and then re-annotating the annotations in the annotation results specifically include: Create a new annotation, delete an annotation, modify the annotation shape, modify the annotation type, and save the annotation on the annotation result; The newly created annotation includes adding a regional annotation outline to the tumor region that is missed in S2; The deleting of the annotations includes deleting the incorrect annotations in S2 on the annotation list, and simultaneously deleting the incorrect annotations in S2 on the annotation display area; Modifying the annotation includes modifying the area outline of the annotation in the display area of ​​the original annotation, and also includes modifying the classification information of the annotation in the annotation list; The saving of annotations includes submitting and saving all current annotation information to a database.

9. The method according to claim 1, characterized in that: The annotation result of displaying the tumor area in the pathological image is drawn based on drawing forms including pencil drawing, polygon drawing, rectangle drawing and circle drawing, and is drawn based on the pape rjs technology according to the moving position of the mouse.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer processor, the method according to any one of claims 1 to 9 is implemented.

11. An automatic annotation system suitable for multi-type pathological images, characterized in that: include: Pathological image management module: configured for pathological experts to upload pathological images, edit relevant information of uploaded pathological images, and view, add, modify, and delete the uploaded pathological image information, the disease type corresponding to the uploaded pathological image, and the patient information corresponding to the uploaded pathological image; Pathological image automatic annotation module: configured to send the uploaded pathological image to a trained deep learning model corresponding to the disease type for identification according to the different disease types displayed in the uploaded pathological image, obtain the classification result of the uploaded pathological image, the position coordinates of each category and the probability corresponding to each category, classify and identify the blocks of the pathological image using the trained deep learning model through a density-based clustering algorithm to obtain the classification result and the position coordinates of each category, cluster according to the classification result and the position coordinates of each category, and calculate the probability mean, standard deviation and variance of each category; when the probability mean, standard deviation and variance of a category are greater than a certain threshold, the category is retained, otherwise, the category is deleted; and the part identified as the tumor area is automatically annotated to generate an annotation result, and the annotation result of the tumor area is displayed in the pathological image, and the annotation result is returned to the pathology expert; Pathology image annotation review module: configured for pathology experts to review the annotation results. If they believe that the annotation results are incorrect, they will perform operations on the annotation results, including adding annotations, modifying annotations, and deleting annotations and then re-annotating. The pathology experts will give review results and diagnostic opinions based on the annotation results.