Weakly supervised pathological image segmentation system and method based on frame-level annotation

Abstract

The present invention provides a weakly supervised pathological image segmentation system and method based on frame-level annotation, wherein the system includes: a pathological slice acquisition subsystem for acquiring target pathological slices; a frame-level annotation subsystem for sending the target pathological slices to a preset annotation node for frame-level annotation and obtaining an annotated image replied by the annotation node; and a segmentation subsystem for performing weakly supervised semantic segmentation on the annotated image based on the WSSS algorithm to obtain a segmentation result. The weakly supervised pathological image segmentation system and method based on frame-level annotation of the present invention sends the acquired target pathological slices to the annotation node for frame-level annotation, obtains an annotated image replied by the annotation node, introduces the WSSS algorithm to perform weakly supervised semantic segmentation on the annotated image to obtain a segmentation result, achieves accurate segmentation of pathological images with less annotation information, reduces annotation costs, and improves the efficiency of assisting doctors in diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a weakly supervised pathological image segmentation system and method based on frame-level annotation. Background Art

[0002] Pathology images are generated by scanning tissue sections at high magnification using a microscope. They offer extremely high resolution and rich cellular structural information. Intelligent pathology applies artificial intelligence (AI) to the pathology diagnosis process to improve the efficiency and accuracy of pathologists. Currently, mainstream intelligent pathology methods include deep learning-based image classification, object detection, and semantic segmentation, such as convolutional neural networks (CNNs), fully convolutional networks (FCNs), and U-Nets. Semantic segmentation is a computer vision task that aims to assign a class label to each pixel in an image. These methods have achieved remarkable results in pathology image diagnosis.

[0003] The invention patent with application number: CN202110621825.5 discloses a pathological image segmentation method based on domain adversarial self-supervised learning, wherein the method includes: obtaining pathological images to establish a pathological image self-supervised dataset; establishing a domain adversarial self-supervised model; using the pathological image self-supervised dataset to perform deep learning training on the domain adversarial self-supervised model; establishing a pathological image segmentation model; using the domain adversarial self-supervised model after deep learning to initialize the pathological image segmentation model; performing pixel-level annotation on the lesion area in the pathological image to establish a pathological image segmentation dataset; using the pathological image segmentation dataset to perform deep learning training on the pathological image segmentation model; using the pathological image segmentation model after deep learning training to segment the unknown lesion area of ​​the pathological image. The above invention adopts the method of domain adversarial self-supervised learning, which effectively alleviates the segmentation model's dependence on a large amount of manual annotation and solves the problem of fluctuations in the model's segmentation performance in different domains.

[0004] However, the above existing technologies still require pixel-level labeled data as training sets, which is time-consuming and costly. Furthermore, the efficiency of assisting doctors in diagnosis is also low.

[0005] In view of this, there is an urgent need for a high-frequency quantitative trading method and system based on deep learning to at least solve the above-mentioned shortcomings. Summary of the Invention

[0006] One of the purposes of the present invention is to provide a weakly supervised pathological image segmentation system based on frame-level annotation, in which the collected target pathological slices are sent to the annotation node for frame-level annotation, the annotated image replied by the annotation node is obtained, and the WSSS algorithm is introduced to perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result. The accurate segmentation of the pathological image is achieved with less annotation information, the annotation cost is reduced, and the efficiency of assisting doctors in diagnosis is improved.

[0007] The weakly supervised pathological image segmentation system based on frame-level annotation provided by the embodiment of the present invention includes:

[0008] Pathology section acquisition subsystem, used to acquire target pathology sections;

[0009] The frame-level annotation subsystem is used to send the target pathological slice to the preset annotation node for frame-level annotation and obtain the annotated image replied by the annotation node;

[0010] The segmentation subsystem is used to perform weakly supervised semantic segmentation on the annotated image based on the WSSS algorithm to obtain the segmentation results.

[0011] Preferably, the frame-level annotation subsystem includes:

[0012] Node attribute acquisition module, used to obtain node attributes of labeled nodes;

[0013] A node type determination module is used to determine the node type of the labeled node based on the node attributes;

[0014] A first sub-annotated image acquisition module, configured to acquire a first sub-annotated image if the node type is a manually annotated node;

[0015] A second sub-annotation image acquisition module, configured to acquire a second sub-annotation image if the node type is an intelligent annotation node;

[0016] The annotated image integration module is configured to use the first sub-annotated image and the second sub-annotated image together as an annotated image.

[0017] Preferably, the second sub-annotated image acquisition module includes:

[0018] A feature of interest extraction submodule is used to extract the features of interest of the target pathological section based on a preset feature of interest extraction template;

[0019] The object bounding box annotation model training submodule is used to train the object bounding box annotation model based on the object detection algorithm and the preset deep learning model according to the features of interest;

[0020] The target bounding box annotation submodule is used to annotate the target bounding box in the target pathological section according to the target bounding box annotation model;

[0021] The key point annotation template determination submodule is used to determine the key point annotation template according to the feature type of the feature of interest corresponding to the target bounding box;

[0022] The key point annotation submodule is used to annotate key points within the target bounding box according to the key point annotation template;

[0023] The pre-verification image determination submodule is used to determine the corresponding target pathological slice as the pre-verification image after all target bounding boxes and key points are labeled;

[0024] The second sub-annotated image determination submodule is used to send the pre-verification image to a preset verification personnel node and obtain the second sub-annotated image verified by the verification personnel node.

[0025] Preferably, the segmentation subsystem includes:

[0026] A first pseudo pixel-level label generation module, configured to generate a first pseudo pixel-level label based on the annotation information of the annotated image;

[0027] An initial segmentation result acquisition module is used to perform initial segmentation on the target pathological image based on the image region proposal technology and the first pseudo pixel-level label to obtain an initial segmentation result;

[0028] The preprocessing segmentation result acquisition module is used to train and semantically segment the labeled image based on the WSSS algorithm according to the preset full convolutional network and the first pseudo pixel-level label to obtain the preprocessing segmentation result;

[0029] The processing module is used to process the preprocessing segmentation results to obtain the segmentation results.

[0030] Preferably, the pre-processing segmentation result acquisition module includes:

[0031] A fully annotated image acquisition submodule is used to acquire a fully annotated image and, at the same time, acquire a second pseudo pixel-level label of the fully annotated image;

[0032] A cross entropy loss function acquisition submodule is used to train a fully convolutional network based on the fully annotated image and the second pseudo pixel-level label corresponding to the fully annotated image based on the WSSS algorithm, and to obtain a cross entropy loss function during the training process of the fully convolutional network;

[0033] A prediction difference determination submodule is used to determine the prediction difference based on the cross entropy loss function;

[0034] The network parameter update submodule is used to update the network parameters of the trained fully convolutional network in real time based on the prediction difference based on the back-propagation algorithm;

[0035] The target full convolutional network acquisition submodule is used to obtain the current full convolutional network and use it as the target full convolutional network when the predicted difference is less than or equal to the preset difference threshold;

[0036] The initial segmentation result determination submodule is used to input the first pseudo pixel-level label into the target fully convolutional network to obtain the initial segmentation result.

[0037] The weakly supervised pathological image segmentation system based on frame-level annotation provided by an embodiment of the present invention further includes:

[0038] The report generation subsystem is used to obtain input information from the target display party after viewing the segmentation results, and generate a pathology diagnosis report based on the input information;

[0039] The report generation subsystem includes:

[0040] An input template library acquisition module is used to acquire an input template library;

[0041] A recommended input template list determination module is used to determine a recommended input template list based on the segmentation results and the input template library;

[0042] A target input template acquisition module is used to acquire a target input template selected by a target display party from a recommended input template list, and to display the target input template to the target display party;

[0043] An operation information acquisition module is used to obtain the operation information of each operable item in the target input template of the target display party;

[0044] An input item determination module is used to determine an input item in the operable item based on an operation instruction library and operation information corresponding to the operable item;

[0045] The input information acquisition module is used to integrate input items and target input templates to obtain input information.

[0046] Preferably, the recommended input template list determination module includes:

[0047] Auxiliary diagnosis model acquisition submodule, used to obtain auxiliary diagnosis models;

[0048] A first auxiliary diagnosis information acquisition submodule is used to input the segmentation result into the auxiliary diagnosis model to obtain first auxiliary diagnosis information;

[0049] The pre-selected input template acquisition submodule is used to parse the input template library and obtain multiple pre-selected input templates;

[0050] The second auxiliary diagnosis information acquisition submodule is used to obtain preset second auxiliary diagnosis information corresponding to the preselected input template;

[0051] An information similarity calculation submodule, configured to calculate the information similarity between the first auxiliary diagnosis information and the second auxiliary diagnosis information;

[0052] The template sorting and obtaining submodule is used to sort the pre-selected input templates according to the similarity of information and obtain the template sorting of the pre-selected input templates;

[0053] The recommended input template list construction submodule is used to construct a recommended input template list based on preset recommended input template list construction rules and according to the template sorting of the pre-selected input templates.

[0054] Preferably, the auxiliary diagnosis model acquisition submodule includes:

[0055] A first process record acquisition unit is used to acquire a first process record of a target doctor viewing the pathological image segmentation information to perform pathological diagnosis;

[0056] a visual dwell time determining unit, configured to determine, based on an attention analysis technique and according to the first procedure record, a visual dwell time of the target doctor in the first segmented area of ​​the segmented image corresponding to the first procedure record;

[0057] A region type acquiring unit, configured to acquire the region type of the first segmented region;

[0058] An analysis importance value determination unit, configured to determine the analysis importance value of a region type according to a preset region type-analysis importance value library;

[0059] an analysis time ideal value determining unit, configured to determine an analysis time ideal value for the first segmented region according to a preset analysis importance value-analysis time ideal value library;

[0060] a reference value calculation unit, configured to divide the visual dwell time of the first segmented area by the ideal analysis time of the first segmented area to obtain a reference value, and associate the reference value with the corresponding first process record;

[0061] a target sum value calculation unit, configured to calculate an average value of the reference values ​​associated with the first process record and use the average value as the target value;

[0062] a second process record determining unit, configured to use the corresponding first process record as the second process record if the target value is greater than or equal to a preset target value threshold;

[0063] The auxiliary diagnosis model training unit is used to train the second process record based on deep learning technology to obtain an auxiliary diagnosis model.

[0064] The embodiment of the present invention provides a weakly supervised pathological image segmentation method based on frame-level annotation, including:

[0065] Step 1: Collect target pathological sections;

[0066] Step 2: Send the target pathological slice to the preset annotation node for frame-level annotation, and obtain the annotated image replied by the annotation node;

[0067] Step 3: Based on the WSSS algorithm, perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result.

[0068] Preferably, step 2: sending the target pathological slice to a preset annotation node for frame-level annotation, and obtaining an annotated image replied by the annotation node, includes:

[0069] Get the node attributes of the annotation node;

[0070] Determine the node type of the labeled node according to the node attributes;

[0071] If the node type is a manually annotated node, obtain the first sub-annotated image;

[0072] If the node type is a smart annotation node, obtain the second sub-annotation image;

[0073] The first sub-annotated image and the second sub-annotated image are used together as the annotated image.

[0074] The beneficial effects of the present invention are:

[0075] The present invention sends the collected target pathological slices to the annotation node for frame-level annotation, obtains the annotated image replied by the annotation node, introduces the WSSS algorithm to perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result, realizes accurate segmentation of pathological images with less annotation information, reduces the annotation cost, and improves the efficiency of assisting doctors in diagnosis.

[0076] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0077] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0079] Figure 1 Schematic diagram of a weakly supervised pathological image segmentation system based on frame-level annotation in an embodiment of the present invention;

[0080] Figure 2Schematic diagram of a weakly supervised pathological image segmentation method based on frame-level annotation in an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0082] The embodiment of the present invention provides a weakly supervised pathological image segmentation system based on frame-level annotation, such as Figure 1 As shown, including:

[0083] The pathology slice acquisition subsystem 1 is used to acquire target pathology slices; wherein the target pathology slices are pathology slices that need to be segmented.

[0084] The frame-level annotation subsystem 2 is used to send the target pathology slice to a preset annotation node for frame-level annotation and obtain the annotated image replied by the annotation node; wherein the preset annotation node is: the communication node of the annotation doctor who annotates the pathology image or the communication node of the model that performs intelligent annotation; frame-level annotation is: marking the position and size of the target object by drawing a rectangular bounding box in the image; the annotated image is: the target pathology slice annotated by the annotation node after the frame-level annotation is completed;

[0085] Segmentation subsystem 3 is configured to perform weakly supervised semantic segmentation on the annotated image based on the WSSS algorithm to obtain a segmentation result. The WSSS (Weakly Supervised Semantic Segmentation) algorithm is an algorithm for weakly supervised semantic segmentation. The segmentation result is an annotated image with a category label annotated with pixel categories. The category label is a pixel region description corresponding to the pixel region in the annotated image. The pixel region description is, for example, what kind of tissue is in what area.

[0086] The working principle and beneficial effects of the above technical solution are:

[0087] This application sends the collected target pathological slices to the annotation node for frame-level annotation, obtains the annotated image replied by the annotation node, introduces the WSSS algorithm to perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result, and achieves accurate segmentation of pathological images with less annotation information, reducing the annotation cost and improving the efficiency of assisting doctors in diagnosis.

[0088] In specific applications, the acquired target pathological image is sent to the annotation node for frame-level annotation. The system automatically performs weakly supervised semantic segmentation on the annotated image to obtain the segmentation result.

[0089] In one embodiment, the frame-level annotation subsystem includes:

[0090] The node attribute acquisition module is used to obtain the node attributes of the labeled node; the node attributes are information describing the properties and characteristics of the node; the properties and characteristics of the node are the corresponding connection parties of the node, such as doctors or models;

[0091] A node type determination module is used to determine the node type of the annotated node based on the node attributes; wherein the node types include: manually annotated nodes and intelligently annotated nodes. Manually annotated nodes are nodes that perform manual frame-level annotation; intelligently annotated nodes are communication nodes of a model that performs artificial intelligence-assisted frame-level annotation.

[0092] A first sub-annotated image acquisition module is configured to acquire a first sub-annotated image if the node type is a manually annotated node; wherein the first sub-annotated image is a frame-level annotated image annotated by the manually annotated node;

[0093] A second sub-annotation image acquisition module is used to acquire a second sub-annotation image if the node type is an intelligent annotation node; wherein the second sub-annotation image is: a frame-level annotation image annotated by the intelligent annotation node;

[0094] The annotated image integration module is configured to use the first sub-annotated image and the second sub-annotated image together as an annotated image.

[0095] The working principle and beneficial effects of the above technical solution are:

[0096] This application introduces node attributes and determines the node type based on them. When the node type is a manually annotated node, a first sub-annotated image is obtained; when the node type is an intelligently annotated node, a second sub-annotated image is obtained. Combining the first and second sub-annotated images as the annotated image improves the comprehensiveness of annotated image acquisition.

[0097] In one embodiment, the second sub-annotation image acquisition module includes:

[0098] The feature of interest extraction submodule is used to extract the features of interest of the target pathological section based on a preset feature of interest extraction template; wherein the preset feature of interest extraction template constrains the extraction of only the features of interest, such as the texture and color of the target pathological section;

[0099] The target bounding box annotation model training submodule is used to train the target bounding box annotation model based on the target detection algorithm and the preset deep learning model according to the features of interest. The target bounding box annotation model is used to replace manual box-level annotation of target pathology images.

[0100] The target bounding box annotation submodule is used to annotate the target bounding box in the target pathological section according to the target bounding box annotation model;

[0101] The key point annotation template determination submodule is used to determine the key point annotation template based on the feature type of the feature of interest corresponding to the target bounding box. The feature type is the feature type of the feature of interest, such as the feature of interest in a gastric pathological section. The key point annotation template is constrained to only annotate the key points of the pathological section corresponding to the feature type, and not annotate other content.

[0102] The key point annotation submodule is used to annotate key points within the target bounding box according to the key point annotation template. The annotated key points are, for example, the locations of cell nuclei, blood vessels, and cell boundaries.

[0103] The pre-verification image determination submodule is used to determine the corresponding target pathological slice as the pre-verification image after all target bounding boxes and key points are labeled;

[0104] The second sub-annotated image determination submodule is used to send the pre-verified image to a preset verification node and obtain the second sub-annotated image that has been verified by the verification node. After the intelligent annotation method completes the bounding box annotation of the region of interest in the pathological image, manual review and correction are necessary to ensure the quality and accuracy of the annotation results. Therefore, the pre-verified image completed with intelligent annotation is sent to the verification personnel. It should be noted that although intelligent annotation still requires manual review, the review and correction work is much more efficient than manual annotation, so it is necessary.

[0105] The working principle and beneficial effects of the above technical solution are:

[0106] In general, in order to ensure the quality and accuracy of the annotation results, pathological images are usually manually annotated at the frame level, but the efficiency of manual annotation is low. At the same time, deep learning technology can be used to learn manual experience to achieve intelligent annotation, but the quality of intelligent annotation depends on the quality of the training data learned. Therefore, this application introduces a feature of interest extraction template to extract the features of interest of the target pathological slice, based on the target detection algorithm and deep learning model, and trains the target bounding box annotation model according to the features of interest to obtain the target bounding box intelligently annotated by the target bounding box annotation model; in addition, the feature type of the feature of interest is introduced, and the key point annotation template corresponding to the feature type is determined, and the key points are annotated in the target bounding box through the key point annotation template. Finally, the pre-verified image with the target bounding box and the annotated key points is sent to the verification personnel node for verification to obtain the second sub-annotated image, thereby improving the reliability of the acquisition of the second sub-annotated image.

[0107] In one embodiment, the segmentation subsystem includes:

[0108] A first pseudo pixel-level label generation module is configured to generate a first pseudo pixel-level label based on the annotation information of the annotated image. When generating the first pseudo pixel-level label based on the annotation information of the annotated image, a region growing algorithm may be used, starting from a seed point in a known bounding box and gradually expanding the region until a preset stop condition is reached. This gradually fills in the pixels of the target object and generates the first pseudo pixel-level label.

[0109] An initial segmentation result acquisition module is used to perform initial segmentation on the target pathology image based on the image region proposal technology and the first pseudo pixel-level label to obtain an initial segmentation result. The initial segmentation result is a segmentation mask containing segmented and non-segmented regions. The segmentation mask is a binary image of the same size as the input image, in which pixels in the segmented region are marked as 1 and pixels in the non-segmented region are marked as 0.

[0110] The preprocessing segmentation result acquisition module is used to train and semantically segment the labeled image based on the WSSS algorithm, according to the preset full convolutional network and the first pseudo pixel-level label, to obtain the preprocessing segmentation result; wherein the preset full convolutional network consists of a convolution layer and an upsampling layer; the preprocessing segmentation result is: a pixel-level labeled image;

[0111] The processing module is used to process the pre-processed segmentation results to obtain segmentation results. Processing the pre-processed segmentation results includes applying morphological operations to fill holes, smooth boundaries, remove small noisy areas, merge pre-processed segmentation results of small areas, and segment pre-processed segmentation results of large areas.

[0112] The working principle and beneficial effects of the above technical solution are:

[0113] This application generates a first pseudo-pixel-level label based on the annotation information and introduces image region proposal technology. Based on the first pseudo-pixel-level label, the target pathology image is initially segmented to obtain an initial segmentation result. The WSSS algorithm is introduced to obtain a preprocessed segmentation result based on a fully convolutional network and the first pseudo-pixel-level label. The preprocessed segmentation result is processed to obtain a segmentation result, which is more accurate.

[0114] In one embodiment, the pre-processing segmentation result acquisition module includes:

[0115] The fully annotated image acquisition submodule is used to acquire a fully annotated image and, at the same time, acquire a second pseudo pixel-level label for the fully annotated image; wherein the fully annotated image is: a pathological image that has been manually annotated before image segmentation; and the second pseudo pixel-level label is: a pixel label of the manually annotated fully annotated image;

[0116] A cross-entropy loss function acquisition submodule is used to train a fully convolutional network based on the fully annotated image and the second pseudo-pixel-level labels corresponding to the fully annotated image based on the WSSS algorithm, and to acquire a cross-entropy loss function during the training of the fully convolutional network; wherein the cross-entropy loss function is a function that characterizes the difference between the prediction result of the fully convolutional network and the true label;

[0117] A prediction difference determination submodule is used to determine the prediction difference based on the cross entropy loss function;

[0118] The network parameter update submodule is used to update the network parameters of the trained fully convolutional network in real time based on the prediction difference based on the backpropagation algorithm. When updating the network parameters of the fully convolutional network, the cross entropy loss function of the corresponding updated fully convolutional network should be minimized.

[0119] The target fully convolutional network acquisition submodule is used to acquire the current fully convolutional network and use it as the target fully convolutional network when the prediction difference is less than or equal to a preset difference threshold. The preset difference threshold is: the prediction difference corresponding to the minimum cross entropy loss function obtained when updating the fully convolutional network in real time;

[0120] The initial segmentation result determination submodule is used to input the first pseudo pixel-level label into the target fully convolutional network to obtain the initial segmentation result.

[0121] The working principle and beneficial effects of the above technical solution are:

[0122] Generally, when performing pathological image segmentation, the WSSS algorithm and the fully convolutional network can be introduced to train and semantically segment the annotated image to obtain the segmentation result. However, the quality of the trained fully convolutional model cannot be guaranteed. Therefore, the present application introduces the fully annotated image and the second pseudo-pixel-level label of the fully annotated image, and trains the fully convolutional network based on the fully annotated image and the second pseudo-pixel-level label. During the training process, the cross-entropy loss function is introduced to determine the predicted difference corresponding to the cross-entropy loss function, and the back-propagation algorithm is introduced to update the network parameters of the fully convolutional network in real time, thereby improving the rationality of the network parameter setting of the fully convolutional network. Determine the target fully convolutional network with a predicted difference less than or equal to the difference threshold, input the first pseudo-pixel-level label into the target fully convolutional network to obtain the initial segmentation result, and further improve the accuracy of the initial segmentation result.

[0123] An embodiment of the present invention provides a weakly supervised pathological image segmentation system based on frame-level annotation, further comprising:

[0124] The report generation subsystem is used to obtain input information from the target display party after viewing the segmentation results, and generate a pathology diagnosis report based on the input information; the target display party is, for example, the attending physician; the input information is the input operation of the attending physician in the report generation subsystem;

[0125] The report generation subsystem includes:

[0126] An input template library acquisition module is used to acquire an input template library; wherein the input template library contains multiple input framework templates for diagnostic reports of different pathological types;

[0127] A recommended input template list determination module is used to determine a recommended input template list based on the segmentation results and the input template library;

[0128] A target input template acquisition module is used to acquire a target input template selected by a target display party from a recommended input template list, and to display the target input template to the target display party;

[0129] The operation information acquisition module is used to obtain the operation information of each operable item in the target input template by the target display party; the operable item is: the editable part of the target input template; the operation information is, for example,: "..." is entered, what is checked, and what is marked;

[0130] The input item determination module is used to determine the input item in the operable item based on the operation instruction library and operation information corresponding to the operable item; wherein the operation instruction library contains operation instructions corresponding to the operation instruction library, and the operable item will present different content according to different operation instructions. The operation instruction corresponding to the operable item is determined and the content presented corresponding to the operation instruction is used as the input item, for example: a 2mm tumor is found in a certain part;

[0131] The input information acquisition module is used to integrate input items and target input templates to obtain input information.

[0132] The working principle and beneficial effects of the above technical solution are:

[0133] Generally, after a doctor has reviewed the pathological image segmentation results, he needs to issue a diagnosis report. However, manually writing a diagnosis report is inefficient and inconvenient. Therefore, this application introduces an input template library. According to the segmentation results and the input template library, a list of recommended input templates is determined, and the target input template in the recommended input template list is determined and displayed to the target display party. According to the operation information of each operable item in the obtained target input template and the operation instruction library corresponding to the operable item, the input item in the operable item is determined, and the input item and the target input template are integrated to obtain the input information. The acquisition of input information is faster and more standardized, and further, the generation efficiency of the pathological diagnosis report is greatly improved.

[0134] In one embodiment, the recommended input template list determination module includes:

[0135] The auxiliary diagnosis model acquisition submodule is used to obtain the auxiliary diagnosis model; wherein the auxiliary diagnosis model is: an AI model that replaces manual auxiliary diagnosis of pathology based on the pathology image segmentation results;

[0136] A first auxiliary diagnosis information acquisition submodule is configured to input the segmentation result into the auxiliary diagnosis model to obtain first auxiliary diagnosis information; wherein the first auxiliary diagnosis information is: diagnosis information output by the auxiliary diagnosis model based on the segmentation result;

[0137] The pre-selected input template acquisition submodule is used to parse the input template library and obtain multiple pre-selected input templates;

[0138] The second auxiliary diagnosis information acquisition submodule is used to acquire preset second auxiliary diagnosis information corresponding to the preselected input template; wherein the second auxiliary diagnosis information is: diagnostic information of the pathology diagnosed corresponding to the preselected input template;

[0139] An information similarity calculation submodule is used to calculate the information similarity between the first auxiliary diagnosis information and the second auxiliary diagnosis information; wherein the information similarity is, for example, 0.9;

[0140] The template sorting acquisition submodule is used to sort the pre-selected input templates according to the similarity of information, and obtain the template sorting of the pre-selected input templates; wherein the template sorting is: which pre-selected input template is sorted first, and which pre-selected input template is sorted next;

[0141] The recommended input template list construction submodule is used to construct a recommended input template list based on the preset recommended input template list construction rules and the template sorting of the pre-selected input templates. The recommended input template list construction rules are as follows: the diagnostic disease types corresponding to the pre-selected input templates are sorted and displayed on the PC.

[0142] The working principle and beneficial effects of the above technical solution are:

[0143] This application introduces an auxiliary diagnosis model, obtains first auxiliary diagnosis information based on the segmentation results and the auxiliary diagnosis model, and simultaneously obtains second auxiliary diagnosis information corresponding to a preselected input template. The preselected input templates are sorted based on the calculated similarity between the first auxiliary diagnosis information and the second auxiliary diagnosis information to determine the template sorting, which is more appropriate. A recommended input template list is constructed based on the recommended input template list construction rules and template sorting, which is more reasonable.

[0144] In one embodiment, the auxiliary diagnosis model acquisition submodule includes:

[0145] a first process record acquisition unit, configured to acquire a first process record of a target doctor reviewing the pathology image segmentation information and performing a pathology diagnosis; wherein the pathology image segmentation information is a display screen showing the segmentation result; and the first process record is a video captured during the target doctor reviewing the pathology image segmentation information and performing a pathology diagnosis, the video being captured by a camera provided on a display device where the pathology image segmentation information is located.

[0146] a visual dwell time determination unit, configured to determine, based on an attention analysis technique and according to the first procedure record, a visual dwell time of the target doctor in a first segmented area of ​​the segmented image corresponding to the first procedure record; wherein the first segmented area is any segmented area in the segmented image; and the visual dwell time is the length of time the target doctor's gaze remains in the first segmented area, for example, 8 seconds;

[0147] A region type acquisition unit, configured to acquire the region type of the first segmented region; wherein the region type is, for example, cell nucleus, cytoplasm, and cell fluid;

[0148] an analysis importance value determination unit, configured to determine an analysis importance value for a region type based on a preset region type-analysis importance value library; wherein the region type-analysis importance value library has a one-to-one correspondence between region types and analysis importance values, and the higher the analysis importance value, the longer the target doctor should review and analyze the first analysis region of the corresponding region type;

[0149] an analysis time ideal value determination unit, configured to determine an analysis time ideal value for the first segmented region based on a preset analysis importance value-analysis time ideal value library, wherein the preset analysis importance value-analysis time ideal value library includes a plurality of one-to-one corresponding analysis importance values ​​and analysis time ideal values;

[0150] a reference value calculation unit, configured to divide the visual dwell time of the first segmented area by the ideal analysis time of the first segmented area to obtain a reference value, and associate the reference value with the corresponding first process record; wherein, if the visual dwell time is, for example, 8 seconds and the ideal analysis time is, for example, 10 seconds, then the reference value is 0.8;

[0151] a target sum value calculation unit, configured to calculate an average value of the reference values ​​associated with the first process record and use the average value as a target value; wherein the larger the target value, the more likely the corresponding first process record is to be used as training data for the auxiliary diagnosis model;

[0152] A second process record determination unit, configured to use the corresponding first process record as the second process record if the target value is greater than or equal to a preset target value threshold; wherein the target value threshold is manually preset;

[0153] The auxiliary diagnosis model training unit is used to train the second process record based on deep learning technology to obtain an auxiliary diagnosis model.

[0154] The working principle and beneficial effects of the above technical solution are:

[0155] This application introduces attention analysis technology to determine the target doctor's visual dwell time in the first segmented area of ​​the segmented image in the acquired first process record. At the same time, the region type of the first segmented area is obtained. According to the introduced region type-analysis important value library and analysis important value-analysis time ideal value library, the ideal analysis time value of the first segmented area is determined. The visual dwell time of the first segmented area is divided by the ideal analysis time value of the first segmented area to obtain a reference value and associate it with the first process record, thereby improving the rationality of the reference value acquisition. The target value determined by calculating the average value of the reference value associated with the first process record is compared with the preset target value threshold to determine the second process record suitable for training, thereby improving the accuracy of the auxiliary diagnosis model for auxiliary diagnosis.

[0156] The embodiment of the present invention provides a weakly supervised pathological image segmentation method based on frame-level annotation, such as Figure 2 As shown, including:

[0157] Step 1: Collect target pathological sections;

[0158] Step 2: Send the target pathological slice to the preset annotation node for frame-level annotation, and obtain the annotated image replied by the annotation node;

[0159] Step 3: Based on the WSSS algorithm, perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result.

[0160] In one embodiment, step 2: sending the target pathological slice to a preset annotation node for frame-level annotation, and obtaining an annotated image replied by the annotation node, includes:

[0161] Get the node attributes of the annotation node;

[0162] Determine the node type of the labeled node according to the node attributes;

[0163] If the node type is a manually annotated node, obtain the first sub-annotated image;

[0164] If the node type is a smart annotation node, obtain the second sub-annotation image;

[0165] The first sub-annotated image and the second sub-annotated image are used together as the annotated image.

[0166] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A weakly supervised pathological image segmentation system based on box-level annotation, characterized by: include: Pathology section acquisition subsystem, used to acquire target pathology sections; The frame-level annotation subsystem is used to send the target pathological slice to the preset annotation node for frame-level annotation and obtain the annotated image replied by the annotation node; The segmentation subsystem is used to perform weakly supervised semantic segmentation on the annotated image based on the WSSS algorithm to obtain the segmentation results; The report generation subsystem is used to obtain input information from the target display party after viewing the segmentation results, and generate a pathology diagnosis report based on the input information; The report generation subsystem performs the following operations: Get the input template library; Determine the recommended input template list based on the segmentation results and the input template library; Obtaining a target input template selected by a target display party from a recommended input template list, and displaying the target input template to the target display party; Obtaining the target displayer's operation information for each operable item in the target input template; Determine the input items in the operable items according to the operation instruction library and operation information corresponding to the operable items; Integrate input items and target input templates to obtain input information; Among them, according to the segmentation results and the input template library, a recommended input template list is determined, including: Obtain auxiliary diagnosis models; Inputting the segmentation result into the auxiliary diagnosis model to obtain first auxiliary diagnosis information; Parse the input template library to obtain multiple pre-selected input templates; Obtaining preset second auxiliary diagnosis information corresponding to the preselected input template; calculating the degree of similarity between the first auxiliary diagnosis information and the second auxiliary diagnosis information; Sorting the preselected input templates according to information similarity to obtain a template ranking of the preselected input templates; Based on the preset recommended input template list construction rules, the recommended input template list is constructed according to the template sorting of the pre-selected input templates; Among them, obtaining auxiliary diagnosis models includes: Obtaining a first process record of a target doctor reviewing pathological image segmentation information to perform pathological diagnosis; Based on the attention analysis technology, according to the first procedure record, determining the visual stay time of the target doctor in the first segmented area of ​​the segmented image corresponding to the record in the first procedure record; Get the region type of the first segmented region; Determine the analysis important value of the regional type according to the preset regional type-analysis important value library; Determine the ideal analysis time value for the first segmented area according to a preset analysis important value-analysis time ideal value library; Dividing the visual dwell time of the first segmented area by the ideal analysis time of the first segmented area to obtain a reference value, and associating the reference value with the corresponding first process record; Calculating an average of the reference values ​​associated with the first process record and using the average as the target value; If the target value is greater than or equal to the preset target value threshold, the corresponding first process record is used as the second process record; The second process record is trained based on deep learning technology to obtain an auxiliary diagnosis model.

2. The weakly supervised pathological image segmentation system based on frame-level annotation according to claim 1, characterized in that The frame-level annotation subsystem includes: Node attribute acquisition module, used to obtain node attributes of labeled nodes; A node type determination module is used to determine the node type of the labeled node based on the node attributes; A first sub-annotated image acquisition module, configured to acquire a first sub-annotated image if the node type is a manually annotated node; A second sub-annotation image acquisition module, configured to acquire a second sub-annotation image if the node type is an intelligent annotation node; The annotated image integration module is configured to use the first sub-annotated image and the second sub-annotated image together as an annotated image.

3. The weakly supervised pathological image segmentation system based on frame-level annotation according to claim 2, characterized in that The second sub-annotation image acquisition module includes: A feature of interest extraction submodule is used to extract the features of interest of the target pathological section based on a preset feature of interest extraction template; The object bounding box annotation model training submodule is used to train the object bounding box annotation model based on the object detection algorithm and the preset deep learning model according to the features of interest; The target bounding box annotation submodule is used to annotate the target bounding box in the target pathological section according to the target bounding box annotation model; The key point annotation template determination submodule is used to determine the key point annotation template according to the feature type of the feature of interest corresponding to the target bounding box; The key point annotation submodule is used to annotate key points within the target bounding box according to the key point annotation template; The pre-verification image determination submodule is used to determine the corresponding target pathological slice as the pre-verification image after all target bounding boxes and key points are labeled; The second sub-annotated image determination submodule is used to send the pre-verification image to a preset verification personnel node and obtain the second sub-annotated image verified by the verification personnel node.

4. The weakly supervised pathological image segmentation system based on frame-level annotation according to claim 1, characterized in that Segmentation subsystem, including: A first pseudo pixel-level label generation module, configured to generate a first pseudo pixel-level label based on the annotation information of the annotated image; An initial segmentation result acquisition module is used to perform initial segmentation on the target pathological image based on the image region proposal technology and the first pseudo pixel-level label to obtain an initial segmentation result; The preprocessing segmentation result acquisition module is used to train and semantically segment the labeled image based on the WSSS algorithm according to the preset full convolutional network and the first pseudo pixel-level label to obtain the preprocessing segmentation result; The processing module is used to process the preprocessing segmentation results to obtain the segmentation results.

5. The weakly supervised pathological image segmentation system based on frame-level annotation according to claim 4, characterized in that Preprocessing segmentation result acquisition module, including: A fully annotated image acquisition submodule is used to acquire a fully annotated image and, at the same time, acquire a second pseudo pixel-level label of the fully annotated image; A cross entropy loss function acquisition submodule is used to train a fully convolutional network based on the fully annotated image and the second pseudo pixel-level label corresponding to the fully annotated image based on the WSSS algorithm, and to obtain a cross entropy loss function during the training process of the fully convolutional network; A prediction difference determination submodule is used to determine the prediction difference based on the cross entropy loss function; The network parameter update submodule is used to update the network parameters of the trained fully convolutional network in real time based on the prediction difference based on the back-propagation algorithm; The target full convolutional network acquisition submodule is used to obtain the current full convolutional network and use it as the target full convolutional network when the predicted difference is less than or equal to the preset difference threshold; The initial segmentation result determination submodule is used to input the first pseudo pixel-level label into the target fully convolutional network to obtain the initial segmentation result.

6. A weakly supervised pathological image segmentation method based on box-level annotation, characterized in that: include: Step 1: Collect target pathological sections; Step 2: Send the target pathological slice to the preset annotation node for frame-level annotation, and obtain the annotated image replied by the annotation node; Step 3: Based on the WSSS algorithm, perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result; The weakly supervised pathological image segmentation method based on box-level annotation also includes: Obtain input information from the target display party after viewing the segmentation results, and generate a pathology diagnosis report based on the input information, including: Get the input template library; Determine the recommended input template list based on the segmentation results and the input template library; Obtaining a target input template selected by a target display party from a recommended input template list, and displaying the target input template to the target display party; Obtaining the target displayer's operation information for each operable item in the target input template; Determine the input items in the operable items according to the operation instruction library and operation information corresponding to the operable items; Integrate input items and target input templates to obtain input information; Among them, according to the segmentation results and the input template library, a recommended input template list is determined, including: Obtain auxiliary diagnosis models; Inputting the segmentation result into the auxiliary diagnosis model to obtain first auxiliary diagnosis information; Parse the input template library to obtain multiple pre-selected input templates; Obtaining preset second auxiliary diagnosis information corresponding to the preselected input template; calculating the degree of similarity between the first auxiliary diagnosis information and the second auxiliary diagnosis information; Sorting the preselected input templates according to information similarity to obtain a template ranking of the preselected input templates; Based on the preset recommended input template list construction rules, the recommended input template list is constructed according to the template sorting of the pre-selected input templates; Among them, obtaining auxiliary diagnosis models includes: Obtaining a first process record of a target doctor reviewing pathological image segmentation information to perform pathological diagnosis; Based on the attention analysis technology, according to the first procedure record, determining the visual stay time of the target doctor in the first segmented area of ​​the segmented image corresponding to the record in the first procedure record; Get the region type of the first segmented region; Determine the analysis important value of the regional type according to the preset regional type-analysis important value library; Determine the ideal analysis time value for the first segmented area according to a preset analysis important value-analysis time ideal value library; Dividing the visual dwell time of the first segmented area by the ideal analysis time of the first segmented area to obtain a reference value, and associating the reference value with the corresponding first process record; Calculating an average of the reference values ​​associated with the first process record and using the average as the target value; If the target value is greater than or equal to the preset target value threshold, the corresponding first process record is used as the second process record; The second process record is trained based on deep learning technology to obtain an auxiliary diagnosis model.

7. The weakly supervised pathological image segmentation method based on frame-level annotation according to claim 6, characterized in that Step 2: Send the target pathology slice to the preset annotation node for frame-level annotation, and obtain the annotated image replied by the annotation node, including: Get the node attributes of the annotation node; Determine the node type of the labeled node according to the node attributes; If the node type is a manually annotated node, obtain the first sub-annotated image; If the node type is a smart annotation node, obtain the second sub-annotation image; The first sub-annotated image and the second sub-annotated image are used together as the annotated image.

Images