Weakly supervised pathological image segmentation system and method based on frame-level labeling

Abstract

The invention provides a weak supervision pathological image segmentation system and method based on frame-level annotation, and the system comprises a pathological section collection subsystem which is used for collecting a target pathological section; the frame-level labeling subsystem is used for sending the target pathological section to a preset labeling node for frame-level labeling, and obtaining a labeling image replied by the labeling node; and the segmentation subsystem is used for carrying out weak supervision semantic segmentation on the annotated image based on a WSSS algorithm to obtain a segmentation result. According to the weak supervision pathological image segmentation system and method based on frame-level labeling, the collected target pathological section is sent to the labeling node for frame-level labeling, the labeling image replied by the labeling node is obtained, the WSSS algorithm is introduced to carry out weak supervision semantic segmentation on the labeling image, and the segmentation result is obtained. Precise segmentation of the pathological image is realized through less annotation information, the annotation cost is reduced, and the efficiency of assisting doctors in diagnosis is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a weakly supervised pathological image segmentation system and method based on frame-level annotation. Background Art

[0002] Pathological images are images generated by scanning tissue sections at high magnification using a microscope. They have extremely high resolution and rich cellular structural information. Smart pathology refers to improving the diagnostic efficiency and accuracy of pathologists by applying artificial intelligence technology to the pathological diagnosis process. At present, the mainstream smart pathology methods include: image classification, target detection and semantic segmentation methods based on deep learning, such as convolutional neural networks (CNN), fully convolutional networks (FCN) and U-Net. Semantic segmentation is a task in the field of computer vision that aims to assign a category label to each pixel in the image. These methods have achieved remarkable results in pathological image diagnosis.

[0003] The invention patent with application number: CN202110621825.5 discloses a pathological image segmentation method based on domain adversarial self-supervised learning, wherein the method includes: obtaining pathological images to establish a pathological image self-supervised dataset; establishing a domain adversarial self-supervised model; using the pathological image self-supervised dataset to perform deep learning training on the domain adversarial self-supervised model; establishing a pathological image segmentation model; using the domain adversarial self-supervised model after deep learning to initialize the pathological image segmentation model; performing pixel-level annotation on the lesion area in the pathological image to establish a pathological image segmentation dataset; using the pathological image segmentation dataset to perform deep learning training on the pathological image segmentation model; using the pathological image segmentation model trained by deep learning to segment the unknown lesion area of ​​the pathological image. The above invention adopts the method of domain adversarial self-supervised learning, which effectively alleviates the segmentation model's dependence on a large amount of manual annotations and solves the problem of fluctuations in the model's segmentation performance in different domains.

[0004] However, the above existing technologies still require pixel-level labeled data as training sets, which is time-consuming and costly. Furthermore, the efficiency of assisting doctors in diagnosis is also low.

[0005] In view of this, there is an urgent need for a high-frequency quantitative trading method and system based on deep learning to at least solve the above-mentioned deficiencies. Summary of the invention

[0006] One of the purposes of the present invention is to provide a weakly supervised pathological image segmentation system based on frame-level annotation, in which the collected target pathological slices are sent to the annotation node for frame-level annotation, the annotated image replied by the annotation node is obtained, and the WSSS algorithm is introduced to perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result, so as to achieve accurate segmentation of the pathological image with less annotation information, reduce the annotation cost, and improve the efficiency of assisting doctors in diagnosis.

[0007] The weakly supervised pathological image segmentation system based on frame-level annotation provided by the embodiment of the present invention includes:

[0008] Pathological section acquisition subsystem, used for acquiring target pathological sections;

[0009] The frame-level annotation subsystem is used to send the target pathological slice to the preset annotation node for frame-level annotation and obtain the annotated image replied by the annotation node;

[0010] The segmentation subsystem is used to perform weakly supervised semantic segmentation on the annotated image based on the WSSS algorithm to obtain the segmentation result.

[0011] Preferably, the frame-level annotation subsystem includes:

[0012] A node attribute acquisition module is used to obtain the node attributes of the labeled node;

[0013] A node type determination module is used to determine the node type of the labeled node according to the node attributes;

[0014] A first sub-annotated image acquisition module, used for acquiring a first sub-annotated image if the node type is a manually annotated node;

[0015] A second sub-annotation image acquisition module, used for acquiring a second sub-annotation image if the node type is an intelligent annotation node;

[0016] The annotated image integration module is used to use the first sub-annotated image and the second sub-annotated image together as an annotated image.

[0017] Preferably, the second sub-annotated image acquisition module includes:

[0018] A feature of interest extraction submodule is used to extract the feature of interest of the target pathological section based on a preset feature of interest extraction template;

[0019] The object bounding box annotation model training submodule is used to train the object bounding box annotation model according to the features of interest based on the object detection algorithm and the preset deep learning model;

[0020] A target bounding box annotation submodule is used to annotate a target bounding box in a target pathological slice according to a target bounding box annotation model;

[0021] A key point annotation template determination submodule is used to determine a key point annotation template according to a feature type of a feature of interest corresponding to a target bounding box;

[0022] The key point annotation submodule is used to annotate key points within the target bounding box according to the key point annotation template;

[0023] A pre-verification image determination submodule is used to use the corresponding target pathological slice as a pre-verification image after all target bounding boxes and annotation key points are annotated;

[0024] The second sub-annotated image determination submodule is used to send the pre-verified image to a preset verification personnel node and obtain the second sub-annotated image verified by the verification personnel node.

[0025] Preferably, the segmentation subsystem comprises:

[0026] A first pseudo pixel-level label generation module, used to generate a first pseudo pixel-level label according to the annotation information of the annotated image;

[0027] An initial segmentation result acquisition module is used to perform initial segmentation on the target pathological image based on the image region proposal technology and the first pseudo pixel-level label to obtain an initial segmentation result;

[0028] A preprocessing segmentation result acquisition module is used to perform training and semantic segmentation on the labeled image based on the WSSS algorithm, according to a preset full convolutional network and a first pseudo pixel-level label, to obtain a preprocessing segmentation result;

[0029] The processing module is used to process the preprocessing segmentation result to obtain the segmentation result.

[0030] Preferably, the preprocessing segmentation result acquisition module includes:

[0031] A fully annotated image acquisition submodule is used to acquire a fully annotated image and, at the same time, acquire a second pseudo pixel-level label of the fully annotated image;

[0032] A cross entropy loss function acquisition submodule is used to train a full convolutional network based on a WSSS algorithm according to a full annotated image and a second pseudo pixel-level label corresponding to the full annotated image, and to acquire a cross entropy loss function during the training process of the full convolutional network;

[0033] A prediction difference determination submodule, for determining the prediction difference based on a cross entropy loss function;

[0034] The network parameter update submodule is used to update the network parameters of the trained fully convolutional network in real time according to the prediction difference based on the back-propagation algorithm;

[0035] The target full convolutional network acquisition submodule is used to acquire the current full convolutional network and use it as the target full convolutional network when the predicted difference is less than or equal to a preset difference threshold;

[0036] The initial segmentation result determination submodule is used to input the first pseudo pixel-level label into the target full convolutional network to obtain an initial segmentation result.

[0037] The weakly supervised pathological image segmentation system based on frame-level annotation provided by an embodiment of the present invention further includes:

[0038] The report generation subsystem is used to obtain the input information of the target display party after viewing the segmentation results, and generate a pathology diagnosis report based on the input information;

[0039] Among them, the report generation subsystem includes:

[0040] An input template library acquisition module is used to acquire an input template library;

[0041] A recommended input template list determination module is used to determine a recommended input template list according to the segmentation result and the input template library;

[0042] A target input template acquisition module is used to acquire a target input template selected by a target display party from a recommended input template list, and display the target input template to the target display party;

[0043] An operation information acquisition module is used to acquire the operation information of each operable item in the target input template of the target display party;

[0044] An input item determination module, used to determine an input item in the operable item according to an operation instruction library and operation information corresponding to the operable item;

[0045] The input information acquisition module is used to integrate the input items and the target input template to obtain the input information.

[0046] Preferably, the recommended input template list determination module includes:

[0047] Auxiliary diagnosis model acquisition submodule, used to obtain auxiliary diagnosis model;

[0048] A first auxiliary diagnosis information acquisition submodule, used for inputting the segmentation result into the auxiliary diagnosis model to obtain first auxiliary diagnosis information;

[0049] A pre-selected input template acquisition submodule is used to parse the input template library and obtain multiple pre-selected input templates;

[0050] The second auxiliary diagnosis information acquisition submodule is used to acquire preset second auxiliary diagnosis information corresponding to the preselected input template;

[0051] An information similarity calculation submodule, used to calculate the information similarity between the first auxiliary diagnosis information and the second auxiliary diagnosis information;

[0052] The template sorting acquisition submodule is used to sort the pre-selected input templates according to the degree of information similarity to obtain the template sorting of the pre-selected input templates;

[0053] The recommended input template list building submodule is used to build a recommended input template list based on preset recommended input template list building rules and according to the template sorting of the pre-selected input templates.

[0054] Preferably, the auxiliary diagnosis model acquisition submodule includes:

[0055] A first process record acquisition unit, used to acquire a first process record of a target doctor viewing the pathological image segmentation information to perform pathological diagnosis;

[0056] a visual dwell time determination unit, configured to determine, based on the attention analysis technology and according to the first procedure record, a visual dwell time of the target doctor in the first segmented area in the segmented image corresponding to the record in the first procedure record;

[0057] A region type acquisition unit, used to acquire the region type of the first segmented region;

[0058] An analysis importance value determination unit, used to determine the analysis importance value of the region type according to a preset region type-analysis importance value library;

[0059] An analysis time ideal value determination unit, used to determine the analysis time ideal value of the first segmented area according to a preset analysis importance value-analysis time ideal value library;

[0060] A reference value calculation unit, used for dividing the visual dwell time of the first segmented area by the ideal value of the analysis time of the first segmented area, obtaining a reference value, and associating it with the corresponding first process record;

[0061] A target and value calculation unit, used for calculating an average value of the reference value associated with the first process record and using it as a target value;

[0062] A second process record determination unit, configured to use the corresponding first process record as the second process record if the target value is greater than or equal to a preset target value threshold;

[0063] The auxiliary diagnosis model training unit is used to train the second process record based on deep learning technology to obtain an auxiliary diagnosis model.

[0064] The weakly supervised pathological image segmentation method based on frame-level annotation provided by the embodiment of the present invention includes:

[0065] Step 1: Collect target pathological sections;

[0066] Step 2: Send the target pathological slice to the preset annotation node for frame-level annotation, and obtain the annotated image replied by the annotation node;

[0067] Step 3: Based on the WSSS algorithm, perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result.

[0068] Preferably, step 2: sending the target pathological slice to a preset annotation node for frame-level annotation, and obtaining an annotated image replied by the annotation node, includes:

[0069] Get the node attributes of the annotation node;

[0070] Determine the node type of the labeled node according to the node attributes;

[0071] If the node type is a manually annotated node, obtain the first sub-annotated image;

[0072] If the node type is a smart annotation node, obtain the second sub-annotation image;

[0073] The first sub-annotated image and the second sub-annotated image are used together as the annotated image.

[0074] The beneficial effects of the present invention are:

[0075] The present invention sends the collected target pathological slices to the annotation node for frame-level annotation, obtains the annotated image replied by the annotation node, introduces the WSSS algorithm to perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result, realizes accurate segmentation of pathological images with less annotation information, reduces the annotation cost, and improves the efficiency of assisting doctors in diagnosis.

[0076] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0077] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0079] Figure 1 Schematic diagram of a weakly supervised pathological image segmentation system based on frame-level annotation in an embodiment of the present invention;

[0080] Figure 2Schematic diagram of a weakly supervised pathological image segmentation method based on frame-level annotation in an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0082] The embodiment of the present invention provides a weakly supervised pathological image segmentation system based on frame-level annotation, such as Figure 1 As shown, including:

[0083] The pathological slice acquisition subsystem 1 is used to acquire target pathological slices; wherein the target pathological slices are: pathological slices that need to be segmented for pathological images;

[0084] The frame-level annotation subsystem 2 is used to send the target pathological slice to the preset annotation node for frame-level annotation, and obtain the annotated image replied by the annotation node; wherein the preset annotation node is: the communication node of the annotation doctor who annotates the pathological image or the communication node of the model that performs intelligent annotation; the frame-level annotation is: marking the position and size of the target object by drawing a rectangular bounding box in the image; the annotated image is: the target pathological slice annotated by the annotation node after the frame-level annotation is completed;

[0085] The segmentation subsystem 3 is used to perform weakly supervised semantic segmentation on the annotated image based on the WSSS algorithm to obtain a segmentation result. The WSSS (Weakly Supervised Semantic Segmentation) algorithm is an algorithm for weakly supervised semantic segmentation; the segmentation result is: an annotated image with a category label of annotated pixel categories, and the category label is: the category label corresponds to a pixel area description of the pixel area in the annotated image, and the pixel area description is, for example: what kind of tissue in what area range.

[0086] The working principle and beneficial effects of the above technical solution are:

[0087] This application sends the collected target pathological slices to the annotation node for frame-level annotation, obtains the annotated image replied by the annotation node, introduces the WSSS algorithm to perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result, realizes accurate segmentation of pathological images with less annotation information, reduces the annotation cost, and improves the efficiency of assisting doctors in diagnosis.

[0088] In specific applications, the acquired target pathological image is sent to the annotation node for frame-level annotation. The system automatically performs weakly supervised semantic segmentation on the annotated image to obtain the segmentation result.

[0089] In one embodiment, the frame-level annotation subsystem includes:

[0090] The node attribute acquisition module is used to acquire the node attributes of the labeled node; wherein the node attributes are information describing the properties and characteristics of the node; the properties and characteristics of the node are the connected party corresponding to the node, such as doctor, or model;

[0091] A node type determination module is used to determine the node type of the labeled node according to the node attributes; wherein the node types include: manually labeled nodes and intelligently labeled nodes, the manually labeled nodes are: the personnel nodes that perform manual frame-level labeling; the intelligently labeled nodes are: the communication nodes of the model that performs artificial intelligence-assisted frame-level labeling;

[0092] A first sub-annotated image acquisition module is used to acquire a first sub-annotated image if the node type is a manually annotated node; wherein the first sub-annotated image is: a frame-level annotated image annotated by the manually annotated node;

[0093] A second sub-annotation image acquisition module is used to acquire a second sub-annotation image if the node type is an intelligent annotation node; wherein the second sub-annotation image is: a frame-level annotation image annotated by the intelligent annotation node;

[0094] The annotated image integration module is used to use the first sub-annotated image and the second sub-annotated image together as an annotated image.

[0095] The working principle and beneficial effects of the above technical solution are:

[0096] This application introduces node attributes, and determines the node type according to the node attributes. When the node type is a manually annotated node, the first sub-annotated image is obtained; when the node type is an intelligently annotated node, the second sub-annotated image is obtained. The first sub-annotated image and the second sub-annotated image are integrated as the annotated image, which improves the comprehensiveness of the annotated image acquisition.

[0097] In one embodiment, the second sub-annotation image acquisition module includes:

[0098] The feature of interest extraction submodule is used to extract the feature of interest of the target pathological slice based on a preset feature of interest extraction template; wherein the preset feature of interest extraction template constrains only the extraction of the feature of interest, and the feature of interest is, for example, the texture and color of the target pathological slice;

[0099] The target bounding box annotation model training submodule is used to train the target bounding box annotation model based on the target detection algorithm and the preset deep learning model according to the features of interest; wherein the target bounding box annotation model is used to replace manual box-level annotation of the target pathology image;

[0100] A target bounding box annotation submodule is used to annotate a target bounding box in a target pathological slice according to a target bounding box annotation model;

[0101] The key point annotation template determination submodule is used to determine the key point annotation template according to the feature type of the feature of interest corresponding to the target bounding box; wherein the feature type is: the feature type of the feature of interest, such as: the feature of interest of the gastric pathological section; wherein the key point annotation template is: constraining only the key point annotation of the pathological section corresponding to the feature type, and not annotating other contents;

[0102] The key point annotation submodule is used to annotate key points within the target bounding box according to the key point annotation template; the annotated key points are, for example, positions such as cell nuclei, blood vessels, and cell boundaries;

[0103] A pre-verification image determination submodule is used to use the corresponding target pathological slice as a pre-verification image after all target bounding boxes and annotation key points are annotated;

[0104] The second sub-annotated image determination submodule is used to send the pre-verified image to the preset verification personnel node, and obtain the second sub-annotated image verified by the verification personnel node. After the intelligent annotation method completes the bounding box annotation of the region of interest in the pathological image, in order to ensure the quality and accuracy of the annotation results, manual review and correction are necessary. Therefore, the pre-verified image completed by intelligent annotation is sent to the verification personnel. It should be noted that although intelligent annotation still requires manual review, the review and correction work is much more efficient than manual annotation, so it is necessary.

[0105] The working principle and beneficial effects of the above technical solution are:

[0106] In general, to ensure the quality and accuracy of the annotation results, pathological images are usually annotated manually at the frame level, but the efficiency of manual annotation is low. At the same time, deep learning technology can be used to learn artificial experience to achieve intelligent annotation, but the quality of intelligent annotation depends on the quality of the training data learned. Therefore, this application introduces a feature extraction template to extract the feature of interest of the target pathological slice, based on the target detection algorithm and the deep learning model, and trains the target bounding box annotation model according to the feature of interest, and obtains the target bounding box intelligently annotated by the target bounding box annotation model; in addition, the feature type of the feature of interest is introduced, and the key point annotation template corresponding to the feature type is determined, and the key points are annotated in the target bounding box through the key point annotation template. Finally, the pre-verification image with the target bounding box and the annotated key points is sent to the verification personnel node for verification, and the second sub-annotated image is obtained, which improves the reliability of the acquisition of the second sub-annotated image.

[0107] In one embodiment, the segmentation subsystem includes:

[0108] A first pseudo pixel-level label generation module is used to generate a first pseudo pixel-level label according to the annotation information of the annotated image; wherein, when generating the first pseudo pixel-level label according to the annotation information of the annotated image, a region growing algorithm can be used to gradually expand the region from a known seed point in a bounding box until a preset stop condition is reached. In this way, the pixels of the target object can be gradually filled and the first pseudo pixel-level label can be generated;

[0109] An initial segmentation result acquisition module is used to perform initial segmentation on the target pathological image based on the image region proposal technology and the first pseudo pixel-level label to obtain an initial segmentation result; wherein the initial segmentation result is: a segmentation mask containing segmented areas and non-segmented areas, and the segmentation mask is a binary image with the same size as the input image, wherein the pixels of the segmented area are marked as 1, and the pixels of the non-segmented area are marked as 0;

[0110] The preprocessing segmentation result acquisition module is used to train and semantically segment the labeled image based on the WSSS algorithm according to the preset full convolution network and the first pseudo pixel-level label to obtain the preprocessing segmentation result; wherein the preset full convolution network is composed of a convolution layer and an upsampling layer; the preprocessing segmentation result is: a pixel-level label image;

[0111] The processing module is used to process the preprocessing segmentation results to obtain the segmentation results. The processing of the preprocessing segmentation results includes: applying morphological operations to fill holes, smooth boundaries or remove small noise areas, merging the preprocessing segmentation results of small areas, and segmenting the preprocessing segmentation results of large areas.

[0112] The working principle and beneficial effects of the above technical solution are:

[0113] This application generates a first pseudo pixel-level label based on the annotation information, introduces the image region proposal technology, and performs initial segmentation on the target pathological image based on the first pseudo pixel-level label to obtain the initial segmentation result. The WSSS algorithm is introduced to obtain the preprocessing segmentation result based on the full convolutional network and the first pseudo pixel-level label. The preprocessing segmentation result is processed to obtain the segmentation result, and the segmentation result is obtained more accurately.

[0114] In one embodiment, the preprocessing segmentation result acquisition module includes:

[0115] The fully annotated image acquisition submodule is used to acquire the fully annotated image and, at the same time, acquire the second pseudo pixel-level label of the fully annotated image; wherein the fully annotated image is: a pathological image that is manually annotated before image segmentation; the second pseudo pixel-level label is: a pixel label of the manually annotated fully annotated image;

[0116] A cross entropy loss function acquisition submodule is used to train a full convolutional network based on a WSSS algorithm according to a full annotated image and a second pseudo pixel-level label corresponding to the full annotated image, and to acquire a cross entropy loss function during the training of the full convolutional network; wherein the cross entropy loss function is a function that characterizes the difference between the prediction result of the full convolutional network and the true label;

[0117] A prediction difference determination submodule, for determining the prediction difference based on a cross entropy loss function;

[0118] The network parameter update submodule is used to update the network parameters of the trained full convolutional network in real time based on the back propagation algorithm and the prediction difference. When updating the network parameters of the full convolutional network, the cross entropy loss function of the corresponding updated full convolutional network should be as small as possible.

[0119] The target full convolution network acquisition submodule is used to acquire the current full convolution network and use it as the target full convolution network when the prediction difference is less than or equal to a preset difference threshold; wherein the preset difference threshold is: the prediction difference corresponding to the minimum cross entropy loss function obtained when the full convolution network is updated in real time;

[0120] The initial segmentation result determination submodule is used to input the first pseudo pixel-level label into the target full convolutional network to obtain an initial segmentation result.

[0121] The working principle and beneficial effects of the above technical solution are:

[0122] Generally, when performing pathological image segmentation, the WSSS algorithm and the fully convolutional network can be introduced to train and semantically segment the annotated images to obtain the segmentation results. However, the quality of the trained fully convolutional model cannot be guaranteed. Therefore, the present application introduces fully annotated images and second pseudo-pixel-level labels of fully annotated images, and trains the fully convolutional network based on the fully annotated images and the second pseudo-pixel-level labels. During the training process, the cross-entropy loss function is introduced to determine the predicted difference corresponding to the cross-entropy loss function, and the back-propagation algorithm is introduced to update the network parameters of the fully convolutional network in real time, thereby improving the rationality of the network parameter setting of the fully convolutional network. Determine the target fully convolutional network whose predicted difference is less than or equal to the difference threshold, input the first pseudo-pixel-level label into the target fully convolutional network to obtain the initial segmentation result, and further improve the accuracy of the initial segmentation result.

[0123] The embodiment of the present invention provides a weakly supervised pathological image segmentation system based on frame-level annotation, further comprising:

[0124] The report generation subsystem is used to obtain input information from the target display party after viewing the segmentation results, and generate a pathology diagnosis report based on the input information; wherein the target display party is, for example, a doctor who receives the patient; the input information is: the input operation of the doctor who receives the patient in the report generation subsystem;

[0125] Among them, the report generation subsystem includes:

[0126] An input template library acquisition module is used to acquire an input template library; wherein the input template library contains input framework templates of multiple diagnosis reports of different pathological types;

[0127] A recommended input template list determination module is used to determine a recommended input template list according to the segmentation result and the input template library;

[0128] A target input template acquisition module is used to acquire a target input template selected by a target display party from a recommended input template list, and display the target input template to the target display party;

[0129] The operation information acquisition module is used to acquire the operation information of each operable item in the target input template of the target display party; wherein the operable item is: the editable part in the target input template; the operation information is, for example: inputting "...", checking something, marking something, etc.;

[0130] The input item determination module is used to determine the input item in the operable item according to the operation instruction library and operation information corresponding to the operable item; wherein the operation instruction library has operation instructions corresponding to the operation instruction library, and the operable item will present different contents according to different operation instructions, and the operation instruction corresponding to the operable item is determined, and the content presented corresponding to the operation instruction is used as the input item, for example: a 2mm tumor is found in a certain part;

[0131] The input information acquisition module is used to integrate the input items and the target input template to obtain the input information.

[0132] The working principle and beneficial effects of the above technical solution are:

[0133] Generally, after the doctor has reviewed the pathological image segmentation results, he needs to issue a diagnosis report. However, it is inefficient and inconvenient to write all the diagnosis reports manually. Therefore, this application introduces an input template library, determines a recommended input template list based on the segmentation results and the input template library, and determines the target input template in the recommended input template list to display to the target display party. According to the operation information of each operable item in the acquired target input template and the operation instruction library corresponding to the operable item, the input item in the operable item is determined, and the input item and the target input template are integrated to obtain the input information. The acquisition of input information is faster and more standardized, and further, the generation efficiency of pathological diagnosis reports is greatly improved.

[0134] In one embodiment, the recommended input template list determination module includes:

[0135] The auxiliary diagnosis model acquisition submodule is used to acquire the auxiliary diagnosis model; wherein the auxiliary diagnosis model is: an AI model that replaces manual auxiliary diagnosis of pathology based on the pathology image segmentation results;

[0136] A first auxiliary diagnosis information acquisition submodule is used to input the segmentation result into the auxiliary diagnosis model to obtain first auxiliary diagnosis information; wherein the first auxiliary diagnosis information is: diagnosis information output by the auxiliary diagnosis model based on the segmentation result;

[0137] A pre-selected input template acquisition submodule is used to parse the input template library and obtain multiple pre-selected input templates;

[0138] The second auxiliary diagnosis information acquisition submodule is used to acquire preset second auxiliary diagnosis information corresponding to the preselected input template; wherein the second auxiliary diagnosis information is: diagnosis information of the pathology diagnosed corresponding to the preselected input template;

[0139] The information similarity calculation submodule is used to calculate the information similarity between the first auxiliary diagnosis information and the second auxiliary diagnosis information; wherein the information similarity is, for example, 0.9;

[0140] The template sorting acquisition submodule is used to sort the pre-selected input templates according to the information similarity, and obtain the template sorting of the pre-selected input templates; wherein the template sorting is: which pre-selected input template is sorted first, and which pre-selected input template is sorted next;

[0141] The recommended input template list construction submodule is used to construct a recommended input template list based on the preset recommended input template list construction rules and the template sorting of the pre-selected input templates. The recommended input template list construction rules are as follows: the diagnostic disease types corresponding to the output of the pre-selected input templates are displayed on the PC side according to the corresponding template sorting.

[0142] The working principle and beneficial effects of the above technical solution are:

[0143] This application introduces an auxiliary diagnosis model, obtains first auxiliary diagnosis information based on the segmentation result and the auxiliary diagnosis model, and at the same time obtains second auxiliary diagnosis information corresponding to the pre-selected input template, sorts the pre-selected input template according to the calculated information similarity between the first auxiliary diagnosis information and the second auxiliary diagnosis information to determine the template sorting, and the template sorting is more appropriate. The recommended input template list is constructed according to the recommended input template list construction rules and template sorting, and the construction of the recommended input template list is more reasonable.

[0144] In one embodiment, the auxiliary diagnosis model acquisition submodule includes:

[0145] A first process record acquisition unit is used to acquire a first process record of a target doctor viewing the pathological image segmentation information for pathological diagnosis; wherein the pathological image segmentation information is: a presentation screen of the segmentation result; the first process record is: a video shot of the process of the target doctor viewing the pathological image segmentation information for pathological diagnosis, the video shot of the process being acquired by a camera provided on a display device where the pathological image segmentation information is located;

[0146] A visual dwell time determination unit is used to determine, based on the attention analysis technology and according to the first process record, the visual dwell time of the target doctor in the first segmented area of ​​the segmented image corresponding to the first process record; wherein the first segmented area is: any segmented area in the segmented image; the visual dwell time is: the length of time the target doctor's sight stays in the first segmented area, for example: 8 seconds;

[0147] A region type acquisition unit, used to acquire the region type of the first segmented region; wherein the region type is, for example, cell nucleus, cytoplasm, cell fluid, etc.;

[0148] An analysis importance value determination unit is used to determine the analysis importance value of a region type according to a preset region type-analysis importance value library; wherein the region type-analysis importance value library has a one-to-one correspondence between region types and analysis importance values, and the higher the analysis importance value, the longer the target doctor should view and analyze the first analysis region of the corresponding region type;

[0149] An analysis time ideal value determination unit, used to determine the analysis time ideal value of the first segmented area according to a preset analysis importance value-analysis time ideal value library; wherein the preset analysis importance value-analysis time ideal value library includes a plurality of one-to-one corresponding analysis importance values ​​and analysis time ideal values;

[0150] A reference value calculation unit is used to divide the visual stay time of the first segmented area by the ideal value of the analysis time of the first segmented area to obtain a reference value, and associate it with the corresponding first process record; wherein the visual stay time is, for example, 8 seconds; the ideal value of the analysis time is, for example, 10 seconds, and the reference value is: 0.8;

[0151] A target and value calculation unit, used to calculate an average value of the reference value associated with the first process record and use it as a target value; wherein the larger the target value, the more likely the corresponding first process record is to be used as training data for the auxiliary diagnosis model;

[0152] A second process record determination unit, configured to use the corresponding first process record as the second process record if the target value is greater than or equal to a preset target value threshold; wherein the target value threshold is preset manually;

[0153] The auxiliary diagnosis model training unit is used to train the second process record based on deep learning technology to obtain an auxiliary diagnosis model.

[0154] The working principle and beneficial effects of the above technical solution are:

[0155] This application introduces attention analysis technology to determine the visual dwell time of the target doctor in the first segmented area of ​​the segmented image in the acquired first process record. At the same time, the region type of the first segmented area is obtained, and the ideal value of the analysis time of the first segmented area is determined according to the introduced region type-analysis important value library and analysis important value-analysis time ideal value library. The visual dwell time of the first segmented area is divided by the ideal value of the analysis time of the first segmented area to obtain a reference value and associate it with the first process record, thereby improving the rationality of the reference value acquisition. The target value determined by calculating the average value of the reference value associated with the first process record is compared with the preset target value threshold to determine the second process record suitable for training, thereby improving the accuracy of the auxiliary diagnosis model for auxiliary diagnosis.

[0156] The embodiment of the present invention provides a weakly supervised pathological image segmentation method based on frame-level annotation, such as Figure 2 As shown, including:

[0157] Step 1: Collect target pathological sections;

[0158] Step 2: Send the target pathological slice to the preset annotation node for frame-level annotation, and obtain the annotated image replied by the annotation node;

[0159] Step 3: Based on the WSSS algorithm, perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result.

[0160] In one embodiment, step 2: sending the target pathological slice to a preset annotation node for frame-level annotation, and obtaining an annotated image replied by the annotation node, includes:

[0161] Get the node attributes of the annotation node;

[0162] Determine the node type of the labeled node according to the node attributes;

[0163] If the node type is a manually annotated node, obtain the first sub-annotated image;

[0164] If the node type is a smart annotation node, obtain the second sub-annotation image;

[0165] The first sub-annotated image and the second sub-annotated image are used together as the annotated image.

[0166] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A weakly supervised pathological image segmentation system based on box-level annotation, characterized by: include: Pathological section acquisition subsystem, used for acquiring target pathological sections; The frame-level annotation subsystem is used to send the target pathological slice to the preset annotation node for frame-level annotation and obtain the annotated image replied by the annotation node; The segmentation subsystem is used to perform weakly supervised semantic segmentation on the annotated image based on the WSSS algorithm to obtain the segmentation result.

2. The weakly supervised pathological image segmentation system based on frame-level annotation as claimed in claim 1, characterized in that: The frame-level annotation subsystem includes: A node attribute acquisition module is used to obtain the node attributes of the labeled node; A node type determination module is used to determine the node type of the labeled node according to the node attributes; A first sub-annotated image acquisition module, used for acquiring a first sub-annotated image if the node type is a manually annotated node; A second sub-annotation image acquisition module, used for acquiring a second sub-annotation image if the node type is an intelligent annotation node; The annotated image integration module is used to use the first sub-annotated image and the second sub-annotated image together as an annotated image.

3. The weakly supervised pathological image segmentation system based on frame-level annotation as claimed in claim 2, characterized in that: The second sub-annotation image acquisition module includes: A feature of interest extraction submodule is used to extract the feature of interest of the target pathological section based on a preset feature of interest extraction template; The object bounding box annotation model training submodule is used to train the object bounding box annotation model according to the features of interest based on the object detection algorithm and the preset deep learning model; A target bounding box annotation submodule is used to annotate a target bounding box in a target pathological slice according to a target bounding box annotation model; A key point annotation template determination submodule is used to determine a key point annotation template according to a feature type of a feature of interest corresponding to a target bounding box; The key point annotation submodule is used to annotate key points within the target bounding box according to the key point annotation template; A pre-verification image determination submodule is used to use the corresponding target pathological slice as a pre-verification image after all target bounding boxes and annotation key points are annotated; The second sub-annotated image determination submodule is used to send the pre-verified image to a preset verification personnel node and obtain the second sub-annotated image verified by the verification personnel node.

4. The weakly supervised pathological image segmentation system based on frame-level annotation as claimed in claim 1, characterized in that: Segmentation subsystem, including: A first pseudo pixel-level label generation module, used to generate a first pseudo pixel-level label according to the annotation information of the annotated image; An initial segmentation result acquisition module is used to perform initial segmentation on the target pathological image based on the image region proposal technology and the first pseudo pixel-level label to obtain an initial segmentation result; A preprocessing segmentation result acquisition module is used to perform training and semantic segmentation on the labeled image based on the WSSS algorithm, according to a preset full convolutional network and a first pseudo pixel-level label, to obtain a preprocessing segmentation result; The processing module is used to process the preprocessing segmentation result to obtain the segmentation result.

5. The weakly supervised pathological image segmentation system based on frame-level annotation as claimed in claim 4, characterized in that: The preprocessing segmentation result acquisition module includes: A fully annotated image acquisition submodule is used to acquire a fully annotated image and, at the same time, acquire a second pseudo pixel-level label of the fully annotated image; A cross entropy loss function acquisition submodule is used to train a full convolutional network based on a WSSS algorithm according to a full annotated image and a second pseudo pixel-level label corresponding to the full annotated image, and to acquire a cross entropy loss function during the training process of the full convolutional network; A prediction difference determination submodule, for determining the prediction difference based on a cross entropy loss function; The network parameter update submodule is used to update the network parameters of the trained fully convolutional network in real time according to the prediction difference based on the back-propagation algorithm; The target full convolutional network acquisition submodule is used to acquire the current full convolutional network and use it as the target full convolutional network when the predicted difference is less than or equal to a preset difference threshold; The initial segmentation result determination submodule is used to input the first pseudo pixel-level label into the target full convolutional network to obtain an initial segmentation result.

6. The weakly supervised pathological image segmentation system based on frame-level annotation as claimed in claim 1, characterized in that: Also includes: The report generation subsystem is used to obtain the input information of the target display party after viewing the segmentation results, and generate a pathology diagnosis report based on the input information; Among them, the report generation subsystem includes: An input template library acquisition module is used to acquire an input template library; A recommended input template list determination module is used to determine a recommended input template list according to the segmentation result and the input template library; A target input template acquisition module is used to acquire a target input template selected by a target display party from a recommended input template list, and display the target input template to the target display party; An operation information acquisition module is used to acquire the operation information of each operable item in the target input template of the target display party; An input item determination module, used to determine an input item in the operable item according to an operation instruction library and operation information corresponding to the operable item; The input information acquisition module is used to integrate the input items and the target input template to obtain the input information.

7. The weakly supervised pathological image segmentation system based on frame-level annotation as claimed in claim 6, characterized in that: The recommended input template list determines the module, including: Auxiliary diagnosis model acquisition submodule, used to obtain auxiliary diagnosis model; A first auxiliary diagnosis information acquisition submodule, used for inputting the segmentation result into the auxiliary diagnosis model to obtain first auxiliary diagnosis information; A pre-selected input template acquisition submodule is used to parse the input template library and obtain multiple pre-selected input templates; The second auxiliary diagnosis information acquisition submodule is used to acquire preset second auxiliary diagnosis information corresponding to the preselected input template; An information similarity calculation submodule, used to calculate the information similarity between the first auxiliary diagnosis information and the second auxiliary diagnosis information; The template sorting acquisition submodule is used to sort the pre-selected input templates according to the degree of information similarity to obtain the template sorting of the pre-selected input templates; The recommended input template list building submodule is used to build a recommended input template list based on preset recommended input template list building rules and according to the template sorting of the pre-selected input templates.

8. The weakly supervised pathological image segmentation system based on frame-level annotation as claimed in claim 7, characterized in that: The auxiliary diagnosis model acquisition submodule includes: A first process record acquisition unit, used to acquire a first process record of a target doctor viewing the pathological image segmentation information to perform pathological diagnosis; a visual dwell time determination unit, configured to determine, based on the attention analysis technology and according to the first procedure record, a visual dwell time of the target doctor in the first segmented area in the segmented image corresponding to the record in the first procedure record; A region type acquisition unit, used to acquire the region type of the first segmented region; An analysis importance value determination unit, used to determine the analysis importance value of the region type according to a preset region type-analysis importance value library; An analysis time ideal value determination unit, used to determine the analysis time ideal value of the first segmented area according to a preset analysis importance value-analysis time ideal value library; A reference value calculation unit, used for dividing the visual dwell time of the first segmented area by the ideal value of the analysis time of the first segmented area, obtaining a reference value, and associating it with the corresponding first process record; A target and value calculation unit, used for calculating an average value of the reference value associated with the first process record and using it as a target value; A second process record determination unit, configured to use the corresponding first process record as the second process record if the target value is greater than or equal to a preset target value threshold; The auxiliary diagnosis model training unit is used to train the second process record based on deep learning technology to obtain an auxiliary diagnosis model.

9. A weakly supervised pathological image segmentation method based on frame-level annotation, characterized in that: include: Step 1: Collect target pathological sections; Step 2: Send the target pathological slice to the preset annotation node for frame-level annotation, and obtain the annotated image replied by the annotation node; Step 3: Based on the WSSS algorithm, perform weakly supervised semantic segmentation on the annotated image to obtain the segmentation result.

10. The weakly supervised pathological image segmentation method based on frame-level annotation as claimed in claim 9, characterized in that: Step 2: Send the target pathological slice to the preset annotation node for frame-level annotation, and obtain the annotated image replied by the annotation node, including: Get the node attributes of the annotation node; Determine the node type of the labeled node according to the node attributes; If the node type is a manually annotated node, obtain the first sub-annotated image; If the node type is a smart annotation node, obtain the second sub-annotation image; The first sub-annotated image and the second sub-annotated image are used together as the annotated image.

Images