Expert guided improved image segmentation

Abstract

Some embodiments relate to machine-learnable image segmentation. The image segmentation model may be applied to a set of input images to obtain a corresponding set of segmented images. The corrected segmented image obtained from the expert determination may be used to train another image segmentation model that predicts an expected error image.

Description

Technical Field

[0001] The subject matter of the present disclosure relates to a method for a machine-learnable image segmentation model, an apparatus for a machine-learnable image segmentation model, and a computer-readable medium. Background Art

[0002] Medical image analysis, such as segmentation and classification for diagnosis, has achieved remarkable performance. For example, deep learning based models can produce high-quality medical image segmentations. However, there are still images for which a given image segmentation model may give the wrong answer. Experiments on image classification have shown that despite high confidence in the model predictions, the trained model can sometimes fail on specific instances.

[0003] As pointed out in the paper "Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods" by Eyke Hülermeier and Willem Waegeman, two sources of uncertainty can be identified: aleatoric uncertainty and epistemic uncertainty. According to these authors, aleatoric (also called statistical) uncertainty refers to the concept of randomness, i.e., the variability of experimental results due to inherent random effects. Epistemic (also called systematic) uncertainty refers to the uncertainty caused by lack of knowledge. In principle, epistemic uncertainty can be reduced using additional information.

[0004] Indeed, an important aspect of the success of these new image segmentation models is the quality of the ground truth training data used to train the image models. To achieve the best performance, human domain experts evaluate a large number of images to improve the training and refinement of the models.

[0005] It would be advantageous to obtain more high-quality data, thereby reducing epistemic uncertainty in image segmentation models. Summary of the invention

[0006] A first possible workflow to get better training data is to use a partially trained image segmentation model. This partially trained image segmentation model can be applied to a set of images, thereby obtaining a corresponding set of segmented images. Now, a library of experts can correct the segmented images where needed, without having to fully segment them by hand. The corrected segmented images thus obtained can then be used to train a better model or to further train an existing model.

[0007] This workflow is more efficient because the applicator does not have to segment those parts of the image that the model already gets right. As the model gets better, it tends to be right more often, so the expert’s work is significantly reduced.

[0008] A disadvantage of this first workflow is that the expert needs to carefully evaluate the segmentation errors proposed by the model. In the second workflow, the (partially trained) image segmentation model not only calculates the segmentation image, but also an uncertainty image for the input image, which indicates the confidence of the image segmentation model in different parts of the segmented image. Methods for calculating such uncertainty images are known per se; some of them are cited herein. In the second workflow, the expert will be faced with the segmentation image produced by the image segmentation model and the corresponding uncertainty image. The expert is now guided to the parts of the image that are most likely to be erroneous, so that the segmentation proposed by the model can be evaluated more quickly. As in the first workflow, the expert will correct the segmentation if necessary, for example for training purposes.

[0009] Although the second workflow produces high-quality training data in a shorter time, the inventors found that the second workflow is still not optimal. The problem they observed is that the uncertainty image tends to mark large parts of the image as uncertain, even if these parts of the image will not actually be corrected. This negates the advantage of the uncertainty image in focusing the expert's attention on the parts of the image that need to be verified. Another problem is that the edges of objects in the uncertainty image often have an uncertainty band around them. One possible reason for this band is that the exact edges of the object are often unclear and different experts may annotate them differently in different images. Although the uncertainty image correctly indicates that the exact boundaries of the image are uncertain, the boundaries are still unlikely to be corrected. The expert evaluating the image may find the location of the object boundaries satisfactory, or he / she may not have a better judgment about these boundaries.

[0010] The embodiment proposes that in addition to learning the first image segmentation model of the desired segmentation, a second image segmentation model is also trained. The second image segmentation model is trained based on the corrections made by the expert to the segmentation produced by the first image segmentation model. In the improved workflow, the expert may be faced with a segmented image produced by the first image segmentation model and a corresponding expected error image calculated by the second image segmentation model.

[0011] This second image segmentation model has advantages over using uncertainty images as discussed for the second workflow. Expected error images tend to flag fewer images as potentially defective compared to uncertainty images. Expected error images can better focus the expert's attention on segmentation regions where true errors may exist.

[0012] Furthermore, the error image is expected to be less affected by uncertainty bands or halos that may appear around objects. Since the exact boundaries of objects tend not to be corrected by the expert, i.e., such correction is often not needed, the second image segmentation model will correctly learn that these boundaries may not shift. This also helps to focus the expert's attention on the parts that are most likely to need review. Therefore, both aspects help to reduce the review time of the model's predicted segmentation.

[0013] In a third workflow, another image segmentation model that predicts an expected error image is used by having it predict an expected error image for an input image, and showing both the segmented image and the expected error image generated by the image segmentation model to an expert. The expected error image will focus on the parts that are most likely to need correction. Therefore, the expert can correct the segmentation faster. This is an improved workflow for obtaining image segmentation. In particular, for medical image segmentation, the time for segmenting and labeling images is reduced compared to the first and second workflows described above. The corrected segmentation can be used for clinical purposes, such as diagnosis, evaluation, treatment planning, etc. Another image segmentation model that predicts an expected error image can be used by having it predict an expected error image for an input image and showing both the segmented image and the expected error image generated by the image segmentation model to an expert. The expected error image can be used to train an image segmentation model or to train another image segmentation model.

[0014] For example, in an embodiment, an image segmentation model is configured to receive an input image and generate as an output a segmented image that classifies portions of the input image. Another image segmentation model that predicts an expected error image of the input image can be obtained by training the image segmentation model on an error image, the error image indicating where the segmented image is corrected by expert judgment. The method is applicable to medical segmentation models, but may also be applied to other fields.

[0015] A knowledge graph is a useful data structure for tracking the various images and other data that appear in an embodiment.

[0016] In an embodiment, the associated uncertainty model is configured to generate an uncertainty image of the input image, the uncertainty image indicating the confidence of the image segmentation model in different parts of the segmented image. Such an uncertainty image can be obtained without access to another image model that has been trained. It is found to be advantageous to provide the uncertainty image as input to another image model.

[0017] Embodiments of the method may be implemented on a computer as a computer-implemented method, or in dedicated hardware, or in a combination of the two. Executable code for embodiments of the method may be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product includes non-transitory program code stored on a computer-readable medium for performing embodiments of the method when the program product is executed on a computer.

[0018] In an embodiment, the computer program comprises computer program code adapted to perform all or part of the steps of an embodiment of the method when the computer program is run on a computer.Preferably, the computer program is embodied on a computer readable medium.

[0019] Another aspect of the disclosed subject matter is a method of making a computer program available for downloading. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Further details, aspects and embodiments will be described by way of example with reference to the accompanying drawings. The elements in the drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale. In the drawings, elements corresponding to elements already described may have the same reference numerals. In the drawings,

[0021] Figure 1 schematically illustrates an example of an embodiment of an image segmentation device,

[0022] Figure 2a schematically illustrates an example of an embodiment of an image segmentation system,

[0023] Figure 2b schematically illustrates an example of an embodiment of an image segmentation system,

[0024] Figure 2c schematically illustrates an example of an embodiment of an image segmentation system,

[0025] Figure 2d schematically illustrates an example of an embodiment of a data structure,

[0026] Figure 2e schematically illustrating an example of an embodiment of a knowledge graph,

[0027] Figure 3a An example of an embodiment of a medical image is shown,

[0028] Figure 3b schematically illustrates an example of an embodiment of a medical image,

[0029] Figure 3c schematically illustrates an example of an embodiment of a segmented image,

[0030] Figure 3d schematically illustrates an example of an embodiment of an uncertainty image,

[0031] Figure 3e schematically illustrates an example of an embodiment of a corrected segmented image,

[0032] Figure 3f Schematically illustrating an example of an embodiment of an error image,

[0033] Figure 3g An example of an embodiment of an expected error image is schematically shown.

[0034] Figure 4 schematically illustrates an example of an embodiment of an image segmentation method,

[0035] Figure 5 schematically illustrates an example of an embodiment of an image segmentation method,

[0036] Figure 6a schematically shows a computer-readable medium having a writable portion comprising a computer program according to an embodiment,

[0037] Figure 6b A representation of a processor system according to an embodiment is schematically shown.

[0038] Reference numerals list

[0039] The following list of reference numerals refers to Figure 1 , 2a -2e, 3a-3g, 6a and 6b are intended to facilitate the explanation of the drawings and should not be construed as limiting the claims.

[0040] 110 Image segmentation equipment

[0041] 112 Image Database

[0042] 130 processor system

[0043] 140 Storage Devices

[0044] 150 communication interface

[0045] 200 Image Segmentation System

[0046] 210 Image Segmentation Model

[0047] 211 segmented image

[0048] 212 Uncertainty Image

[0049] 213 Input image

[0050] 215 Associated uncertainty models

[0051] 220Another image segmentation model

[0052] 221Expected error image

[0053] 230 Expert judgment interface

[0054] 231Corrected segmented image

[0055] 232 Error Image

[0056] 233Further corrected segmented image

[0057] 241First Model Trainer

[0058] 242 Second model trainer

[0059] 250 EXPERT SELECT UNIT

[0060] 260 Knowledge Graph

[0061] 261-263 Additional Data

[0062] 265 Data Structure

[0063] 300 medical images

[0064] 301 Schematic rendering of image 300

[0065] 310 Segmented Image

[0066] 320 Uncertainty Image

[0067] 321 First Area

[0068] 322 Second Area

[0069] 330 Corrected segmented image

[0070] 331 Correction Section

[0071] 340 error image,

[0072] 341 Correction Section

[0073] 350Expected error image

[0074] 1000, 1001 Computer readable medium

[0075] 1010 Writable part

[0076] 1020 Computer Programs

[0077] 1110(s) integrated circuit

[0078] 1120 Processing Unit

[0079] 1122 Memory

[0080] 1124 ASIC

[0081] 1126 Communication Components

[0082] 1130 Interconnect

[0083] 1140 processor system DETAILED DESCRIPTION

[0084] Although the subject matter of the present disclosure may be embodied in many different forms, one or more specific embodiments are shown in the drawings and will be described in detail herein, it being understood that the present disclosure is to be considered as an example of the principles of the subject matter of the present disclosure and is not intended to be limited to the specific embodiments shown and described.

[0085] Hereinafter, for the sake of understanding, the elements of the embodiments are described in operation. However, it will be apparent that the respective elements are arranged to perform the functions described by them. In addition, the subject matter of the present disclosure is not limited to the embodiments, but also includes every other combination of the features described herein or cited in mutually different dependent claims.

[0086] The success of image segmentation models is based in part on advanced segmentation algorithms (e.g., based on deep convolutional neural networks) and in part on accurate training data. Advanced segmentation models, especially for medical image segmentation, still require large, representative, and high-quality annotated datasets. However, obtaining perfect training datasets is rare, especially in the field of medical imaging, where the cost of acquiring data and annotations is high. In the radiology workflow, automated AI medical image segmentation using partially trained algorithms and human-assisted correction using editing tools can help improve the quality of training datasets. This workflow reduces the need for secondary reading and / or manual supervision by radiologists and improves productivity.

[0087] Medical images such as MRI, CT, etc. are the primary modalities for diagnosing various medical conditions ranging from tumors to ligament tears. Medical image labeling is not only a key requirement for supervised machine learning and deep learning models, but is also used for clinical purposes.

[0088] For example, one of the challenges involves detecting accidental and epistemic uncertainty in machine learning, unavailability of representative data, generalization failure of annotated models (especially in multi-site settings), unavailability of expert radiologists, intra- and inter-annotator variability due to different skill levels of radiologists. Radiologists may need to perform labor-intensive corrections to the segmentation masks automatically predicted by the AI ​​model.

[0089] Segmentation uncertainty can originate from different source types, including physical equipment issues, annotation tool issues, hidden features in the model, or incorrect decision boundaries. These uncertainties can be broadly divided into aleatory and epistemic uncertainties. Aleatory uncertainty involves the objective or physical concept of uncertainty inherent in the data. The source of aleatory uncertainty is often unclear. Epistemic uncertainty involves the subjective or personal concept of uncertainty. It occurs due to lack of knowledge or neglect of certain features during the data generation process or training the model.

[0090] Improved methods for improving the performance of AI image segmentation models are described herein in various embodiments. For example, embodiments may use uncertainty predictors used in intelligent annotation workflows. Knowledge graphs may be advantageously used to organize data in a system.

[0091] In traditional intelligent annotation scenarios, image segmentation models can generate approximate segmented images, also called segmented images. Then, editing tools are provided to experts (e.g., radiologists) to correct (one or more) automatically segmented regions.

[0092] According to the intelligent annotation scenario of the embodiment, the image segmentation model can generate an approximate segmentation image. The segmentation image and the expected error image obtained from the model based on the previous correction training are shown to the expert. The expert is provided with editing tools to correct the automatically segmented area; his attention is guided by the expected error image to reduce the review time.

[0093] In an embodiment, a model is trained to learn and model uncertainty corrections from human annotators. For example, such data can be stored in a flexible data structure such as a knowledge graph. A neural network trained with the corrections can be used to predict uncertain areas in an image. These uncertain images are later used to correct or evaluate image segmentation models and / or improve the productivity of expert annotators. Intelligent annotations automatically populated with high certainty reduce expert assistance, reduce secondary reading and / or manual supervision by radiologists, and improve productivity.

[0094] Figure 1 An example of an embodiment of an image segmentation device 110 is schematically shown. The image segmentation device 110 is configured for a machine learnable image segmentation model. For example, the device 110 can be used to apply the image segmentation model to one or more input images to obtain a corresponding segmented image. Such a device is useful in many fields, especially in the medical field, for example when applied to medical images. Other fields include, for example: manufacturing, where image segmentation can be used to detect faults in manufactured products; assisted or autonomous driving, where image segmentation can be used to detect objects in street images.

[0095] The device 110 may be configured to obtain a set of corrected segmented images and a corresponding set of error images, the set of corrected segmented images correcting the set of segmented images according to expert judgment, the error images indicating where the segmented images were corrected; and train another image segmentation model that predicts an expected error image 221 for an input image. Such a trained another image segmentation model is advantageous because it better indicates where expert review is useful. Embodiments may be applied to image segmentation models in various fields;

[0096] Medical segmentation models are a good example.

[0097] The device 110 may be configured to train the other image segmentation model, but this is not required. In a useful embodiment, the device 110 is alternatively (or additionally) configured to apply another image segmentation model that has been trained, for example, on a different device. For example, a pre-trained error prediction model may be obtained from a third party as a useful supplement to the semi-autonomous segmentation workflow. The corrected segmentation produced by the expert in the semi-autonomous segmentation workflow can be advantageously used to further train both the image segmentation model and the other image segmentation model, but this is not required.

[0098] The image segmentation device 110 may include a processor system 130, a storage device 140, and a communication interface 150. The storage device 140 may be, for example, an electronic storage device, a magnetic storage device, etc. The storage device may include a local storage device, such as a local hard drive or an electronic memory. The storage device 140 may include a non-local storage device, such as a cloud storage device. In the latter case, the storage device 140 may include a storage interface to the non-local storage device. The storage device may include a plurality of discrete sub-storage devices, which together constitute the storage device 140. The storage device may include a volatile writable portion (e.g., RAM), a non-volatile writable portion (e.g., flash memory), and a non-volatile non-writable portion (e.g., ROM).

[0099] The storage device 140 may be a non-transitory storage device. For example, the storage device 140 may store data while power is on, such as a volatile memory device, such as a random access memory (RAM). For example, the storage device 140 may store data while power is on and when power is off, such as a non-volatile memory device, such as a flash memory.

[0100] Device 110 may be connected to a database 112. Database 112 may be internal or external to device 110. For example, device 110 may store input images, segmentations, corrections, etc. Database 112 may be shared by multiple devices 110, for example, to facilitate multiple experts to work in parallel, for example, to improve training data in parallel.

[0101] Device 110 can communicate internally with other systems, other devices, external storage devices, input devices, output devices and / or one or more sensors through a computer network. The computer network can be the Internet, an intranet, a local area network, a wireless local area network, etc. The computer network can be the Internet. Device 110 includes a connection interface arranged to communicate within system 100 or outside system 100 as needed. For example, the connection interface may include a connector, such as a wired connector, such as an Ethernet connector, an optical connector, etc.; or a wireless connector, such as an antenna, such as a Wi-Fi, 4G or 5G antenna.

[0102] The communication interface 150 may be used to send or receive digital data, for example, receive an input image, receive a correction, send a segmented image, a correction image, an uncertainty image, an error image, etc.

[0103] The execution of the device 110 can be implemented in a processor system. The device 110 may include functional units for implementing various aspects of the embodiment. The functional units may be part of the processor system. For example, the functional units shown herein may be implemented in whole or in part in computer instructions stored in a storage device of the device and executable by the processor system.

[0104] The processor system may include one or more processor circuits, such as a microprocessor, a CPU, a GPU, etc. The device 110 may include multiple processors. The processor circuit may be implemented in a distributed manner, for example, as multiple sub-processor circuits. For example, the device 110 may use cloud computing. In an embodiment, the device 110 is a locally integrated device. In an embodiment, the device 110 is distributed in multiple geographically different locations.

[0105] Typically, the image segmentation device 110 includes a microprocessor that executes appropriate software stored at the device; for example, the software may have been downloaded and / or stored in a corresponding memory, such as a volatile memory such as RAM or a non-volatile memory such as flash memory.

[0106] Instead of using software to implement the functions, the device 110 may be implemented in whole or in part in programmable logic, for example as a field programmable gate array (FPGA). The device may be implemented in whole or in part as a so-called application specific integrated circuit (ASIC), for example an integrated circuit (IC) customized for its specific purpose. For example, the circuit may be implemented in CMOS, for example using a hardware description language such as Verilog, VHDL, etc. In particular, the image segmentation device 110 may include circuits for neural network processing and / or arithmetic processing, for example.

[0107] Execution of the image segmentation apparatus and / or method may be implemented in a processor circuit, examples of which are shown herein. Figure 2a-2c Functional units that may be functional units of a processor circuit are shown. For example, the drawings may be used as blueprints of possible functional organizations of a processor circuit. In addition to the organizations shown in these figures, other organizations are possible.

[0108] Processor circuitry is not shown separately from the units in these figures. For example, the functional units may be implemented in whole or in part in computer instructions stored at device 100 (e.g., in an electronic memory of device 100) and executable by a microprocessor of device 100. In hybrid embodiments, the functional units are implemented partially in hardware, e.g., as a coprocessor, such as a neural network coprocessor, and partially in software stored and executed on the device.

[0109] Figure 2a Schematically shown is an example of an embodiment of an image segmentation system 200. The image segmentation system 200 may be implemented, for example, in a device such as the device 110. For example, in an embodiment, a workstation or an imaging apparatus comprises the system 200.

[0110] The system 200 is configured with an image segmentation model 210. The image segmentation model 210 is configured to input an image and produce as output a segmented image that classifies portions of the input image. Figure 2a An input image 213 and a segmented image 211 are shown.

[0111] The image segmentation model is advantageously a medical segmentation model; the image segmentation model 213 may also be applied to different types of images, for example, manufactured product images, street images, etc.

[0112] The input image 213 may be obtained from various medical imaging techniques. The input image 213 may be multi-dimensional image data, such as a two-dimensional (2D), three-dimensional (3D), or four-dimensional (4D) image. The input image 213 may be acquired by various acquisition modalities, such as but not limited to standard X-ray imaging, full-field digital mammography, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), positron emission tomography (PET), single photon emission computed tomography (SPECT), and nuclear medicine (NM). The image segmentation model 210 may be configured for one or more of these image modalities.

[0113] The image segmentation model 210 is at least partially trained, i.e., the image segmentation model 210 is indeed capable of generating the segmented image 211. However, the image segmentation model 210 does not need to be fully trained. As a result, some errors in the segmentation of the image segmentation model 210 are acceptable. In particular, errors due to imperfect training material. Embodiments can be used to improve and / or obtain training data with fewer errors.

[0114] Image segmentation can include identifying regions of interest (ROIs) in image data. Medical image segmentation can be used, for example, for computer-aided diagnosis. Image segmentation can, for example, segment regions in an image, such as segmenting body organs, tissues, tumor detection and / or segmentation, and mass detection. For example, an image segmentation model can segment a tumor in an image of a patient's breast. Image segmentation can also be used to simulate physical properties on a computer, or to virtually position a CAD-designed implant in a patient's body.

[0115] For example, in an embodiment, a medical image, such as an MRI image, may be obtained, and an image segmentation model may be applied to the image to obtain a segmented image. The segmented image indicates segmentation into one or more regions of interest, for example, to separate breast tissue and a tumor therein. The segmented image may include or be associated with a label indicating the type of segmentation. The resulting segmentation may be used for diagnosis, planning potential treatment, etc. Segmentation of medical images may be helpful, for example, in examining the growth of tumors, controlling drug dosages, and radiation exposure doses.

[0116] For example, MRI planning may involve aligning box geometry with anatomical features.Using image segmentation models, on-the-fly box planning may be performed based on, for example, a scout scan or a previous scan.

[0117] The image segmentation model 210 is a machine-learnable model that can be trained and / or refined on training data, for example, including a plurality of training items. The training items can include sample images and desired results, for example, segmentations that the model should produce.

[0118] Image segmentation models can be based on various artificial intelligence techniques. Segmentation techniques include classification and clustering methods, such as nearest neighbor models, support vector machines (SVM), and artificial neural networks (ANN), in particular convolutional neural networks. In particular, image segmentation based on neural networks shows high promise in segmentation. Neural networks can be improved by using higher quality training sets, although any machine-learnable model that relies on training data can be improved by improving the quality of the training data. Image segmentation models can include convolutional neural networks (CNNs), for example, neural networks that include one or more convolutional layers. For example, image segmentation models can include a so-called u-net.

[0119] Segmentation can be for a specific element. For example, in an embodiment, an image segmentation model can be trained to segment only tumor tissue in an image (e.g., in a mammogram). Segmentation can be for multiple elements. For example, in an embodiment, an image segmentation model can be trained to segment different kinds of tissues, organs, etc. in an image. For example, this can be achieved by generating multiple images, for example, one image for each label, or by associating each pixel or voxel with a label vector indicating which label is applied or a probabilistic label vector indicating the probabilities of various labels, etc. Segmentation of multiple labels can also be indicated with visual elements, such as color or pattern fills, boundary curves, etc.

[0120] In an embodiment, the system 200 is configured to apply the image segmentation model 210 to a set of input images to obtain a corresponding set of segmented images. Figure 2a In FIG. 2 , an input image 213 and a segmented image 211 are shown. The set of segmented images may be partially correct and partially wrong.

[0121] A set of corrected segmented images is obtained for the set of input images. The correction is obtained through expert judgment, which may be a technician in a particular field of interest, such as a radiologist. Possible expert judgment may be a different artificial intelligence (not shown) trained in some other way, for example, obtained from a third party and used to train the image model 210. For example, the different model may be a previous generation image segmentation model with high quality segmentation, but for some reason will be replaced by the image model 210. We will assume in this article that the expert judgment is obtained from an expert, but the embodiments can be modified to rely on other types of expert judgment.

[0122] For example, the system 200 may include an expert judgment interface 230. The expert judgment interface 230 may be configured to enable an expert to correct the segmented image, thereby obtaining a corrected segmented image. Figure 2a A corrected segmented image 231 corresponding to the input image 213 and the segmented image 211 is shown.

[0123] The corrected segmented image can be used to improve the image model 210, for example, the training items can include the input image 211 and the corresponding corrected segmented image. The training of the image model 210 can be a complete retraining of the new image segmentation model, possibly with the addition of additional training data. The training of the image model 210 can also be a retraining or refinement, for example, using the existing image model 210 as a starting point for additional training iterations; it is possible that the existing image model 210 is adapted by introducing noise, discarding, etc.

[0124] Figure 2aAn embodiment is shown that includes an element (i.e., a first model trainer 241) configured to train the image segmentation model 210. For example, the first model trainer 241 can be configured for a back propagation algorithm. For example, the first model trainer 241 can be configured to use an Adam optimizer.

[0125] In addition to the corrected segmented images, a corresponding set of error images are also obtained. Figure 2a An error image 232 corresponding to the correction made in the corrected segmentation 231 is shown. The error image 232 can be automatically derived from the corrected segmentation 231, or vice versa. The error image can use the same format as the input image or segmented image. For example, the error image can include a two-dimensional set of gray values ​​or label vectors, etc. In particular, the values ​​(e.g., pixel values) in the error image can indicate the similarity of the error correction. For example, the error image can use a simpler format, such as a binary format.

[0126] For example, the error image can be a difference image, e.g., indicating the difference between the segmented image produced by the model 210 and the corresponding corrected segmented image. The error image can be obtained in other ways. For example, the error image can indicate where the segmentation is correct, e.g., where the boundary is corrected, e.g., where points of the boundary are pulled or pushed, e.g., where support points on the spline of the analog are moved. In any case, the error image indicates where the expert corrected the segmented image.

[0127] Once a set of error images are obtained, another image segmentation model 220 can be trained to predict the expected error image of the input image. Another image segmentation 220 model can use the same image segmentation technology as the image segmentation model 210. For example, the model 220 can also include a neural network, such as a CNN. The image segmentation model 220 can be smaller than the image segmentation model 210. For example, the image segmentation model 220 can have fewer layers and / or fewer neurons. For example, the number of neurons in the image segmentation model 220 may be at most half that of the image segmentation model 210. This is possible because the segmentation quality of the model is allowed to be lower than the segmentation quality of the model 210. However, this is not required, and the image segmentation model 220 can be as large as or larger than the model 210.

[0128] Another image segmentation model 220 may partially use the same training parameters as the image model 210. For example, the image segmentation models 210 and 220 may be different heads on top of a common body.

[0129] Another image segmentation model 220 learns to predict where an expert can correct the segmented image. This is advantageous and helps overcome challenges in manual and semi-automatic annotation of training data.

[0130] Figure 2a An embodiment is shown that includes an element (i.e., a second model trainer 242) configured to train another image segmentation model 220. For example, the second model trainer 242 can be configured for a back propagation algorithm. For example, the first model trainer 242 can be configured for the same training algorithm as the trainer 241.

[0131] There are a variety of ways to obtain the correction, such as using the expert judgment interface 230. For example, in an embodiment, the image segmentation model 210 can be applied to the input image 213 to obtain the corresponding segmented image 211. Then, another image segmentation model can be applied to obtain the expected error image 221 corresponding to the input image 213 and the segmented image 213. For example, the expert judgment interface 230 can be configured to display both the segmented image 211 and the expected error image 221 to the expert. For example, the display can use a display, such as a monitor.

[0132] The expert may use a user interface to input corrections. For example, he may use a pointing tool such as a mouse, stylus, etc. to identify a point on the boundary of the segmentation. For example, the point may be pushed or pulled by the expert. For example, the expert may use a pointing device to draw a new boundary for the segmentation. If the expert judgment is not human judgment, other interfaces may be more useful, such as digital interfaces.

[0133] Thus, a corrected segmented image for the segmented image is obtained from the expert, from which an error image can be derived.

[0134] Embodiments have several advantages. For example, some problems that embodiments can solve include:

[0135] Availability of experts: Experts (e.g., radiologists) have various roles and responsibilities to fulfill. However, experts play a key role in obtaining high-precision labels. Given these parameters, it is difficult and costly to obtain sufficient expert time at a high enough skill level for medical image annotation. Using another image model according to an embodiment, expert time is used more efficiently by accounting for uncertainties in the annotation process. Therefore, the image segmentation produced by model 210 will be of higher quality because the model summarizes a larger expert annotation library and / or a higher quality library.

[0136] Labor-intensive and time-consuming: In a normal clinical setting, it takes several minutes or longer for a trained radiologist to review or edit the medical image segmentation regions and prepare a report, depending on the complexity of the image. Semi-automatic methods take about 5-10 minutes to annotate medical images for segmentation. Manual methods take longer. The annotation task is labor-intensive and time-consuming because of the large amount of training data used to train image segmentation models. A dynamic approach, which involves learning from a real-world uncertainty knowledge graph derived from a variety of human annotators, shortens the review time.

[0137] Intra- and inter-annotator variance: Another issue is the variance found in the manual annotations by one or more annotators. For example, different radiologists may interpret the same medical image differently due to different skill levels. Semi-automatic methods perform better compared to manual methods, but they are also prone to variance. This issue leads to poor labeling quality for image segmentation models. Image segmentation models that are explicitly trained on the regions in the image where corrections are actually made will focus the expert’s attention on the part of the image where other experts made corrections on similar images and, therefore, the current expert may also consider making the correction. Without this information, the expert may not consider the correction. Thus, variance is reduced.

[0138] Availability of representative data: One of the challenges in training machine learnable models (e.g., neural networks) is the limited availability of medical imaging data for training. The availability of medical data is limited due to various factors such as regulation, cost, etc. Moreover, within the available data, the data may not represent all variations. If the unseen data varies with respect to the training data, the machine learnable model may fail in this regard. The proposed invention addresses this critical aspect by providing information about failure cases by identifying possible human label corrections.

[0139] Segmentation tasks such as labeling, classification, landmark annotation, etc. can benefit from embodiments.

[0140] Figure 2a A configuration is shown that includes an element configured to train an image segmentation model 210 (ie, a first model trainer 241) and an element configured to train another image segmentation model 220 (ie, a second model trainer 242).

[0141] Neither the first model trainer 241 nor the second model trainer 242 is required, and either or both may be omitted. For example, in an embodiment, the system 200 may not be configured to train or retrain the model 210, for example, when only another image segmentation model 220 is needed. In addition, the training may be complete, or may be completed on a different device or system. For example, in an embodiment, the system 200 may not be configured to train or retrain the model 220, for example, a pre-trained model may be obtained from another source.

[0142] For example, an embodiment may use another image segmentation model that has already been trained. Such a system may be configured to, for example, apply another image segmentation model to obtain an expected error image 221, and display an expected error image 232 to an expert to guide him / her to correct the segmented image. Such a correct image may then be used to refine and / or retrain the image segmentation model 210. As desired, the error image may be displayed as superimposed on the input image 213 or a copy of the input image 213.

[0143] It is expected that another image segmentation model 220 is accurate, but the penalty for occasional wrong predictions is not as severe as for model 210, so training of model 220 may be terminated sooner than model 210.

[0144] Figure 2a An optional uncertainty image 212 generated for an input image 213 is shown. The uncertainty image shows the confidence of the image segmentation model 210 in different parts of the segmented image. Obtaining the uncertainty image does not require training of an expert's correction (e.g., on an error image). Uncertainty images are relatively easy to obtain, which is an advantage, but they also have various disadvantages. The uncertainty images can be displayed to the expert, but the inventors have found that they are more advantageously used as input to another image segmentation model 220, perhaps an additional input. Uncertainty images are discussed further herein.

[0145] Another image segmentation model 220 may be configured to receive the uncertainty image 212 as input instead of the input image 213 .

[0146] The other image segmentation model 220 may be configured to receive the input image 213 as input, but not use the uncertainty image 212. This option may be appropriate if a relatively powerful image segmentation model is trained for the other image segmentation model 220.

[0147] Another image segmentation model 220 may be configured to receive both the input uncertainty image 212 and the input image 213 .

[0148] Figure 2b An example of embodiment of an image segmentation system is schematically shown. Figure 2bOptional details of the expert correction are shown. For example, Figure 2b The expert selection unit 250 in may be incorporated into the system 200 .

[0149] Expert judgment may be obtained from at least one expert in a pool of experts. Which expert is used may vary. For some corrections, a higher skill level may be required than for other corrections. Figure 2b An expert selection unit 250 is shown. The expert selection unit 250 is configured to select an expert from a pool of experts to obtain a corrected segmented image. Which expert is selected may depend on the uncertainty level in the uncertainty image corresponding to the segmented image. For example, if the expected error image 221 shows that a larger portion of the image may require correction, the correction may require an expert with a higher skill level.

[0150] For example, unit 250 may include a list of experts and corresponding skill sets, such as segmentation skill levels. The segmentation skill level may be general or specific to a type of image. Unit 250 may match segmentation to an expert based on the level of uncertainty in the image, but other factors may also be used. For example, an expert in mammography may prefer to handle corrections in this area.

[0151] The uncertainty level can be derived from the expected error image 221, but the uncertainty level can also be derived from additional or alternative sources discussed herein (e.g., the input image 211 and the uncertainty image 212). For example, the uncertainty level can be an area in the segmented image that may need correction. The uncertainty level can be an integral of the uncertainty in the segmented image, for example, summing the uncertainty over the image.

[0152] In an embodiment, if the uncertainty in the image is low enough, no correction is needed at all. For example, the system 200 can be configured to determine an uncertainty level of the segmented image that indicates an overall expected error. The expected error can be calculated based on an associated expected error image and / or an associated uncertainty image. If the uncertainty level is below a threshold indicating a low error expectation, the correction step can be skipped. The input image 213 and the segmented image 211 can still be added to the training data.

[0153] The accepted uncertainty level is a threshold for filtering high and low uncertainty samples. The value of the threshold can be determined empirically, for example, by evaluating a sample of rejected and accepted images. The value of the threshold can also be determined by requiring a certain fixed percentage of the verification images (e.g., 50%, 10%, etc.).

[0154] In an embodiment, the system 200 is configured to sort, e.g., rank, a set of segmented images and obtain labels for priority samples first; e.g., samples that are expected to require correction. This helps improve the performance of the segmentation model more quickly.

[0155] Figure 2c An example of embodiment of an image segmentation system is schematically shown. Figure 2c An optional uncertainty model is shown. For example, an uncertainty model can be incorporated into system 200. Figure 2c An associated uncertainty model 215 is shown, which is configured to generate an uncertainty image 212 given an input image 213 .

[0156] The associated uncertainty model is configured to generate an uncertainty image of the input image, the uncertainty image indicating the confidence of the image segmentation model in different parts of the segmented image. The uncertainty model can be applied to a set of training data to obtain a set of uncertainty images corresponding to the set of segmented images, and another image segmentation model at least takes the corresponding uncertainty images as input. For example, the uncertainty image can estimate the variance among multiple potential segmentations of pixels of the input image.

[0157] There are a variety of ways to obtain an associated uncertainty model. For example, the uncertainty model can be obtained together with the image segmentation model. For example, the uncertainty model can be created as a byproduct of training. For example, the uncertainty model can be a different head on the volume shared with the image segmentation model.

[0158] Various methods of creating uncertainty models are known and may be applied in the context of embodiments.

[0159] For example, the associated uncertainty model 215 can be configured to estimate the variance in multiple potential segmentations of pixels of the input image. For example, in addition to the segmentation 211, multiple potential segmentations can also be obtained. For example, multiple potential segmentations can be obtained from one or more of the following: probabilistic segmentations of multiple labels, an ensemble of segmentation models, multi-head and / or iterative Monte Carlo dropout. The variance calculated on multiple segmented images (e.g., the variance calculated per pixel) can be regarded as an uncertainty image.

[0160] Examples of known uncertainty estimators can be found in Biraja Ghoshal’s paper “Estimatinguncertainty in deep learning for reporting confidence to centroscopy inmedical image segmentation and diseases detection”. The paper gives various options, in particular, Section 4.1 provides a working example of U-net. This uncertainty model can be used for model 215.

[0161] In addition to the uncertainty image, an uncertainty score can be calculated based on the obtained uncertainty image. For example, the uncertainty score can be the sum, maximum, integral calculated on the uncertainty image. As with the total expected error obtained from the expected error image, the uncertainty score can be compared to a threshold to decide whether a review is required and / or whether it needs to be reviewed by a high-skilled or low-skilled reviewer. The threshold can be defined by the annotator based on the acceptance error, or can be manually calculated based on the model performance on the validation dataset, etc.

[0162] In an embodiment, the segmentation may be obtained from the image segmentation model 210, for example based on a pre-trained model for predicting the segmentation. The uncertainty model 215 may be applied to obtain an uncertainty image. The uncertainty image may be displayed to the expert when correcting the segmentation. This may be used, for example, before the model 220 is trained. Once the model 220 is trained, the expected error image may be displayed instead of the uncertainty image or in addition to the uncertainty image.

[0163] The uncertainty image partially reflects areas where the training data has inter-expert or intra-expert variability. Even if the same image is segmented by the same expert, he / she will not place the segmentation boundaries in exactly the same place. The variation may be even worse if a library of experts is used and different images are considered. Therefore, the image model 210 naturally has uncertainty about the segment boundaries, which reflects the uncertainty present in the training data. However, in practice, such boundaries are often not corrected because the boundaries seem to be placed accurately enough. This can lead to distracting uncertainty halos around objects. The model 220 trained with practical corrections is less affected by this problem.

[0164] In an example uncertainty algorithm, a base image segmentation model is obtained. In an embodiment, a dropout layer is added to the model 210 after the convolutional layer or each convolutional layer and / or as an additional layer at the end of the network architecture. Bayesian uncertainty detection can use the dropout layer to detect uncertainty. Variational inference methods can add two fully connected layers, similar to the last layer of the network. After modifying the existing network into an uncertainty detection network, the model can be retrained with the training data.

[0165] During prediction, the model 210 may run a defined number of Monte Carlo iterations. The mean and variance of each sample may be captured. The uncertainty of the sample may be approximated as the calculated variance. Various uncertainty models in the prior art may be employed.

[0166] Embodiments may generate a large amount of relevant data. In embodiments, the data structure includes at least the input image 213 and the corrected segmented image 231, but more data may be retained. Figure 2d An example of an embodiment of the data structure 265 is schematically shown. Figure 2d As shown, data structure 265 is associated with input image 213, segmented image 211, uncertainty image 212, expected error image 221, corrected segmented image 231, error image 232, for example, as discussed herein. However, embodiments may use more data. For example, the same input image may be segmented by different kinds of models 210, thereby producing multiple segmented images; the same segmented image may be corrected by multiple experts, thereby producing multiple corrected segmented images (one such further corrected segmented image 233 is shown). In addition to this, many types of additional data may be retained, such as who corrected the image, when this occurred, what his or her skill level is, whether labels were corrected in addition to or instead of correcting regions in the segmentation, etc. Three additional data 261-263 are shown, and there may be less or more than 3 additional data. The data structure may be a file, a database, a pointer list, etc.

[0167] The stored data is not necessarily constant and can change dynamically. The inventors have found that an advantageous data structure to use is a knowledge graph. Figure 2e Schematically illustrated is an example of an embodiment of a knowledge graph 260. The knowledge graph 260 contains the same information as the data structure 265, but can be easily extended or modified as more information related to the image 213 is obtained in some way.

[0168] like Figure 2eAs shown, the knowledge graph 260 includes an input image 213. The input image 213 is associated with, e.g. linked to, a segmentation image 211, an uncertainty image 212, and an expected error image 221. The latter three images are obtained from the input image by applying one or more of the model 210, the uncertainty model 215, and the other model 220.

[0169] Segmented image 211 is associated with corrected segmented image 231 and error image 232. Further corrected segmented image 233 is shown associated with corrected segmented image 231, for example to reflect that this is a reviewer's review; further corrected segmented image 233 may alternatively be associated with segmented image 211, or may also be associated with it, reflecting that it is also a segmentation of that image. Where appropriate, further data 261-263 are linked to the image.

[0170] Therefore, knowledge graphs provide a dynamic and flexible data structure in which varying amounts of data can be stored efficiently.

[0171] For example, in an embodiment, multiple radiologists and / or human annotators may feed a continuous learning system to segment (e.g., annotate) images, but also train or refine models 210 and 220. Model 210 may integrate derived knowledge from annotators through a knowledge graph. The knowledge graph may track the actions of the annotators during correction of uncertain regions. These actions may then be integrated into the existing trained model to reduce uncertainty in the segmentation. The updated model 210 / 220 may then be integrated into system 200 to further optimize human intervention.

[0172] For example, model 210 may segment input image 213, thereby obtaining segmentation 211 and (optionally) uncertainty image 212. If the uncertainty is low enough, segmentation 211 may be immediately sent to an annotated image repository for annotated images.

[0173] Uncertain segmentations can be sent to expert annotators. Experts correct segmentations. Provenance can be stored in a knowledge graph. Corrected segmentations can be stored in annotated image repository.

[0174] Once a sufficient number of corrected segmentations are obtained, another model 220 can be trained to produce the expected error image. The input image 213 and / or the uncertainty image 212 and possibly other information (e.g., stored in the knowledge graph) can be used to train another model 220. We refer to the uncertainty model (e.g., model 215) as a weak uncertainty detector and model 220 as a strong uncertainty detector. A strong uncertainty detector can detect uncertainty more accurately, for example, to reduce human intervention. Therefore, a strong uncertainty detector can be used to guide an expert instead of an uncertainty image; although the uncertainty image can still be used as an input to model 220. Once the annotated image library is large enough, models 210 and / or 220 can be refined, retrained, or replaced by a newly trained model. Once the annotated image library is large enough, a production-level image segmentation model can be trained, for example, with more layers and / or neurons than the model 210 used to support the annotation.

[0175] Figure 3a An example of an embodiment of a medical image 300 is shown. The medical image 300 is a reproduction of an actual medical image. For example, the image 300 may be an input image, such as the input image 213, for an image segmentation model.

[0176] Figure 3b Schematically shown is an example of an embodiment of a medical image 301. The medical image 301 is a schematic representation of the medical image 300. The medical image 301 is manually drawn following the medical image 300. Figure 3b-3g The images shown in are hand-drawn diagrams, although they correspond schematically to Figure 2a Actual 2D images taken during the prototype of the following embodiment. To improve clarity and avoid the use of color, the images have been replaced herein with hand-drawn representations.

[0177] Figure 3c An example of an embodiment of a segmented image 310 is schematically shown. For example, the segmented image 310 may be generated by an image segmentation model, such as the segmented image 211 generated by the model 210. Two objects showing different pattern fills are visible in the segmented image 310. The pattern fills enable the image segmentation model 210 to identify two objects in the image. In embodiments, instead of pattern fills, the image may use color to indicate segmentation, or use moving or flashing or colored boundaries or any other visual indication of the generated segmentation.

[0178] Figure 3d An example of an embodiment of an uncertainty image 320 is schematically shown.

[0179] The uncertainty image 320 is generated by the uncertainty algorithm. For example, the uncertainty image can reflect that different models and / or using different training data will produce the same segmentation. Therefore, the uncertainty image gives information about the probability that the segmentation is correct.

[0180] exist Figure 3d , areas where the uncertainty exceeds a threshold have been indicated with a pattern. Area 322 surrounds the segmented object with a halo of uncertainty. Area 321 shows that a large portion of the underlying object is indicated as uncertain.

[0181] An expert responsible for correcting image 310 may be presented, for example, input image 300 (301), segmentation 310, and uncertainty image 320. However, the utility of uncertainty image 320 is limited. For example, uncertainty halo 322 may be due to variability in the training data, which does not correspond to a lack of knowledge, but rather to the fact that the perfect location of such a boundary is unclear. Because image 320 shows that large portions of the image are uncertain, the expert focus is not directed where it is most needed.

[0182] Figure 3e An example of an embodiment of a corrected segmented image 330 is schematically shown. The expert concluded that, except for the partial boundary shown at 331, Figure 3c The segmentation shown is substantially correct. The correction is input by the expert via a suitable user interface, which may be a graphical user interface with a corresponding pointing input device.

[0183] Figure 3f Schematically shown is an example of an embodiment of an error image 340. Error image 340 shows the difference between images 330 and 301. Note that only thin strips at the edge shown as 341 are indicated as corrected, e.g. indicated as errors.

[0184] Figure 3g An example of an embodiment of an expected error image 350 is schematically shown. The figure shows the output of another image segmentation model that was trained. The model indicates the location of the expected correction.

[0185] In an embodiment, the expert may be shown, for example, the input image 300 (301), the segmentation 310, and the expected error image 350. The image 350 has a higher utility for the expert. He / she can now focus on the parts that may be erroneous. Therefore, the input time of the expected image will be reduced.

[0186] Figure 4 An example of an embodiment of an image segmentation method 400 is schematically shown. The image segmentation method 400 may be implemented using at least one computer processor. The method 400 may include:

[0187] obtaining (410) an image segmentation model, the image segmentation model being at least partially trained,

[0188] applying (420) the image segmentation model to a set of input images to obtain a corresponding set of segmented images,

[0189] obtaining (430) a set of corrected segmented images and a corresponding set of error images, the set of corrected segmented images correcting the set of segmented images according to expert judgment, the error images indicating where the segmented images were corrected,

[0190] Another image segmentation model is trained (440) to predict an expected error image for the input image.

[0191] Figure 5 An example of an embodiment of an image segmentation method 500 is schematically shown. The method 500 may include:

[0192] 510: Collect a small training set

[0193] 520: Training a simple expected error model

[0194] 530: Collecting a large set of unlabeled data

[0195] 540: Use image segmentation models to segment unlabeled data

[0196] 550: Determine whether the uncertainty in the annotation exceeds the threshold based on the expected error model

[0197] 561: If the uncertainty is low, add the annotated image to the annotated image library

[0198] 562: Using Expert Judgment to Label the Most Uncertain Images

[0199] 570; Add images annotated by experts to the annotated image library

[0200] 580: Retrain image segmentation model and / or expected error.

[0201] The method 500 may be expanded or changed in various ways. For example, an uncertainty model may be added to be used as an input to the expected error model.

[0202] Many different ways of performing the method are possible, as will be apparent to one skilled in the art. For example, the order of the steps may be performed in the order shown, but the order of the steps may also be changed, or some steps may be performed in parallel. In addition, other method steps may be inserted between steps. The inserted steps may represent an improvement of the method described herein, or may be unrelated to the method. For example, some steps may be performed at least partially in parallel. In addition, a given step may not be fully completed before starting the next step.

[0203] Embodiments of the method may be performed using software including instructions for causing a processor system to perform method 400 or 500. The software may include only those steps taken by a specific sub-entity of the system. The software may be stored in a suitable storage medium, such as a hard disk, a floppy disk, a memory, an optical disk, etc. The software may be sent as a signal along a wired or wireless or using a data network (e.g., the Internet). The software may be downloaded and / or used remotely on a server. Embodiments of the method may be performed using a bitstream arranged to configure programmable logic (e.g., a field programmable gate array (FPGA)).

[0204] It should be understood that the subject matter of the present disclosure also extends to computer programs suitable for putting the subject matter of the present disclosure into practice, in particular computer programs on or in a carrier. The program can be in the form of source code, object code, code intermediate source and object code, such as partially compiled form, or any other form suitable for implementing an embodiment of the method. An embodiment involving a computer program product includes computer executable instructions corresponding to each processing step of at least one of the described methods. These instructions can be subdivided into subroutines and / or stored in one or more files that can be statically or dynamically linked. Another embodiment involving a computer program product includes computer executable instructions corresponding to each device, unit and / or part of at least one of the described systems and / or products.

[0205] Figure 6aA computer readable medium 1000 having a writable portion 1010 is shown, as is a computer readable medium 1001 also having a writable portion. The computer readable medium 1000 is shown in the form of an optically readable medium. The computer readable medium 1001 is shown in the form of an electronic memory, in this case a memory card. The computer readable media 1000 and 1001 may store data 1020, wherein the data may indicate instructions, which, according to an embodiment, when executed by a processor system, cause the processor system to perform an image segmentation method. The computer program 1020 may be embodied on the computer readable medium 1000 as a physical mark or by magnetization of the computer readable medium 1000. However, any other suitable embodiments are also contemplated. Furthermore, it should be understood that although the computer readable medium 1000 is shown here as an optical disk, the computer readable medium 1000 may be any suitable computer readable medium, such as a hard disk, a solid state memory, a flash memory, etc., and may be non-recordable or recordable. The computer program 1020 includes instructions for causing the processor system to perform the image segmentation method.

[0206] Figure 6b A schematic diagram shows a processor system 1140 according to an embodiment of an image segmentation device. The processor system comprises one or more integrated circuits 1110. Figure 6b The architecture of one or more integrated circuits 1110 is schematically shown in FIG. The circuit 1110 includes a processing unit 1120, such as a CPU, for running a computer program component to perform a method according to an embodiment and / or implement its module or unit. The circuit 1110 includes a memory 1122 for storing programming code, data, etc. A portion of the memory 1122 may be read-only. The circuit 1110 may include a communication element 1126, such as an antenna, a connector, or both. The circuit 1110 may include a dedicated integrated circuit 1124 for performing some or all of the processing defined in the method. The processor 1120, the memory 1122, the dedicated IC 1124, and the communication element 1126 may be connected to each other via an interconnect 1130 (e.g., a bus). The processor system 1110 may be arranged to perform contact and / or contactless communication using an antenna and / or a connector, respectively.

[0207] For example, in an embodiment, the processor system 1140 (e.g., an image segmentation device) may include a processor circuit and a memory circuit, and the processor is arranged to execute software stored in the memory circuit. For example, the processor circuit may be an Intel Core i7 processor, an ARM Cortex-R8, etc. The memory circuit may be a ROM circuit or a non-volatile memory, such as a flash memory. The memory circuit may be a volatile memory, such as an SRAM memory. In the latter case, the device may include a non-volatile software interface, such as a hard drive, a network interface, etc., arranged to provide software.

[0208] Although the device 1110 is shown as including one of each of the described components, various components may be replicated in various embodiments. For example, the processor may include multiple microprocessors that are configured to independently perform the methods described herein, or are configured to perform the steps or subroutines of the methods described herein, so that multiple processors cooperate to implement the functions described herein. In addition, in the case where the device 1110 is implemented in a cloud computing system, various hardware components may belong to separate physical systems. For example, the processor may include a first processor in a first server and a second processor in a second server.

[0209] It should be noted that the above-mentioned embodiments illustrate rather than limit the disclosed subject matter, and that those skilled in the art will be able to design many alternative embodiments.

[0210] In the claims, any reference numerals placed between brackets shall not be construed as limiting the claim. The use of the verb 'comprise' and its variations do not exclude the presence of elements or steps other than those described in the claim. The definition "one" or "an" before an element does not exclude the presence of multiple such elements. When preceding a list of elements, an expression such as "at least one" means selecting all or any subset of elements from the list. For example, the expression "at least one of A, B, and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The subject matter of the present disclosure can be implemented by hardware including several different elements and by a suitably programmed computer. In a device claim that lists several parts, several of these parts can be embodied by the same hardware item. The fact that certain measures are cited in mutually different dependent claims does not indicate that the combination of these measures cannot be advantageous.

[0211] In the claims, references in parentheses refer to reference numerals in the drawings of exemplary embodiments or formulas of the embodiments, thereby increasing the intelligibility of the claims. These references should not be construed as limiting the claims.

Claims

1. A computer-implemented method (400) for training a machine-learnable image segmentation model, the method comprising: obtaining (410) an image segmentation model, the image segmentation model being at least partially trained, applying (420) the image segmentation model to a set of input images to obtain a corresponding set of segmented images, obtaining (430) from a memory a set of corrected segmented images and a corresponding set of error images, the set of corrected segmented images correcting the set of segmented images based on expert judgment, wherein the error images include indications of locations in the set of segmented images where the segmented images were corrected, Another image segmentation model is trained (440) for predicting an expected error image for an input image based on the set of error images.

2. The method according to claim 1, wherein: The image segmentation model is a medical segmentation model.

3. A method according to any one of the preceding claims, comprising: applying the image segmentation model to an input image to obtain a corresponding segmented image, applying the other image segmentation model to obtain an expected error image for the input image, displaying both the segmented image and the expected error image to the expert, A corrected segmented image of the segmented image is obtained from the expert.

4. A method according to any one of the preceding claims, comprising capturing the error image and / or the corrected segmented image in a knowledge graph.

5. A method according to any one of the preceding claims, comprising training the image segmentation model on expert-corrected segmented images.

6. A method according to any one of the preceding claims, wherein: An associated uncertainty model is associated with the image segmentation model, and the associated uncertainty model is configured to generate an uncertainty image of an input image, wherein the uncertainty image indicates the confidence of the image segmentation model in different parts of the segmented image. The method includes applying the uncertainty model to obtain a set of uncertainty images corresponding to the set of segmented images, and the other image segmentation model at least takes the corresponding uncertainty images as input.

7. The method according to claim 6, wherein: The associated uncertainty model includes estimating variance among multiple potential segmentations of pixels of the input image.

8. The method according to claim 7, wherein: The plurality of potential segmentations can be obtained from one or more of the following: Probabilistic segmentation of multiple labels, A collection of segmentation models, Long, and / or Iterated Monte Carlo Dropout.

9. The method according to any one of claims 6 to 8, wherein: The expert judgment is obtained from at least one expert in an expert database, and the method comprises: According to the uncertainty level in the uncertainty image corresponding to the segmented image, an expert is selected from the expert library to obtain a corrected segmented image.

10. The method according to any one of claims 6 to 9, comprising: determining an uncertainty level for the segmented image, the uncertainty level being indicative of an overall expected error from an associated expected error image and / or from an associated uncertainty image, The corrected segmented image is obtained only when the uncertainty level is above a threshold indicating a high error expectation.

11. A computer-implemented method for training a machine-learnable image segmentation model, the method comprising: obtaining an image segmentation model, the image segmentation model being at least partially trained, applying the image segmentation model to an input image to obtain a corresponding segmented image, applying another image segmentation model to obtain an expected error image for the input image, the another image segmentation model having been trained to predict the expected error image for the input image, displaying both the segmented image and the expected error image to the expert, and A corrected segmented image of the segmented image is obtained from the expert.

12. The method for a machine-learnable image segmentation model according to claim 11, comprising: The image segmentation model is trained on the expert-corrected segmented images.

13. A device for training a machine-learnable image segmentation model, the device comprising a communication interface configured to obtain one or more input images, and a processor system configured to execute the method of any one of claims 1-12.

14. A transitory or non-transitory computer-readable medium comprising: Computer instructions, which are configured to execute the method according to any one of claims 1 to 12 when the computer instructions are executed on a computer, and / or representing parameters of another image segmentation model trained according to claim 1 that predicts an expected error image for an input image, and / or Represents parameters of an image segmentation model trained according to claim 5 and / or claim 12 and configured to generate a segmented image.

Images