Image segmentation with improved expert guidance

Abstract

Some embodiments relate to machine learning enabled image segmentation, where an image segmentation model is applied to a sequence of input images to obtain a corresponding sequence of segmentation images, and the modified segmentation images obtained from expert judgment are used to train a further image segmentation model that predicts an expected error image.

Description

[Technical Field]

[0001] The subject matter of this disclosure relates to methods for machine-learnable image segmentation models, devices for machine-learnable image segmentation models, and computer-readable media. [Background technology]

[0002] For example, medical image analysis, such as diagnostic segmentation and classification, has achieved impressive performance, with deep learning-based models capable of producing high-quality medical image segmentations. Summary of the Invention [Problem to be solved by the invention]

[0003] However, there remain images for which a given image segmentation model may give incorrect answers. Experiments on image classification have shown that trained models sometimes fail in certain cases, despite being very confident in their predictions.

[0004] As pointed out in the paper "Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods" by Eyke Hullermeier and Willem Waegeman, two sources of uncertainty are identified: aleatoric and epistemic. According to these authors, aleatoric uncertainty (also known as statistical uncertainty) refers to the concept of randomness, i.e., the variability in experimental results that is essentially due to random effects. Epistemic uncertainty (also known as systematic uncertainty) refers to the uncertainty caused by lack of knowledge. Epistemic uncertainty can, in principle, be reduced using additional information.

[0005] Indeed, a key factor in the success of these new image segmentation models is the quality of the ground truth training data on which the image models are trained. To achieve the best performance, human experts evaluate large numbers of images to improve model training and improvement.

[0006] To reduce the epistemic uncertainty in image segmentation models, it is advantageous to acquire more and higher quality data. [Means for solving the problem]

[0007] A first possible workflow for obtaining better training data is to use a partially trained image segmentation model. The partially trained image segmentation model is applied to a set of images, thus obtaining a set of corresponding segmentation images. Here, the segmentation images can be modified as needed, rather than being fully segmented manually by various experts (a pool of experts). The modified segmentation images thus obtained can be used to train a better model or to further train an existing model.

[0008] This workflow is more efficient because it does not require the expert to segment parts of the image that the model has already gotten right. As the model gets better, it becomes more and more correct, greatly reducing the expert's work.

[0009] A drawback of this first workflow is that an expert needs to carefully evaluate the segmentation proposed by the model for errors. In the second workflow, the (partially trained) image segmentation model not only calculates a segmentation image, but also calculates an uncertainty image for the input image, which indicates the confidence of the image segmentation model for different parts of this segmentation image. Methods for calculating such uncertainty images are known per se, and some of these methods are cited herein. In the second workflow, the expert views the segmentation image generated by the image segmentation model and the corresponding uncertainty image. Here, the expert is guided to the parts of the image that are most likely to be erroneous, and can therefore more quickly evaluate the segmentation proposed by the model. As in the first workflow, the expert corrects the segmentation if necessary, for example, for training purposes.

[0010] While the second workflow generates high-quality training data in less time, the inventors have found that it is still suboptimal. A problem the inventors have noted is that uncertainty images tend to mark large portions of an image as uncertain, even if they do not actually correct those portions. This negates the advantage of uncertainty images in drawing experts' attention to portions of the image that require verification. Another problem is that object edges in uncertainty images tend to have ribbons of uncertainty around them. A possible reason for these ribbons is that the exact edges of objects are often unclear and are labeled differently by different experts in different images. While the uncertainty image correctly indicates that the exact boundaries of an image are uncertain, the boundaries are nevertheless unlikely to be corrected. It is possible that the experts evaluating the images are either satisfied with the location of object boundaries, or that the experts do not make good judgments about such boundaries.

[0011] The embodiment proposes training a second image segmentation model in addition to the first image segmentation model that learns the desired segmentation. The second image segmentation model is trained with corrections made by an expert to the segmentation generated by the first image segmentation model. In an improved workflow, the expert can view the segmentation images generated by the first image segmentation model and the corresponding expected error images calculated by the second image segmentation model.

[0012] Such a second image segmentation model has advantages over the use of an uncertainty image, as described in the second workflow: the expected error image tends to mark fewer images as possible defects than the uncertainty image, and the expected error image performs better in drawing the expert's attention to regions of the segmentation that may have real errors.

[0013] Furthermore, the expected-error image is less susceptible to ribbons of uncertainty, or halos, that appear around objects. Because the exact boundaries of objects tend not to be corrected by experts, or to require such correction, the second image segmentation model correctly learns that such boundaries are likely to remain stationary. This also helps focus the expert's attention on portions of the image that are likely to require review. Both aspects therefore contribute to reducing the time it takes to review the model-predicted segmentation.

[0014] In a third workflow, the further image segmentation model that predicts an expected error image can be used by having the model predict an expected error image for an input image and presenting both the segmentation image generated by the image segmentation model and the expected error image to an expert. The expected error image focuses on areas that are most likely to require correction. Thus, the expert can correct the segmentation more quickly. This improves the workflow for obtaining an image segmentation. Particularly for medical image segmentation, the time required to segment and label an image is reduced compared to the first and second workflows described above. The corrected segmentation is used for clinical purposes, such as diagnosis, evaluation, treatment planning, etc. The further image segmentation model that predicts an expected error image can be used by having the model predict an expected error image for an input image and presenting both the segmentation image generated by the image segmentation model and the expected error image to an expert. The expected error image can be used to train the image segmentation model or to train the further image segmentation model.

[0015] For example, in one embodiment, an image segmentation model is configured to receive an input image and generate as output a segmentation image that classifies portions of the input image. A further image segmentation model that predicts an expected error image for the input image is obtained by training the image segmentation model with an error image that indicates where the segmentation image has been modified by expert judgment. This approach works well for medical segmentation models, but can be applied to other fields as well.

[0016] A knowledge graph is an advantageous data structure for tracking the various images and other data that occur in an embodiment.

[0017] In an embodiment, the associated uncertainty model is configured to generate an uncertainty image for the input image, indicating the confidence of the image segmentation model for different parts of the segmentation image. Such an uncertainty image can be obtained without access to a trained further image model. It has been found advantageous to provide the uncertainty image as input to said further image model.

[0018] The method embodiments can be implemented in a computer as a computer-implemented method, or in dedicated hardware, or a combination of both. Executable code of the method embodiments can be stored in a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product comprises non-transitory program code stored in a computer-readable medium for performing the method embodiments when the computer program product is run on a computer.

[0019] In an embodiment, a computer program comprises computer program code configured to perform all or part of the steps of the method embodiments when the computer program is run on a computer. Preferably, the computer program is embodied on a computer readable medium.

[0020] Another aspect of the subject matter of this disclosure is a method of making a computer program downloadable. [Brief explanation of the drawings]

[0021] Further details, aspects and embodiments are described, by way of example, with reference to the drawings, in which elements in the drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale, and in which elements corresponding to elements already described have the same reference numerals. [Figure 1]FIG. 1 shows a schematic diagram of an example embodiment of an image segmentation device. [Figure 2a] FIG. 2a shows a schematic diagram of an example embodiment of an image segmentation system. [Figure 2b] FIG. 2b shows a schematic diagram of an example embodiment of an image segmentation system. [Figure 2c] FIG. 2c shows a schematic diagram of an example embodiment of an image segmentation system. [Figure 2d] FIG. 2d shows a schematic diagram of an example of an embodiment of the data structure. [Figure 2e] FIG. 2e shows a schematic diagram of an example of a knowledge graph embodiment. [Figure 3a] FIG. 3a shows an example of a medical image embodiment. [Figure 3b] FIG. 3b shows a schematic example of an embodiment of a medical image. [Figure 3c] FIG. 3c shows a schematic example of an embodiment of a segmentation image. [Figure 3d] FIG. 3d shows a schematic example of an embodiment of the uncertainty image. [Figure 3e] FIG. 3e shows a schematic example of an embodiment of a modified segmentation image. [Figure 3f] FIG. 3f shows a schematic example of an embodiment of the error image. [Figure 3g] FIG. 3g shows a schematic example of an embodiment of an expected error image. [Figure 4] FIG. 4 shows a schematic diagram of an example embodiment of an image segmentation method. [Figure 5] FIG. 5 shows a schematic diagram of an example embodiment of an image segmentation method. [Figure 6a] FIG. 6a illustrates schematically a computer readable medium having a writeable portion having a computer program according to an embodiment. [Figure 6b] FIG. 6b shows a schematic diagram of a processor system according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] The subject matter of the present disclosure may be embodied in many different forms as shown in the figures, and while one or more specific embodiments have been described in detail, it should be understood that the present disclosure is to be considered as an exemplification of the principles of the subject matter of the present disclosure and is not intended to be limited to the specific embodiments shown and described.

[0023] The following describes the elements of the embodiments in terms of their operations for ease of understanding. However, it is clear that each element is arranged to perform the functions that they are described to perform. Furthermore, the subject matter of the present disclosure is not limited to the embodiments only, but also includes all other combinations of features described herein or recited in mutually different dependent claims.

[0024] The success of image segmentation models is based in part on advanced segmentation algorithms, e.g., based on deep convolutional neural networks, and in part on accurate training data. Advanced segmentation models, particularly medical image segmentation, still require large, representative, and high-quality annotated datasets. Perfect training datasets are rarely obtained, especially in the field of medical imaging, where data and annotations are expensive to obtain. Automated AI medical image segmentation using partially trained algorithms and human-assisted correction using editing tools in radiology workflows can help improve the quality of training datasets. Such workflows reduce the need for radiologists' second readings and / or human supervision, increasing productivity.

[0025] Medical imaging, such as MRI and CT scans, is the primary form of diagnosis for various medical conditions, from tumors to ligament tears. Labeling medical images is not only a key requirement for supervised machine learning and deep learning models, but also for clinical purposes.

[0026] For example, one of the challenges relates to detecting intra- and inter-annotator discrepancies due to aleatoric and epistemic uncertainties in machine learning, non-availability of representative data, failure of labeling models to generalize especially in multi-site environments, non-availability of expert radiologists, and different skill levels of radiologists, which requires labor-intensive corrections by radiologists of segmentation masks automatically predicted by AI models.

[0027] Segmentation uncertainty arises from a variety of sources, including problems with the physical device, problems with the annotation tool, hidden features in the model, or incorrect decision boundaries. These uncertainties can be broadly divided into aleatoric and epistemic uncertainty. Aleatoric uncertainty relates to the objective or physical concept of uncertainty, which is inherent in the data. The source of uncertainty is often unknown. Epistemological uncertainty relates to the subjective or personal concept of uncertainty. Epistemological uncertainty arises from lack of knowledge or ignoring certain features during the data generation process or model training.

[0028] Improved techniques for improving the performance of AI image segmentation models are described herein in various embodiments. For example, embodiments use uncertainty predictors used in smart annotation workflows. Knowledge graphs can be advantageously used to organize data within the system.

[0029] In a traditional smart labeling scenario, an image segmentation model can generate an approximate segmentation image, also known as a segmentation image, and then an expert, e.g., a radiologist, is provided with editing tools to modify the automatically segmented regions.

[0030] In a smart labeling scenario according to an embodiment, an image segmentation model can generate an approximate segmentation image. An expert is shown the segmented image and the expected error image obtained from the model trained on previous modifications. The expert is provided with editing tools to modify the automatically segmented regions, and the expected error image guides the expert's attention, reducing review time.

[0031] In embodiments, a model is trained to learn and model uncertainty corrections from human annotators. For example, such data is stored in a flexible data structure such as a knowledge graph. A neural network trained with the corrections can be used to predict regions of uncertainty in images. These uncertainty images are later used to correct or evaluate image segmentation models and / or improve the productivity of expert annotators. Reducing expert assistance with smart annotations that are applied automatically with high confidence reduces the need for radiologists to second-read images and / or human oversight, increasing productivity.

[0032] 1 illustrates a schematic diagram of an exemplary embodiment of an image segmentation device 110. The image segmentation device 110 is configured for a machine-learning enabled image segmentation model. For example, the device 110 may be used to apply the image segmentation model to one or more input images and obtain corresponding segmented images. Such devices are useful in many fields, particularly in the medical field, e.g., when applied to medical images. Other fields include, for example, manufacturing, where image segmentation is used to detect defects in manufactured products, and assisted or autonomous driving, where image segmentation is used to detect objects in road images.

[0033] The device 110 is configured to obtain a series of corrected segmentation images that correct the series of segmentation images from expert judgment, and a series of corresponding error images, where the error images indicate where the segmentation images were corrected, to train a further image segmentation model that predicts an expected error image 221 for the input image. Such a trained further image segmentation model is advantageous because it provides a better indication that expert review is useful. Embodiments can be applied to image segmentation models in various fields, with medical segmentation models being a good example.

[0034] Device 110 is configured to train the further image segmentation model, although this is not required. In a useful embodiment, device 110 is instead (or additionally) configured to apply a further image segmentation model, e.g., that has already been trained on a different device. For example, a pre-trained error prediction model may be obtained from a third party as a useful addition to a semi-autonomous segmentation workflow. The corrected segmentation that an expert generates in a semi-autonomous segmentation workflow may be advantageously used for further training of both the image segmentation model and the further image segmentation model, although this is not required.

[0035] The image segmentation device 110 includes a processor system 130, a storage device 140, and a communication interface 150. The storage device 140 may be, for example, an electronic storage device, a magnetic storage device, or the like. The storage device may include local storage, such as a local hard drive or electronic memory. The storage device 140 may include non-local storage, such as cloud storage. In the latter case, the storage device 140 may include a storage interface with the non-local storage device. The storage device may include multiple individual sub-storage devices that make up the storage device 140. The storage device may include a volatile, writable portion, such as RAM, a non-volatile, writable portion, such as flash, or a non-volatile, non-writable portion, such as ROM.

[0036] Storage device 140 may be a non-transitory storage device. For example, storage device 140 may be a volatile storage device, such as RAM, that can store data when power is present. For example, storage device 140 may be a non-volatile memory device, such as flash memory, that can store data when power is present and when power is absent.

[0037] The device 110 is connected to a database 112. The database 112 may be internal or external to the device 110. For example, the device 110 may store input images, segmentations, corrections, etc. The database 112 may be shared by multiple devices 110, for example, to facilitate multiple experts working in parallel, e.g., to improve training data in parallel.

[0038] Device 110 can internally communicate with other systems, other devices, external storage devices, input devices, output devices, and / or one or more sensors via a computer network. The computer network may be the Internet, an intranet, a LAN, a WLAN, etc. The computer network may be the Internet. Device 110 has a connection interface arranged for communication within system 100 or for communication outside system 100, as needed. For example, the connection interface can have a connector, e.g., a wired connector such as an Ethernet connector or an optical connector, or a wireless connector, e.g., an antenna, e.g., a Wi-Fi, 4G, or 5G antenna.

[0039] The communications interface 150 is used to send or receive digital data, for example, to receive an input image, receive corrections, send segmentation images, correction images, uncertainty images, error images, and the like.

[0040] Execution of device 110 is performed in a processor system. Device 110 has functional units for performing aspects of the embodiments. The functional units may be part of the processor system. For example, the functional units shown herein are implemented in whole or in part in computer instructions stored in a storage device of the device and executable by the processor system.

[0041] A processor system includes one or more processor circuits, e.g., a microprocessor, a CPU, a GPU, etc. Device 110 can include multiple processors. A processor circuit may be implemented in a distributed manner, e.g., as multiple sub-processor circuits. For example, device 110 can use cloud computing. In one embodiment, device 110 is a locally integrated device. In one embodiment, device 110 is distributed across multiple geographic locations.

[0042] Typically, the image segmentation device 110 has a microprocessor that executes appropriate software stored on the device, which may, for example, be downloaded and / or stored in corresponding memory, e.g., volatile memory such as RAM, or non-volatile memory such as flash.

[0043] Instead of implementing functions using software, device 110 may be implemented in whole or in part in programmable logic, for example as a field programmable gate array (FPGA). The device may also be implemented in whole or in part as a so-called application specific integrated circuit (ASIC), e.g., an integrated circuit (IC) customized for a specific application. For example, the circuitry may be implemented in CMOS using a hardware description language such as Verilog, VHDL, etc. In particular, image segmentation device 110 may comprise circuitry for, for example, neural network processing and / or arithmetic processing.

[0044] The implementation of the image segmentation device and / or method may be implemented in a processor circuit, examples of which are shown herein. Figures 2a-2c show functional units of the processor circuit. For example, these diagrams can be used as blueprints for possible functional organizations of the processor circuit. Organizations other than those shown in these diagrams are also possible.

[0045] The processor circuitry is not shown separately from the units in these figures. For example, the functional units may be implemented in whole or in part in device 100, e.g., in computer instructions stored in electronic memory of device 100 and executable by a microprocessor of device 100. In mixed embodiments, the functional units are implemented partly in hardware, e.g., as a coprocessor such as a neural network coprocessor, and partly in software stored and executed on the device.

[0046] 2a illustrates a schematic diagram of an example embodiment of an image segmentation system 200. The image segmentation system 200 may be implemented in a device, such as the device 110. For example, in an embodiment, a workstation or an imaging device may comprise the system 200.

[0047] The system 200 is configured with an image segmentation model 210. The image segmentation model 210 is configured to receive an input image and generate as output a segmentation image that classifies portions of the input image. Figure 2a shows an input image 213 and a segmentation image 211.

[0048] The image segmentation model is advantageously a medical segmentation model, and the image segmentation model 213 can also be applied to different types of images, for example images of manufactured products, images of roads, etc.

[0049] The input images 213 can be acquired from a variety of medical imaging techniques. The input images 213 may be multidimensional image data, for example, two-dimensional (2D), three-dimensional (3D), or four-dimensional (4D) images. The input images 213 are acquired by a variety of acquisition modalities, such as, but not limited to, standard X-ray imaging, full-field digital mammograms, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), positron emission tomography (PET), single-photon emission computed tomography (SPECT), and nuclear medicine (NM). The image segmentation model 210 is configured for one or more of these imaging modalities.

[0050] The image segmentation model 210 is at least partially trained, i.e., the image segmentation model 210 is capable of actually generating a segmented image 211. However, the image segmentation model 210 does not need to be fully trained. As a result, some errors in the segmentation of the image segmentation model 210 are tolerable, especially errors due to less than perfect training material. Embodiments are applied to improve and / or obtain training data with fewer errors.

[0051] Image segmentation can include identifying regions of interest (ROI) within image data. Medical image segmentation is used, for example, in computer-aided diagnosis. Image segmentation can be, for example, segmenting regions within an image, such as detecting and / or segmenting body organs, tissues, tumors, and mass detection. For example, an image segmentation model can segment a tumor in an image of a patient's breast. Image segmentation can also be used to simulate physical features by computer, i.e., virtually positioning a CAD-designed implant within a patient.

[0052] For example, in embodiments, a medical image, e.g., an MRI image, can be acquired, and an image segmentation model can be applied to the image to obtain a segmentation image. The segmentation image shows the segmentation into one or more regions of interest, e.g., to isolate breast tissue and a tumor therein. The segmentation image has or is associated with a label that indicates the type of segmentation. The resulting segmentation can be used for diagnosis, planning possible treatments, etc. Segmentation of medical images can be useful, for example, for checking tumor growth, controlling drug dosage, and radiation dose.

[0053] For example, MRI planning can involve aligning box-shaped shapes along anatomical features. Image segmentation models can be used for immediate box planning, for example, based on survey scans or previous scans.

[0054] The image segmentation model 210 is a machine-learnable model that can be trained and / or improved using training data, e.g., having a plurality of training items. The training items can include sample images and desired results, e.g., segmentations, that the model should produce.

[0055] Image segmentation models can be based on various artificial intelligence techniques. Segmentation techniques include classification and clustering methods, such as nearest neighbor models, support vector machines (SVMs), and artificial neural networks (ANNs), especially convolutional neural networks. In particular, neural network-based image segmentation shows great promise in segmentation. While neural networks improve by using higher-quality training sets, any machine-learnable model that relies on training data improves by improving the quality of the training data. The image segmentation model can include a convolutional neural network (CNN), e.g., a neural network with one or more convolutional layers. For example, the image segmentation model can include a so-called u-net.

[0056] The segmentation may be for one particular element. For example, in embodiments, an image segmentation model is trained to segment only tumor tissue in an image, e.g., for a mammogram. The segmentation may also be for multiple elements. For example, in embodiments, an image segmentation model is trained to segment different types of tissue or organs, etc., in an image. For example, this may be implemented by generating multiple images, e.g., one image for each label, or by associating each pixel or voxel with a label vector indicating which label applies, or a probability label vector indicating the probability of various labels, etc. The segmentation for multiple labels may be indicated using visual elements, e.g., color or pattern fill, boundary curves, etc.

[0057] In an embodiment, the system 200 is configured to apply an image segmentation model 210 to a sequence of input images, thereby obtaining a sequence of corresponding segmentation images. In Figure 2a, one input image 213 and one segmentation image 211 are shown. The sequence of segmentation images may be partly correct and partly flawed.

[0058] For a series of input images, a series of modified segmentation images is obtained. The modifications are obtained from expert judgment, perhaps the judgment of a person skilled in a particular field of interest, such as a radiologist. In some cases, this expert judgment can be a different artificial intelligence (not shown) trained in some other way, which can be obtained, for example, from a third party and used to train image model 210. For example, the different model can be a previous generation image segmentation model that had high-quality segmentation but is replaced by image model 210 for some reason. Here, it is assumed that the expert judgment is obtained from an expert, but embodiments can be modified to rely on other types of expert judgment.

[0059] For example, the system 200 can have an expert judgment interface 230. The expert judgment interface 230 is configured to allow an expert to make corrections to the segmentation image, thus obtaining a corrected segmentation image. Figure 2a shows a corrected segmentation image 231 corresponding to the input image 213 and the segmentation image 211.

[0060] The modified segmentation images can be used to improve the image model 210; for example, training items can include the input image 211 and the corresponding modified segmentation images. The training of the image model 210 can be a complete retraining of a new image segmentation model, or in some cases, additional training data can be added. The training of the image model 210 can also be a retraining or refinement, for example, using an existing image model 210 as a starting point for additional training iterations, or in some cases, the existing image model 210 can be made more adaptive by incorporating noise, dropouts, etc.

[0061] 2a illustrates an embodiment including an element configured to train an image segmentation model 210, namely, a first model trainer 241. For example, the first model trainer 241 is configured for a backpropagation algorithm. For example, the first model trainer 241 is configured to use an Adam optimizer.

[0062] In addition to the corrected segmentation image, a series of corresponding error images are obtained. FIG. 2a shows an error image 232 corresponding to the corrections made to the corrected segmentation 231. The error image 232 may be automatically derived from the corrected segmentation 231, or vice versa. The error image may use the same format as the input image or as the segmentation image. For example, the error image may include a two-dimensional set of gray values, label vectors, etc. In particular, values ​​in the error image, e.g., pixel values, may indicate the similarity of the corrections made by the error. The error image may, for example, be in binary format, although simpler formats may also be used.

[0063] For example, the error image may be a difference image showing the differences between, for example, a segmentation image generated by model 210 and a corresponding corrected segmentation image. The error image may also be obtained in other ways. For example, the error image may show where the segmentation was correct, e.g., where a boundary has been corrected, e.g., where a boundary point has been pulled or pushed, e.g., where a support point of a spline or the like has been moved. In either case, the error image shows where the segmentation image was corrected by an expert.

[0064] Once the series of error images has been obtained, a further image segmentation model 220 is trained to predict an expected error image for the input image. The further image segmentation model 220 can use the same image segmentation technique as the image segmentation model 210. For example, the model 220 can comprise a neural network, such as a CNN. The image segmentation model 220 can be smaller than the image segmentation model 210. For example, the image segmentation model 220 can have fewer layers and / or fewer neurons. For example, the image segmentation model 220 can have at most half the number of neurons as the image segmentation model 210. This is possible because the segmentation quality of the model is allowed to be lower than that of the model 210. However, this is not required, and the image segmentation model 220 can be the same size as the model 210 or larger than the model 210.

[0065] The further image segmentation model 220 may use some of the same training parameters as the image model 210. For example, the image segmentation models 210 and 220 may have different heads on a common body.

[0066] The further image segmentation model 220 learns to predict where an expert can modify the segmentation image, which is advantageous and helps overcome challenges in manual and semi-automated labeling of training data.

[0067] 2a illustrates an embodiment including elements configured to train a further image segmentation model 220, namely, second model trainer 242. For example, second model trainer 242 can be configured for a backpropagation algorithm. For example, first model trainer 242 can be configured for the same training algorithm as trainer 241.

[0068] There are various ways to obtain the corrections, for example, using the expert judgment interface 230. For example, in one embodiment, an image segmentation model 210 can be applied to an input image 213 to obtain a corresponding segmentation image 211. A further image segmentation model is then applied to obtain an expected error image 221 that corresponds to the input image 213 and the segmentation image 211. For example, the expert judgment interface 230 is configured to display both the segmentation image 211 and the expected error image 221 to the expert. For example, the display may use a display, such as a monitor.

[0069] The expert can use a user interface to input the corrections. For example, a user can use a pointing tool, such as a mouse, stylus, or the like, to identify a point on the segmentation boundary. For example, this point can be pushed or pulled by the expert. For example, the expert can use a pointing device to draw a new boundary for the segmentation. If the expert's judgment is not human, other interfaces, such as a digital interface, may be more useful.

[0070] In this way, a modified segmentation image for the segmentation image is obtained from the expert, from which an error image may be derived.

[0071] The embodiments have several advantages. For example, some of the problems that the embodiments can address include:

[0072] Expert availability: Experts, e.g., radiologists, have various roles and responsibilities to fulfill. However, experts play a key role in obtaining labels with good accuracy. Given these parameters, it is difficult and expensive to allocate sufficient time to experts with a sufficiently high skill level to annotate medical images. By using additional image models according to one embodiment to point out uncertainties in the labeling process, the expert's time is used more productively. Thus, image segmentations generated by model 210 are of higher quality because the model generalizes to a larger and / or higher quality pool of expert annotations.

[0073] Labor-intensive and time-consuming In a typical clinical environment, a trained radiologist can take several minutes or more to review or edit segmentation regions in a medical image and generate a report, depending on the image's complexity. Labeling a medical image segmentation using semi-automated methods takes approximately 5-10 minutes. Manual methods take even longer. Because training image segmentation models uses large amounts of training data, the labeling task is intensive and time-consuming. Dynamic methods, including learning from real-world uncertainty knowledge graphs obtained from various human annotators, can reduce review time.

[0074] Inter- and intra-annotator variation A further problem is the variability found in manual labeling by one or more annotators. For example, different radiologists may interpret the same medical image differently due to different skill levels. Although semi-automated methods are better than manual methods, they are also prone to variability. This problem reduces the quality of the labels for the image segmentation model. An image segmentation model that is explicitly trained on regions of an image where actual corrections have been made can direct the expert's attention to parts of the image where other experts have made corrections in similar images, so that the current expert can consider the corrections as well. Without this information, the expert might not have considered the corrections. This reduces variability.

[0075] Availability of representative data One of the challenges for training machine-learnable models, such as neural networks, is the limited availability of medical imaging data for training. The availability of medical data is limited by various factors, such as regulations, cost, etc. Also, even within the available data, the data may not represent all variability. Machine-learnable models can fail in this manner if unseen data has variability relative to the training data. The proposed invention addresses this important aspect by providing information about failure cases by identifying possible human label modifications.

[0076] Segmentation tasks such as labeling, classification, landmark annotation, etc. can benefit from embodiments, etc.

[0077] FIG. 2a shows an arrangement including an element configured to train an image segmentation model 210, i.e., a first model trainer 241, and an element configured to train a further image segmentation model 220, i.e., a second model trainer 242.

[0078] Neither first model trainer 241 nor second model trainer 242 is required, and either or both can be omitted. For example, in one embodiment, system 200 may not be configured to train or retrain model 210, e.g., if only additional image segmentation model 220 is desired. Furthermore, training may be completed or may occur on a different device or system. For example, in one embodiment, system 200 may not be configured to train or retrain model 220, e.g., a pre-trained model may be obtained from another source.

[0079] For example, one embodiment may use an already trained further image segmentation model. Such a system may be configured, for example, to apply the further image segmentation model to obtain an expected error image 221, and display the expected error image 232 to the expert to guide the expert in modifying the segmentation image. Such modified images may then be used to improve and / or retrain the image segmentation model 210. The error image may be conveniently overlaid on top of the input image 213 or on a copy of the input image 213.

[0080] Although it is desirable to have an accurate further image segmentation model 220, the training of model 220 can be stopped sooner than for model 210, since the penalty of occasionally making faulty predictions is less severe than for model 210.

[0081] 2a shows an optional uncertainty image 212 generated for an input image 213. The uncertainty image indicates the confidence of the image segmentation model 210 in different portions of the segmentation image. Obtaining the uncertainty image does not require training with expert corrections, e.g., training with an error image. While uncertainty images are relatively easy to obtain, which is an advantage, they have various drawbacks. While uncertainty images are shown to experts, the inventors have found that they are more advantageously used as inputs, possibly as additional inputs, to a further image segmentation model 220. Uncertainty images are described further herein.

[0082] The further image segmentation model 220 can be configured to receive the uncertainty image 212 as input, but not the input image 213 .

[0083] The further image segmentation model 220 may be configured to receive the input image 213 as input, but not use the uncertainty image 212 .

[0084] This option is appropriate when a relatively powerful image segmentation model is trained for the further image segmentation model 220 .

[0085] The further image segmentation model 220 may be configured to receive both the input uncertainty image 212 and the input image 213 .

[0086] 2b shows a schematic diagram of an example embodiment of an image segmentation system, and FIG. 2b shows the details of the expert correction option, for example, an expert selection unit 250 is incorporated into the system 200.

[0087] The expert judgment can be obtained from at least one expert in a pool of experts. Which expert is used can make a difference. Some corrections require a higher skill level than others. Figure 2b shows an expert selection unit 250. The expert selection unit 250 is configured to select an expert from the pool of experts to obtain a corrected segmentation image. Which expert is selected depends on the level of uncertainty in the uncertainty image corresponding to the segmentation image. For example, if the expected error image 221 indicates that many parts of the image need correction, an expert with a higher skill level is required for this correction.

[0088] For example, unit 250 has a list of experts and corresponding skill sets, e.g., segmentation skill levels. The segmentation skill levels can be general or specific to the type of image. Unit 250 matches segmentations to experts based on the level of uncertainty in the image, although other factors can be used. For example, a mammography expert would preferably be able to handle modifications in this area.

[0089] The level of uncertainty may be derived from the expected error image 221, although the level of uncertainty may be derived from additional or alternative sources as discussed herein, such as the input image 211 and the uncertainty image 212. For example, the level of uncertainty may be an area of ​​the segmentation image that requires modification. The level of uncertainty may be an integral over the uncertainty in the segmentation image, such as the total uncertainty over the image.

[0090] In one embodiment, if the uncertainty in the image is sufficiently low, no correction may be required. For example, the system 200 may be configured to determine an uncertainty level for the segmentation image that indicates the overall expected error. The expected error is calculated from the associated expected error image and / or from the associated uncertainty image. If the uncertainty level is below a threshold that indicates a low expected error, the correction step is skipped. The input image 213 and the segmentation image 211 are still added to the training data.

[0091] The accepted uncertainty level is a threshold for filtering high and low uncertainty samples. The threshold value can be determined empirically, for example, by evaluating a sample of rejected and accepted images. The threshold value can be determined by requiring a certain percentage of images to be verified, such as 50%, 10%, etc.

[0092] In one embodiment, the system 200 is configured to order, e.g., rank, a set of segmentation images to prioritize samples, e.g., obtain labels first for samples expected to require correction, which helps to more quickly improve the performance of the segmentation model.

[0093] 2c shows a schematic diagram of an example embodiment of an image segmentation system. Fig. 2c illustrates an optional uncertainty model. For example, the uncertainty model may be incorporated into the system 200. Fig. 2c illustrates an associated uncertainty model 215 configured to generate an uncertainty image 212 for a given input image 213.

[0094] The associated uncertainty model is configured to generate an uncertainty image for the input image, indicating the confidence of the image segmentation model for different parts of the segmentation image. The uncertainty model is applied to the set of training data, thus obtaining a set of uncertainty images corresponding to the set of segmentation images, and a further image segmentation model takes as input at least the corresponding uncertainty image. For example, the uncertainty image can estimate the variance among multiple possible segmentations for a pixel of the input image.

[0095] There are various ways to obtain the associated uncertainty model. For example, the uncertainty model may be obtained together with the image segmentation model. For example, the uncertainty model can be created as a by-product of training. For example, the uncertainty model may be a different head on a body that is shared with the image segmentation model.

[0096] Various methods for creating uncertainty models are known and may be applied in the context of the embodiments.

[0097] For example, the associated uncertainty model 215 is configured to estimate the variance in multiple possible segmentations for a pixel of the input image. For example, multiple possible segmentations may be obtained in addition to the segmentation 211. For example, the multiple possible segmentations may be obtained from one or more of probabilistic segmentation for multiple labels, an ensemble of segmentation models, multiple heads, and / or iterated Monte-Carlo dropout. The variance calculated over the multiple segmentation images, for example calculated pixel by pixel, is obtained as an uncertainty image.

[0098] Examples of known uncertainty estimators can be found in the paper "Estimating uncertainty in deep learning for reporting confidence to clinicians in medical image segmentation and diseases detection" by Biraja Ghoshal. The paper gives various options. In particular, Section 4.1 provides a paradigm for the U-net. Such uncertainty models can be used for model 215.

[0099] In addition to the uncertainty image, an uncertainty score is calculated based on the obtained uncertainty image. For example, this uncertainty score may be a sum, maximum, or integral calculated over the uncertainty image. Similar to the total expected error obtained from the expected error image, the uncertainty score is compared to a threshold to determine whether review is required and / or whether review by a reviewer with higher or lower skills is required. The threshold may be defined by the annotator based on an acceptable error or may be calculated manually based on the model's performance on a validation dataset, etc.

[0100] In one embodiment, the segmentation is obtained from the image segmentation model 210, for example, based on a model previously trained to predict the segmentation. An uncertainty model 215 is applied to obtain an uncertainty image. This uncertainty image may be displayed to the expert when revising the segmentation. This may be used, for example, before the model 220 is trained. Once the model 220 is trained, an expected error image is displayed instead of or in addition to the uncertainty image.

[0101] The uncertainty image partially reflects areas where the training data has inter- and intra-expert variability. Even if the same image is segmented by the same expert, the expert will not place the segmentation boundary in the same exact place. The variability is even worse when using different experts (a pool of experts) and considering different images. As a result, the image model 210 is naturally uncertain about the segmentation boundary, reflecting the uncertainty present in the training data. However, in practice, such boundaries often go uncorrected because they appear to be placed accurately enough. This causes a distracting halo of uncertainty around the object. A model 220 trained with real corrections is less susceptible to this problem.

[0102] In one exemplary uncertainty algorithm, a base image segmentation model is obtained. In one embodiment, a dropout layer is added to the model 210 after the convolutional layer, or after each convolutional layer, and / or as an additional layer at the end of the network architecture. Bayesian uncertainty detection uses a dropout layer to detect uncertainty. A variational inference method can add two fully connected layers similar to the final layers of the network. After modifying the existing network into an uncertainty detection network, the model is retrained using the training data.

[0103] During prediction, the model 210 is run for a specified number of Monte Carlo iterations. The mean and variance for each sample are captured. The uncertainty of the sample is approximated as the calculated variance. From the state of the art, various uncertainty models are employed.

[0104] An embodiment can generate a large amount of associated data. In one embodiment, the data structure includes at least the input image 213 and the corrected segmentation image 231, although more data can be maintained. FIG. 2d schematically illustrates an example embodiment of a data structure 265. As shown in FIG. 2d, the data structure 265 is associated with, for example, the input image 213, the segmentation image 211, the uncertainty image 212, the expected error image 221, the corrected segmentation image 231, and the error image 232, as discussed herein. However, an embodiment can use additional data. For example, the same input image can be segmented by different models 210, resulting in multiple segmentation images, or the same segmentation image can be corrected by multiple experts, resulting in multiple corrected segmentation images (one such additional corrected segmentation image 233 is shown). Additionally, many types of additional data can be held, such as who modified the image, when the modification was made, what the skill level of the expert was, modifying labels in addition to or instead of modifying regions in the segmentation, etc. Although three additional data 261-263 are shown, there may be fewer or more than three additional data. The data structure may be a file, a database, a list of pointers, etc.

[0105] The data stored is not necessarily constant, but can change dynamically. The inventors have found that an advantageous data structure to use is a knowledge graph. Figure 2e shows a schematic of one embodiment of knowledge graph 260. Knowledge graph 260 contains the same information as data structure 265, but can be easily expanded or modified as more information becomes available in some way related to image 213.

[0106] As shown in Figure 2e, knowledge graph 260 has an input image 213. Input image 213 is associated, e.g., linked, to segmentation image 211, uncertainty image 212, and expected error image 221. The latter three images are all obtained from the input image by applying one or more of model 210, uncertainty model 215, and further model 220.

[0107] Segmentation image 211 is associated with modified segmentation image 231 and error image 232. A further modified segmentation image 233 is shown in association with modified segmentation image 231, for example to reflect that this is a reviewer's review, and a further modified segmentation image 233 can alternatively or additionally be associated with segmentation image 211 to reflect that it is a segmentation of this image. Further data 261-263 is linked to the image wherever appropriate.

[0108] Thus, knowledge graphs provide a dynamic and flexible data structure that can efficiently store various amounts of data.

[0109] For example, in one embodiment, multiple radiologists and / or human annotators may be fed into the intermittent learning system not only to segment, e.g., label, images, but also to train or refine models 210 and 220. Model 210 can incorporate knowledge derived from the annotators through a knowledge graph. The knowledge graph can track the actions of the annotators during modification of uncertainty regions. These actions are then incorporated into the existing trained model to reduce uncertainty in the segmentation. The updated models 210 / 220 are then incorporated into system 200 to further optimize human intervention.

[0110] For example, model 210 segments an input image 213 obtaining a segmentation image 211 and an (optional) uncertainty image 212. If the uncertainty is low enough, segmentation image 211 is immediately sent to a pool of labeled images.

[0111] The uncertain segmentation may be sent to an expert annotator, who will refine the segmentation. The provenance may be stored in a knowledge graph. The refined segmentation may be stored in a pool of labeled images.

[0112] Once a sufficient number of corrected segmentations are obtained, a further model 220 is trained to generate an expected error image. The further model 220 can be trained using the input image 213 and / or the uncertainty image 212, and possibly other information stored, for example, in a knowledge graph. The uncertainty model, e.g., model 215, is referred to as a weak uncertainty detector, and model 220 is referred to as a strong uncertainty detector. A strong uncertainty detector can more accurately detect uncertainty, for example, to reduce human intervention. Therefore, a strong uncertainty detector is used to guide the expert rather than the uncertainty image, but the uncertainty image is still used as input for model 220. Once the pool of labeled images is large enough, models 210 and / or 220 are refined, retrained, or replaced by a newly trained model. Once the pool of labeled images is large enough, a production-grade image segmentation model is trained, for example, with more layers and / or neurons than model 210 used to assist in labeling.

[0113] 3a shows an example embodiment of a medical image 300. The medical image 300 is a reproduction of an actual medical image. For example, the image 300 may be an input image for an image segmentation model, such as the input image 213.

[0114] Figure 3b shows a schematic representation of an example embodiment of a medical image 301. Medical image 301 is a schematic representation of medical image 300. Medical image 301 has been hand-drawn after medical image 300. The images shown in Figures 3b-3g roughly correspond to actual two-dimensional images obtained from a prototype of the latter embodiment of Figure 2a, but these images are hand-drawn schematics. To improve clarity and avoid the use of color, these images have been replaced here by hand-drawn representations.

[0115] 3c schematically illustrates an example of an embodiment of a segmentation image 310. For example, the segmentation image 310 may be generated by an image segmentation model, such as the segmentation image 211 generated by the model 210. In the segmentation image 310, two objects can be seen that exhibit different pattern fills. The pattern fills visualize that the image segmentation model 210 has identified two objects in the image. In one embodiment, instead of a pattern fill, the image can use color to indicate the segmentation, or a border that is moving, flashing, colored, or any other visual indication of the generated segmentation.

[0116] FIG. 3 d shows a schematic example of an embodiment of the uncertainty image 320 .

[0117] The uncertainty image 320 is generated by the uncertainty algorithm. For example, the uncertainty image can reflect that using different models and / or variant training data produces the same segmentation. Thus, the uncertainty image provides information about the likelihood that the segmentation is correct.

[0118] In Figure 3d, the patterns show the regions where the uncertainty exceeds the threshold: Region 322 surrounds the segmented object with a halo of uncertainty, and region 321 shows that most of the underlying object is marked as uncertain.

[0119] An expert tasked with modifying image 310 is presented with, for example, input image 300 (301), segmentation 310, and uncertainty image 320. However, uncertainty image 320 is of limited utility. For example, the halo of uncertainty 322 is likely due to variability in the training data, which does not represent a lack of knowledge but rather the fact that the exact location of such boundaries is unclear. Because image 320 shows most of the image as uncertain, the expert's attention is not directed where it is most needed.

[0120] Figure 3e shows schematically an example of an embodiment of a corrected segmentation image 330. The expert has concluded that the segmentation shown in Figure 3c is essentially correct, except for the boundary portion indicated at 331. The corrections are entered by the expert via an appropriate user interface, possibly a graphical user interface with a corresponding pointing input device.

[0121] Figure 3f shows a schematic example of an embodiment of error image 340. Error image 340 shows the difference between image 330 and image 301. Note that only the thin strip at the edge, indicated at 341, is shown to be corrected, e.g., error.

[0122] Figure 3g shows a schematic example of an embodiment of an expected error image 350. This figure shows the output of a further trained image segmentation model, which indicates where the corrections are expected.

[0123] In one embodiment, an expert is shown, for example, input image 300 (301), segmentation 310, and expected error image 350. Image 350 is highly useful to the expert. The expert can quickly focus on areas that are likely to be incorrect. Therefore, it is expected that the time it takes to input the image will be reduced.

[0124] 4 illustrates a schematic diagram of an example embodiment of an image segmentation method 400. The image segmentation method 400 is implemented using at least one computer processor. The method 400 includes: Obtaining (410) an at least partially trained image segmentation model; applying (420) the image segmentation model to a series of input images to obtain a series of corresponding segmented images; obtaining (430) the series of segmentation images from expert judgment and a series of corresponding error images showing where the segmentation images are to be modified; ) training (440) a further image segmentation model that predicts an expected error image for an input image; It has.

[0125] FIG. 5 illustrates a schematic example of an embodiment of an image segmentation method 500 . Method 500 is 510: Collect a small training set 520: Training a simple expected error model 530: Collecting large amounts of unlabeled data 540: Segment unlabeled data using an image segmentation model 550: Determine whether the label uncertainty exceeds a threshold according to the expected error model 561: If uncertainty is low, add the labeled image to the pool of labeled images 562: Using expert judgment to label the most uncertain images 570: Add expert-labeled images to the pool of labeled images 580: Retrain the image segmentation model and / or expected error It can have:

[0126] Method 500 can be extended or modified in various ways, for example, an uncertainty model can be added and used as an input for the expected error model.

[0127] As will be apparent to those skilled in the art, many different ways of performing the method are possible. For example, while the order of steps may be performed in the order shown, the order of steps may be changed or some steps may be performed in parallel. Furthermore, steps of other methods may be inserted between steps. The inserted steps may represent improvements to the method as described herein or may be unrelated to the method. For example, some steps may be performed at least partially in parallel. Furthermore, a given step may not be completely completed before the next step is initiated.

[0128] Embodiments of the methods may be implemented using software having instructions for causing a processor system to perform method 400 or 500. The software may include only steps performed by a particular sub-entity of the system. The software may be stored on a suitable storage medium, e.g., hard disk, floppy, memory, optical disk, etc. The software may be transmitted as a signal, wired, wirelessly, or using a data network, e.g., the Internet. The software may be made available for download and / or remote use on a server. Embodiments of the methods may be implemented using a bitstream arranged to configure programmable logic, e.g., a field programmable gate array (FPGA), to perform the method.

[0129] It will be understood that the subject matter of this disclosure also extends to computer programs, particularly computer programs on or in a carrier, adapted to carry out the subject matter of this disclosure. The program may be in the form of source code, object code, code intermediates, and object code, e.g., partially compiled, or any other form suitable for use in carrying out the method embodiments. An embodiment relating to a computer program product comprises computer-executable instructions corresponding to each processing step of at least one of the described methods. These instructions may be divided into subroutines and / or stored in one or more statically or dynamically linked files. Another embodiment relating to a computer program product comprises computer-executable instructions corresponding to each device, unit, and / or component of at least one of the described systems and / or products.

[0130] 6a shows a computer-readable medium 1000 having a writable portion 1010 and a computer-readable medium 1001 also having a writable portion. The computer-readable medium 1000 is shown in the form of an optically readable medium. The computer-readable medium 1001 is shown in the form of an electronic memory, in this case a memory card. The computer-readable media 1000 and 1001 can store data 1020, which, when executed by a processor system, can represent instructions that cause the processor system to perform an image segmentation method according to one embodiment. The computer program 1020 can be embodied on the computer-readable medium 1000 as a physical mark or by magnetization of the computer-readable medium 1000. However, any other suitable embodiment is equally contemplated. Furthermore, although the computer readable medium 1000 is shown here as an optical disk, it will be understood that the computer readable medium 1000 may be any suitable computer readable medium, such as a hard disk, solid state memory, flash memory, etc., and may be non-recordable or recordable. A computer program 1020 comprises instructions for causing a processor system to perform the image segmentation method.

[0131] FIG. 6b shows a schematic diagram of a processor system 1140 according to one embodiment of an image segmentation device. The processor system includes one or more integrated circuits 1110. The architecture of the one or more integrated circuits 1110 is shown schematically in FIG. 6b. The circuit 1110 includes a processing unit 1120, e.g., a CPU, for executing a method according to one embodiment and / or executing computer program components for implementing a module or unit thereof. The circuit 1110 includes a memory 1122 for storing program code, data, etc. Part of the memory 1122 may be read-only. The circuit 1110 may include a communication element 1126, e.g., an antenna, a connector, or both. The circuit 1110 may include a dedicated integrated circuit 1124 for performing some or all of the processing defined in the method. The processor 1120, the memory 1122, the dedicated IC 1124, and the communication element 1126 may be connected to each other via an interconnect 1130, e.g., a bus. The processor system 1110 may be configured for contact and / or contactless communication using antennas and / or connectors, respectively.

[0132] For example, in one embodiment, the processor system 1140, e.g., an image segmentation device, can have a processor circuit and a memory circuit, where the processor is configured to execute software stored in the memory circuit. For example, the processor circuit can be an Intel Core i7 processor, an ARM Cortex-R8, etc. The memory circuit can be a ROM circuit or a non-volatile memory, e.g., flash memory. The memory circuit can also be a volatile memory, e.g., SRAM memory. In the latter case, the device can have a non-volatile memory software interface, e.g., a hard drive, a network interface, etc., configured to provide the software.

[0133] Although device 1110 is shown as including one of the described components, in various embodiments, various components may be duplicated. For example, a processor may include multiple microprocessors configured to perform the methods described herein alone or configured to perform the steps or subroutines of the methods described herein, such that the multiple processors cooperate to achieve the functionality described herein. Furthermore, when device 1110 is implemented in a cloud computing system, the various hardware components may reside in separate physical systems. For example, a processor may include a first processor located on a first server and a second processor located on a second server.

[0134] It should be noted that the above-described embodiments are illustrative rather than limiting of the subject matter of the present disclosure, and that those skilled in the art will be able to design many alternative embodiments.

[0135] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb "to comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The absence of a plurality of elements does not exclude the presence of a plurality of such elements. The use of phrases such as "at least one of" following a list of elements indicates the selection of all or any subset of the elements in that list. For example, the phrase "at least one of A, B, and C" should be understood to include A only, B only, C only, both A and B, both A and C, both B and C, or all of A, B, and C. The subject matter disclosed herein can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a device claim enumerating several components, several of these components may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0136] In the claims, reference signs in parentheses refer to reference signs in the drawings of exemplary embodiments or in formulas of embodiments, thus facilitating the understanding of the claims, and are not to be construed as limiting the claims. [Explanation of symbols]

[0137] The following list of reference numbers refers to Figures 1, 2a-2e, 3a-3g, 6a, and 6b and is provided to facilitate interpretation of the drawings and is not to be construed as limiting the scope of the claims. 110 Image Segmentation Device 112 Image Database 130 processor system 140 Storage device 150 Communication Interface 200 Image Segmentation System 210 Image Segmentation Model 211 segmentation images 212 Uncertainty Images 213 input images 205 Related Uncertainty Models 220 Further Image Segmentation Models 221 Expected Error Image 230 Expert Judgment Interface 231 Corrected segmentation image 232 Error Image 233 Further modified segmentation images 241 First Model Trainer 242 Second Model Trainer 250 Professional Elective Units 260 Knowledge Graph 261-263 Additional Data 265 Data Structures 300 Medical Images 301 Schematic rendering of image 300 310 Segmentation Images 320 Uncertainty Images 321 First Area 322 Second Area 330 Corrected Segmentation Images 331 Modified parts 340 Error Image 341 Modified parts 350 expected error images 1000, 1001 Computer-readable medium 1010 Writable area 1020 Computer Program 1110 Integrated Circuits 1120 Processing Unit 1122 memory 1124 dedicated integrated circuits 1126 Communication Elements 1130 Interconnect 1140 processor system

Claims

1. 1. A computer-implemented method for training a machine-learnable image segmentation model, the method comprising: obtaining an at least partially trained image segmentation model; applying the image segmentation model to a sequence of input images to obtain a corresponding sequence of segmented images; obtaining from memory a series of modified segmentation images that modify the series of segmentation images based on expert judgment, and a corresponding series of error images; training a further image segmentation model to predict an expected error image for an input image based on the series of error images; and the error image having an indication of the positions at which segmentation images in the series of segmentation images have been modified; Computer-implemented methods.

2. The method of claim 1 , wherein the image segmentation model is a medical segmentation model.

3. applying the image segmentation model to an input image to obtain a corresponding segmented image; applying the further image segmentation model to obtain an expected error image for an input image; displaying both the segmentation image and the expected error image to an expert; obtaining a modified segmentation image for the segmentation image from the expert; and 3. The method of claim 1 or 2, comprising:

4. 4. The method of claim 1, further comprising incorporating the error image and / or the modified segmentation image into a knowledge graph.

5. The method of claim 1 , further comprising training the image segmentation model on the segmentation images corrected by an expert.

6. an associated uncertainty model is associated with the image segmentation model, the associated uncertainty model being configured to generate an uncertainty image for the input image indicating the confidence of the image segmentation model for different portions of the segmentation image; The method includes applying the uncertainty model to obtain a series of uncertainty images corresponding to the series of segmentation images; the further image segmentation model takes as input at least the corresponding uncertainty image; 6. The method according to any one of claims 1 to 5.

7. The method of claim 6 , wherein the associated uncertainty model comprises estimating the variance in multiple potential segmentations for pixels of the input image.

8. The plurality of potential segmentations comprises: Probabilistic segmentation for multiple labels. A set of segmentation models, multiple heads, and / or Iterative Monte Carlo Dropout The method of claim 7, wherein the signal can be obtained from one or more of:

9. The expert judgment is obtained from at least one expert in a pool of experts, and the method comprises:

9. The method according to claim 6, further comprising selecting an expert from the pool of experts to obtain a modified segmentation image according to a level of uncertainty in the uncertainty image corresponding to the segmentation image.

10. determining an uncertainty level indicative of an overall expected error for the segmentation image from the associated expected error image and / or from the associated uncertainty image; acquiring the modified segmentation image only if the uncertainty level is above a threshold that indicates a high expected error; 10. The method according to any one of claims 6 to 9, comprising:

11. 1. A computer-implemented method for training a machine-learnable image segmentation model, the method comprising: obtaining an at least partially trained image segmentation model; applying the image segmentation model to an input image to obtain a corresponding segmented image; applying a further image segmentation model trained to predict an expected error image for the input image to obtain an expected error image for the input image; displaying both the segmentation image and the expected error image to an expert; obtaining a modified segmentation image for the segmentation image from the expert; and 1. A computer-implemented method for training a machine-learnable image segmentation model, comprising:

12. training the image segmentation model with the segmentation images corrected by an expert; 12. The computer-implemented method for training a machine-learnable image segmentation model of claim 11, comprising:

13. 1. A device for training a machine-learnable image segmentation model, the device comprising: a communications interface configured to acquire one or more input images; a processor system configured to perform the method of any one of claims 1 to 12; A device having:

14. Computer instructions configured to carry out the method of any one of claims 1 to 12 when executed on a computer; and / or Parameters representing a further image segmentation model, trained according to claim 1, for predicting an expected error image for an input image; and / or Parameters representing an image segmentation model trained according to claim 5 and / or claim 12 and configured to generate a segmentation image.

1. A transitory or non-transitory computer-readable medium having:

Images