Selecting training data for annotation

Abstract

A computer-implemented method for use in selecting training data for annotation by an annotator as part of an active learning machine learning process, where the selected training data is for use in training a model for taking a medical image as input and outputting a segmentation of the medical image. The method comprises the steps of providing an unlabeled medical image as input to the model and obtaining a segmentation of the unlabeled medical image as output from the model, a) determining whether geometric features of a segment of the segmentation satisfy geometric criteria for a radiological feature represented by the segment, and / or b) determining whether an edge region of the segment satisfies an edge criterion. The method then comprises the step of selecting the unlabeled medical image as training data to be annotated by an annotator as part of an active learning machine learning process if the segment does not satisfy the geometric and / or edge criteria.

Description

[Technical Field]

[0001] While this disclosure relates more particularly to machine learning, non-exclusively, embodiments herein relate to selecting training data for annotation by an annotator as part of an active learning machine learning process to train a model for segmenting medical images. [Background technology]

[0002] Segmentation of anatomical structures and lesions is an essential part of treatment planning for cancer and other similar diseases. Manual segmentation of anatomical features is time-consuming and can only be performed by qualified, specialized radiologists. Therefore, various automated segmentation methods have been developed, some of which use deep learning to segment medical images into distinct segments or parts corresponding to different features in each image. With recent advances in the field of AI, machine learning (ML) models for segmentation are beginning to exceed human capabilities. ML models for segmentation are described in a paper by S. Minaee, YY Boykov, F. Porikli, AJ Plaza, N. Kehtamavaz, and D. Terzopoulos, “Image Segmentation Using Deep Learning: A Survey,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109 / TPAMI.2021.3059968. Summary of the Invention [Problem to be solved by the invention]

[0003] These tools generally work well on data similar to the one used to train the model, but because the quality of the imaging and other image characteristics constantly change, the models need to be frequently updated and retrained as part of ongoing maintenance. Further training requires new annotated data, such as new radiology images that are manually segmented to provide the "ground truth" or correct annotations used to train the model. Because obtaining annotations for segmentation problems requires significantly more time and skill than classification problems involving simple descriptive labels, obtaining datasets large enough to train AI models to segment medical images is difficult.

[0004] Therefore, an objective of the embodiments herein is to reduce the annotation burden on skilled human annotators when generating training data for segmentation models, without reducing the quality of the resulting models. [Means for solving the problem]

[0005] Active learning / in-product learning is a supervised machine learning method in which certain unlabeled data examples are strategically selected for annotation and used in further training the model according to which data examples are most likely to provide the greatest benefit (e.g., the greatest improvement) to model training. Active learning is a special case of supervised learning in which the active learning process itself can interactively query or request labels for the most complex data examples or data examples that the model can label with the lowest confidence. For example, the model may select data examples for which the model has low confidence in its output, edge cases, or examples in previously underrepresented regions of the input feature space. In this way, the machine learning process prioritizes annotation efforts, i.e., training data is strategically selected for annotation based on which data is most uncertain / complex and / or useful to seed the model, rather than providing training data in a more random manner. The selected training examples are then sent to an annotator for annotation, and the model is then trained on the selected annotated data. This process is repeated until the model's scoring metrics improve. In this way, the model is trained only on the most relevant data, and full annotation only needs to be provided for a selected subset of data examples, rather than the full dataset that would be required if training data were selected using traditional methods. By learning from the most informative examples first, active learning tends to produce results similar to, or generally better than, traditional fully supervised methods.

[0006] Active learning is often used for classification problems where it is relatively easy to select samples from a pool of unlabeled data examples that will perturb the model. For example, in classification problems, output probabilities can be used as a criterion for selecting samples from a pool of unlabeled data based on the output probability values (e.g., by selecting data examples from the pool that the model classified with low confidence).

[0007] In the medical field, radiologists spend a lot of time segmenting images (outlining structures in the images) of lesions (e.g., tumors), organs at risk, and organs for treatment planning. Deep learning-based segmentation models can help radiologists optimize their workflow. In theory, active learning provides an efficient mechanism for selecting appropriate training data examples for labeling from a pool of unlabeled training examples, thus maximizing the model improvement for each annotation received while minimizing the annotation effort required by the radiologist. However, in practice, the criteria for selecting data examples from a pool of unlabeled samples are less well-defined for segmentation problems than for simpler classification problems and are generally more complex.

[0008] It is therefore an object of the embodiments herein to provide an improved method for selecting unlabeled medical images for annotation and then training a model for segmenting the medical images.

[0009] Thus, according to a first aspect, there is provided a computer-implemented method for use in selecting training data for annotation by a human annotator as part of an active learning machine learning process to train a model for taking a medical image as input and outputting a segmentation of the medical image, the method comprising the steps of: providing an unlabeled medical image as input to a model; obtaining a segmentation of the unlabeled medical image as output from the model; (a) determining whether geometric features of a segment of the segmentation satisfy one or more geometric criteria for a radiological feature represented by the segment; and / or (b) determining whether an edge region of the segment satisfies one or more edge criteria; and if the segment does not satisfy the one or more geometric criteria and / or the one or more edge criteria, selecting the unlabeled medical image as training data to be annotated by a human annotator as part of the active learning machine learning process.

[0010] According to a second aspect, there is a computer program product having a computer readable medium having computer readable code embodied therein, the computer readable code being configured, when executed by a suitable computer or processor, to cause the computer or processor to perform the method of the first aspect.

[0011] According to a third aspect, there is provided a system for use in selecting training data for annotation by a human annotator as part of an active learning machine learning process for training a model that takes a medical image as input and outputs a segmentation of the medical image. The system includes a memory having instruction data representing a set of instructions, and a processor in communication with the memory and configured to execute the set of instructions. The set of instructions, when executed by the processor, causes the processor to provide an unlabeled medical image as input to the model and obtain a segmentation of the unlabeled medical image as output from the model. The processor further determines whether (a) geometric features of a segment of the segmentation satisfy one or more geometric criteria for a radiological feature represented by the segment, and / or (b) determines whether an edge region of the segment satisfies one or more edge criteria. The processor further causes the unlabeled medical image to be selected as training data to be annotated by a human annotator as part of the active learning machine learning process if the segment does not satisfy the one or more geometric criteria and / or the one or more edge criteria.

[0012] These and other aspects will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief explanation of the drawings]

[0013] Exemplary embodiments will now be described, by way of example only, with reference to the following drawings, in which: [Figure 1] FIG. 1 is an apparatus according to some embodiments herein. [Figure 2] FIG. 2 illustrates a method according to some embodiments of the present disclosure. [Figure 3]FIG. 3 shows an example in which clusters of image components in the boundary region of a liver segment are compared with the central region of this segment to determine whether over-segmentation or under-segmentation has occurred. [Figure 4] FIG. 4 shows an example where volume information from an atlas of liver shape is compared with volume information of segments of the liver to determine over-segmentation or under-segmentation. [Figure 5] FIG. 5 illustrates an exemplary method according to an embodiment of the present disclosure. [Figure 6] FIG. 6 illustrates an exemplary GUI according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] As noted above, an objective of embodiments herein is to provide more efficient training of machine learning models for use in segmenting medical images. This objective is achieved herein through the use of active learning and through improved methods for selecting unlabeled medical images from a pool of unlabeled medical images for labeling by a human expert or oracle. By improving the selection of unlabeled medical images for labeling, the systems and methods herein save the time and resources of the human expert while providing models of the same or higher quality.

[0015] 1 , in some embodiments, there is an apparatus 100 for use in selecting training data for annotation by an annotator as part of an active learning machine learning process, according to some embodiments herein. Typically, the apparatus may form part of a computing device or system, such as, for example, a laptop, desktop computer, or other computing device. In some embodiments, the apparatus 100 may form part of a distributed computing configuration or cloud.

[0016] The apparatus includes a memory 104 having instruction data representing a series of instructions 106, and a processor 102 (e.g., a processing circuit or logic circuit) in communication with the memory and configured to execute the series of instructions 106. Generally, the series of instructions, when executed by the processor, cause the processor to perform any of the embodiments of method 200, as described below.

[0017] An embodiment of the apparatus 100 is for use in selecting training data for annotation by a human annotator as part of an active learning machine learning process, the selected training data being for use in training a model that takes a medical image as input and outputs a segmentation of the medical image. More specifically, the set of instructions, when executed by a processor, cause the processor to provide an unlabeled medical image as input to the model and obtain a segmentation of the unlabeled medical image as output from the model. The processor further a) determines whether geometric features of a segment of the segmentation satisfy geometric criteria for a radiological feature represented by the segment, and / or b) determines whether an edge region of the segment satisfies edge criteria, and if the segment does not satisfy the geometric and / or edge criteria, selects the unlabeled medical image as training data to be annotated by a human annotator as part of the active learning machine learning process.

[0018] Processor 102 may include one or more processors, processing units, multi-core processors, or modules configured or programmed to control device 100 in the manner described herein. In certain implementations, processor 102 may include multiple software and / or hardware modules, each configured or intended to perform individual or multiple steps of the methods described herein. Processor 102 may include one or more processors, processing units, multi-core processors, and / or modules configured or programmed to control device 100 in the manner described herein. In some implementations, for example, processor 102 may include multiple (e.g., interoperating) processors, processing units, multi-core processors, and / or modules configured for distributed processing. It will be understood by those skilled in the art that such processors, processing units, multi-core processors, and / or modules may be located in different locations and may perform different steps of the methods described herein and / or different portions of a single step.

[0019] The memory 104 is configured to store program code executed by the processor 102 to perform the methods described herein. Alternatively or additionally, one or more memories 104 may be external to the device 100 (i.e., separate or remote from the device). For example, one or more memories 104 may be part of another device. The memory 104 may be used to store medical images, segmentations, models, and / or any other information or data received, calculated, or determined by the processor 102 of the device 100, or received, calculated, or determined from any interface, memory, or device external to the device 100. The processor 102 may be configured to control the memory 104 to store the medical images, segmentations, models, and / or any other information.

[0020] In some embodiments, memory 104 may have multiple sub-memories, each capable of storing a portion of instruction data, such as at least one sub-memory capable of storing instruction data representing at least one instruction of the sequence of instructions, while at least one other sub-memory may store instruction data representing at least one other instruction of the sequence of instructions.

[0021] 1 shows only the components needed to illustrate this aspect of the disclosure, and it is understood that in an actual embodiment, device 100 may include additional components to those shown. For example, device 100 may further include a display. The display may include, for example, a computer screen and / or a screen on a mobile phone or tablet. The device may further include a user input device, such as a keyboard, mouse, or other input device, that allows a user to interact with the device, for example, to provide initial input parameters used in the methods described herein. Device 100 may include a battery or other power source for powering device 100, or a means for connecting device 100 to a mains power source.

[0022] The medical images referred to herein may be two-dimensional (2D) or three-dimensional (3D) images, and may be of any modality, such as a computed tomography (CT) image, a magnetic resonance (MR) image, an ultrasound (US) image, an X-ray image, a positron emission tomography (PET) image, a single photon emission computed tomography (SPECT) image, a nuclear medicine image, or any other medical image.

[0023] A medical image herein may be composed of (e.g., have) image components, which, as used herein, refer to pixels in a 2D image or voxels in a 3D image. A medical image herein may be in any suitable format, such as, for example, the Digital Imaging and Communications in Medicine (DICOM) image format.

[0024] The model is for use in segmenting medical images. Those skilled in the art will be familiar with segmentation (e.g., image segmentation), but in essence, segmentation involves extracting shape / morphological information about objects or shapes captured in an image. This is accomplished by converting the image into constituent blocks or "segments," where the pixels or voxels assigned to each segment share common attributes. For example, segmentation may involve outlining structures in a medical image (creating a structure outline), labeling image elements according to their corresponding structures, and / or adding color or texture to different structures in a medical image.

[0025] The systems and methods herein relate to machine learning (ML) segmentation, which uses ML models to convert an image into multiple constituent shapes (e.g., block shapes or block volumes) based on similar pixel / voxel values and image gradients. A recent review of the use of deep learning in image segmentation is described in the paper by S. Minaee, YY Boykov, F. Porikli, AJ Plaza, N. Kehtamavaz, and D. Terzopoulos, “Image Segmentation Using Deep Learning: A Survey,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109 / TPAMI.202L3059968.

[0026] The model can be any model trained using a machine learning process to take a medical image as input and output a segmentation of one or more structures in the medical image. For example, the model is trained to segment an anatomical feature (e.g., a portion of an anatomical structure) or a lesion(s) present in a particular anatomical feature. As an example, the model may be trained to segment the liver and / or liver lesions. However, this is merely an example, and the model can be trained to segment other anatomical features and / or lesions in other anatomical features.

[0027] Examples of models trained to perform image segmentation include, but are not limited to, convolutional neural networks, U-Nets, and encoder-decoder architectures (see Badrinarayanan et al., "SegNet," 2015). These are merely examples, and those skilled in the art will understand that the disclosure herein applies equally to any type of model trained to perform segmentation tasks on medical images that is trained using an active learning process.

[0028] The model herein may have initial training. For example, the model may be trained on an initial training dataset including training examples, each of which includes i) an example medical image and ii) a segmentation of the example medical image. The segmentation of each training example may be provided by a radiologist. Thus, the segmentation of each training example represents a "ground truth" segmentation for that training example. The initial training dataset may be selected in any manner (e.g., randomly, based on available annotated medical images, etc.). The model may be initially trained on the initial training dataset according to a known supervised machine learning process (e.g., using techniques such as backpropagation or gradient descent). For example, the initial training may be performed using an open-source library such as Scikit-learn, as described in the paper "Scikit-learn: Machine Learning in Python"; Pedregosa et al., JMLR 12, pp. 2825-2830, 2011.

[0029] Following the initial training, the model is then trained using an active learning process. Such subsequent or further training is performed, for example, to update or refine the model. The update may be for the purpose of retraining or improving the model. Further training may be performed to enable the model to be able to segment a wider range of input images (e.g., to extend the model), to enable the model to keep up with changes in the input images (e.g., due to quality improvements, different manufacturers of imaging equipment, new demographics of patient data, using the model at a different hospital), or for any other reason.

[0030] Typically, some initial training (as described above) is performed before active learning is performed, but it will be appreciated that active learning can be used directly on new models as well (e.g., using random weights and / or biases).

[0031] Those skilled in the art will be familiar with active learning, also known as query learning or in-product learning. Assuming that a large amount of unlabeled data is freely available, active learning uses a model to determine which unlabeled data needs to be labeled and fed to the model for retraining. For example, it determines which examples from a pool of unlabeled data samples should be labeled and used as training data to best improve the model. In this way, training data can be selectively labeled, better utilizing the expertise and limited resources of human labelers. Active learning techniques are outlined in a paper titled "Active Learning Literature Survey" by Settles, Burr (2009), published by the University of Wisconsin-Madison Department of Computer Sciences.

[0032] As used herein, a human expert may be referred to as a labeler, "oracle," or annotator of unlabeled data examples selected in the active learning process. An annotator is any expert qualified to segment data, e.g., typically a radiologist, who can provide correct ("ground truth") segmentations for training data selected from a pool of unlabeled data samples.

[0033] 2, there is a computer-implemented method 200 for use in selecting training data for annotation by an annotator as part of an active learning machine learning process, where the selected training data is for use in training a model that takes medical images as input and outputs segmentations of the medical images. For example, the model may output segmentations of anatomical features and / or lesions in the medical images. Embodiments of method 200 may be performed, for example, by an apparatus such as apparatus 100 described above.

[0034] Briefly, in a first step 202, the method 200 comprises providing an unlabeled medical image as input to a model and obtaining a segmentation of the unlabeled medical image as output from the model. In step 204, the method comprises a) determining whether geometric features of a segment of the segmentation satisfy geometric criteria for the radiological feature represented by the segment, and / or b) determining whether an edge region of the segment satisfies edge criteria. In step 206, if the segment does not satisfy the geometric and / or edge criteria, the method comprises selecting the unlabeled medical image as training data to be annotated by a human annotator as part of an active learning machine learning process.

[0035] The unlabeled medical images are obtained from a pool of unlabeled medical images, for example, from a hospital or any other database of hospital or related data. For example, the unlabeled medical images can be obtained from a medical picture archiving and communication system (PACS). "Unlabeled" in this sense means that they have not been sent to an oracle, for example, a radiologist, for labeling to obtain a ground truth segmentation.

[0036] The unlabeled medical images may be selected from a pool of unlabeled medical images according to the model's input parameters, e.g., the images are of the same modality and of the same anatomical features as those captured as input by the model. Furthermore, the objects (e.g., patients) in the images satisfy any demographic constraints that can be applied to valid inputs to the model (e.g., the model may be trained to capture only inputs for a particular patient's demographic data).

[0037] In step 202, an unlabeled medical image is provided (or fed) (in the usual way) as input to the model, which processes the unlabeled medical image and provides a segmentation of the medical image as output.

[0038] In option a) of step 204, it is determined whether the geometric features of the segmentation segment satisfy the geometric criteria of the radiological feature represented by this segment, as described above. In option b) (alternatively or in addition to option a), it is determined whether the edge region of the segment satisfies the edge criteria. Thus, by using the geometric and / or edge criteria, it is automatically identified whether the segment is likely to be correct or likely to be incorrect. If the segment is likely to be incorrect, in step 206, an unlabeled image is selected for annotation by an annotator (radiologist), and this image, together with its annotations (taken as ground truth), is used as training data for training the model.

[0039] As described above, the model outputs a segmentation of an unlabeled medical image, with segments (sometimes known as masks) corresponding to specific radiological features in the unlabeled medical image.

[0040] A radiological feature is an anatomical feature (e.g., an organ, the liver, the heart, a part of the vasculature, or any other anatomical feature), a lesion (e.g., a tumor), or any other feature depicted in an unlabeled medical image.

[0041] The geometric criteria may be criteria related to the particular shape of the radiological feature to which the segment corresponds. For example, if the segment corresponds to a lesion, the geometric criteria may relate to a typical (or expected) geometric shape associated with a lesion. If the segment corresponds to an anatomical feature, the geometric criteria may relate to a typical (or expected) geometric shape associated with a particular anatomical feature.

[0042] As will be explained in more detail below, the geometrical feature relates to the fractal dimension of the surface of the segment, the sphericity or any other shape feature that can be associated with the radiological feature to which the segment corresponds.

[0043] Generally, lesions in different organs have different geometric structures and constraints. For example, liver lesions are generally spherical in shape. Therefore, the inventors of the present application have realized that geometry-based inference can be used to select informative examples for annotation in an active learning framework.

[0044] In some embodiments, if an unlabeled medical image contains an image of a lesion (e.g., a tumor or cancer), a segment representing the lesion is validated (e.g., as likely to correspond to an actual tumor) according to the fractal dimension (FD) of the surface of the segment. Generally, FD increases with structural complexity and roughness. For cancers in particular, it has been shown that FD increases with aggressiveness. See the paper by Dobrescu titled "Diagnosis of Breast Cancer from Mammograms by Using Fractal Measures." Malignant tumors generally have higher FD because they are more bumpy and spiky than benign masses. Therefore, if a lower FD is obtained from a segment of a predicted lesion, it indicates either that the lesion is benign or that the model has under-segmented it. In either case, unlabeled medical images are selected and provided to an annotator for annotation to improve model performance. Therefore, a threshold fractal dimension can be placed on the FD of the segment of the predicted lesion structure. If the FD is less than the threshold fractal dimension, it is likely not a cancerous region, and therefore, it is an incorrect segmentation of the unlabeled medical image that should be selected in step 206 to reduce false positives. The value of the threshold fractal dimension varies depending on the segmentation task. For example, the value is set according to the type of anatomical structure and lesion to be detected. Therefore, an appropriate value is set by a subject matter expert, such as a human engineer. The appropriate value of the threshold fractal dimension may be set depending on the training requirements of the model. For example, if more training examples are needed, a wider range of fractal dimension values may be selected.

[0045] The fractal dimension of a complex structure such as a lesion can be calculated in many ways. As an example, the box counting method can be used. In the box counting method, the surface is covered with small grids. To calculate the fractal dimension, we determine how the number of grids that cover the lesion changes as the size of the grid is changed. If N(ε) boxes are required to cover a surface contour S with a grid of side length ε, then the fractal dimension (D) of the contour S is given by:

number

[0046] In other words, in some embodiments of method 200, the segment represents a lesion. In some embodiments, the lesion is a tumor. As described above, a lesion is, for example, a wound, a lesion, an infected area, or an abnormal area such as a tumor. For example, the model can label a segment of the segmentation as a lesion. In such an embodiment, the geometric feature is a fractal dimension of the surface of the segment, and the geometric criterion is based on a fractal dimension observed on the surface of the (actual) lesion. For example, determining 204 whether the geometric feature of a segment of the segmentation satisfies the geometric criterion of the radiological feature represented by this segment includes determining a fractal dimension of the surface of the segment, comparing the determined fractal dimension to a threshold fractal dimension, and selecting unlabeled medical images as training data if the determined fractal dimension is less than the threshold fractal dimension.

[0047] An appropriate threshold fractal dimension can be empirically determined, for example, from the fractal dimension of actual lesions. The threshold fractal dimension may also be based on the average (e.g., mode, mean, or median) fractal dimension observed in the lesions. The threshold fractal dimension is set according to the number of samples required for annotation and further training of the model (e.g., a higher threshold will be selected less than a lower threshold in step 206). It will be understood that this applies to embodiments in which the medical images are 2D or 3D.

[0048] In this way, a segment of a lesion can be verified as either correctly corresponding to a real lesion or corresponding to a false positive based on the fractal dimension of the surface of this segment.

[0049] In other embodiments, lesion segments may also be verified and / or determined to be false positives based on their sphericity in 3D images (or their circularity in 2D images). Sphericity values range from 0 to 1. Because lesions do not grow in a particular direction, it is expected that lesions will tend to have higher sphericity values. Thus, in some embodiments, unlabeled medical images for which the model outputs lesion segments with sphericity below a threshold sphericity are selected for annotation by an annotator in step 206 according to an active learning method.

[0050] The sphericity coefficient S is

number

[0051] In other words, in some embodiments, the segment represents a lesion, the geometric feature is based on the sphericity of the segment, and the geometric criterion is based on the sphericity observed in the (real) lesion. For example, determining 204 whether the geometric feature of a segment of the segmentation satisfies the geometric criterion of the radiological feature represented by this segment comprises determining a measure of sphericity of the segment, e.g., measure S described above, comparing the determined measure of sphericity to a threshold sphericity, and selecting unlabeled medical images as training data if the determined measure of sphericity is less than the threshold sphericity.

[0052] An appropriate threshold sphericity can be empirically determined, for example, from the sphericity of actual lesions. The threshold sphericity may also be based on the average (e.g., mode, mean, or median) sphericity observed in lesions. Furthermore, the threshold sphericity is set according to the number of samples needed for annotation and further training of the model (e.g., higher thresholds will result in fewer selections in step 206 than lower thresholds).

[0053] It is understood that sphericity is just one example, and any other measure of sphericity can be used as well. For example, a measure of how widespread the lesion is can be used as well, such as the ratio of the minor axis to the major axis. The criterion in such an example is that lesions with a major axis (minor axis / major axis<threshold) are likely to be mislabeled and should therefore be sent for annotation in step 206.

[0054] In this way, lesion segments can be verified as correctly corresponding to actual lesions or as corresponding to false positives based on the sphericity of the segments.

[0055] Turning now to other embodiments, in some embodiments, a shape atlas is used to determine whether a segment matches a radiological feature, for example, if a segment corresponds to a subject's liver, the segment is compared to a liver shape atlas.

[0056] In other words, in some embodiments, determining 204 whether the geometric features of a segment of the segmentation meet the geometric criteria of the radiological feature represented by this segment comprises fitting a shape to said segment from an atlas of shapes of radiological features, and selecting unlabeled medical images as training data if the overlap between the best-fitting shape from the shape atlas is less than a threshold overlap.

[0057] The threshold overlap may vary depending on the particular training requirements of the model and / or the number of training examples required, for example, the threshold overlap may be increased if too many unlabeled medical images are flagged for annotation, or decreased if more unlabeled images are needed.

[0058] More specifically, imaging of the human body typically involves segmenting internal organs, and most organs tend to have similar shapes. By mapping an atlas to all segmentation outputs, incorrectly segmented samples can be identified. Mapping the atlas is done by aligning the atlas to the mask / segment. The alignment method used can be modified to obtain a better estimate. One way to measure the quality of the segmented output is by the following equation:

number

[0059] The above formula helps to calculate the overlap coefficient, which is the intersection of the atlas area and the segmented area, and then divided by the atlas area. This gives a good estimate when there is a lot of under-segmentation observed after registration.

number

number

[0060] Thus, in embodiments where the radiological feature is a lesion or tumor associated with an organ, method 200 includes fitting a shape from a shape atlas to a second segment (in the segmentation) representing the organ. The best-fitting shape from this atlas can then be used to determine which organ the lesion is associated with or forms part of. The relative locations of the lesion and organ can be established. Furthermore, a false-positive label of a lesion associated with a particular organ is identified when a segment corresponding to this lesion is found not to be co-located with another segment corresponding to the particular organ. Co-location in this sense means that the segments at least partially overlap.

[0061] Generally, an appropriate atlas needs to be selected to calculate the above-mentioned atlas coefficients. The atlas selection process depends on the patient's demographic information, such as age, gender, race, and country of origin. For a particular age group within a particular gender, the atlas may be selected to make the atlas coefficients more "context-aware" and accurate. Because significant physiological variations in patients of different races and different countries have been observed, separate atlases may exist for each country or location.

[0062] A "context-aware" rule engine can be used to link patient-specific metadata with model performance, where patient-specific information such as gender, age group, geographic location, and / or organ is extracted to create an organ-specific atlas. The variation from such an atlas to the model's predicted segments (as output by the model) can be calculated using a distance matrix. The distance matrix (e.g., KL divergence) is used to select or reject images for active learning.

[0063] Figure 4 (top left) shows an exemplary CT slice through the liver 402. The image in the middle of Figure 4 shows an accurate segmentation of the liver 404, and the image on the right shows an inaccurate segmentation of the liver 406. Histograms are shown for the distribution of intensity values for the entire image slice 408, the accurate liver segmentation 410, and the inaccurate liver segmentation 412.

[0064] Statistical features, such as radiomics, and histogram-based features can be extracted from the segmented regions. For example, in the case of a CT scan, histogram analysis can be used to determine the performance of a segmentation model (i.e., liver or lung) from the data distribution, as shown in FIG. 4. This allows for determining whether the model is performing poorly. Organ-specific volumetric information can help determine the over- or under-segmentation behavior of the model. For example, in the example of FIG. 4, the circled region of the histogram corresponding to the incorrect liver segmentation 406 indicates over-segmentation and therefore contains image components that do not match the expected liver distribution, causing a second peak (circled) in the segment's distribution, which is not present in the histogram of the correct segment 410.

[0065] Kullback-Leibler (KL) Divergence D KL allows for the calculation / quantification of the divergence between two distributions. As given in the equation below, the KL divergence can be used to quantify the divergence between, for example, the histogram of the expected liver segmentation from the atlas, “Q”, and the histogram of the expected liver segmentation from the model, “P”.

number

[0066] Additionally, the nth percentile (e.g., 5th, 10th, 90th, 95th) of the predicted distribution (from the atlas) is calculated and compared to the distribution of segment outputs from the model.

number

[0067] As an example, for illustration purposes, Condition 1: Suppose N=999 and p=10. This results in R=100, and therefore the 100th element from the sorted array S is selected. Condition 2: Suppose N=100 and p=10, This gives R=10.1, so the integer part R I = 10 and the fractional part RF = 0.1.

[0068] In this way, the volume information of the segments and the atlas can also be compared. Thus, in step 204, the method 200 can include matching radiological features to an atlas of shapes of radiological features according to the demographic characteristics of the subject of the medical image. The method can then include obtaining predicted volume information for the radiological features from the matched shapes from the atlas of shapes. CT and MR scans are generally 3D scans, and volume information in this sense refers to any information (features) extracted over such 3D scans. The predicted volume information for the radiological features is then compared to the predicted volume information for the radiological features, e.g., using the D method described above, to determine whether there is under-segmentation or over-segmentation of the segments. KL and percentile-like measures may be used to compare the volumetric information of the segments. For example, a histogram of the predicted volumetric information from the atlas can be compared with a histogram of the volumetric information of the segments.

[0069] Referring now to other embodiments, in step 204, instead of or in addition to the geometric criteria herein, an edge criterion can be used to determine whether the edge region of the segment satisfies the edge criterion. This edge criterion can be a criterion that defines a clearly defined edge of the segment. Thus, the edge or edge region can be further used to determine whether the segment is likely to be over-segmented or under-segmented, and therefore whether the unlabeled medical image should be sent for annotation in step 206.

[0070] An edge region may be any region that surrounds a portion of an edge of a segment. In some examples, the edge region follows the contour of the surface of the segment, e.g., the edge region is a band that surrounds the edge region and follows the contour of the edge region (e.g., has outer and inner surfaces parallel to the surface of the segment). The edge region may alternatively be defined relative to the center of gravity, or centroid, of the segment, e.g., the edge region may be bounded by two concentric spheres centered on the centroid of the segment.

[0071] In some embodiments, the edge criterion is a threshold gradient between the value of an image component (e.g., pixel or voxel) inside a segment and the value of an adjacent image component outside the segment, for the segment to be classified as having a well-defined edge.

[0072] As an example, if the sum of the gradients in the original image along the edge / surface of each segment (in other words, at the contour of the predicted mask) is less than a threshold gradient, then this segment is labeled as an uncertain segmentation to be sent for annotation in step 206. For example, a gradient coefficient G for the contour outlining the segment can be calculated: N x , N y , N zare the coordinates of a set of points on the periphery of the predicted contour / segment S; dl are the derivatives of the segmented volume in the x, y and z directions; ∇ (x,y,z) is the gradient value, e.g., the change in intensity at each point on the periphery of the segment.

number

[0073] Then, in step 204, G can be compared to a threshold G value. The threshold value of the gradient coefficient G value may be set according to the training requirements. Generally, a lower G value indicates that the predicted segmentation is closer to the expected segmentation, and a higher G value indicates that the model is unable to predict the expected segmentation. Therefore, if the G value is greater than the threshold G value, samples are selected for active learning. The value of G may be set by a human engineer, and a higher G value will result in fewer unlabeled images being selected compared to a lower G value; therefore, lowering the threshold will increase the number of unlabeled medical images for annotation.

[0074] Referring now to other embodiments, the edge criterion may be a criterion based on statistical characteristics of clusters of image components in a boundary region (referred to herein as a penumbra region) surrounding the outer edge of the segment.

[0075] The bounding region may be defined relative to the center of gravity or center of mass of the segment. For example, the center of mass of the segment may be determined. The outer surface of the bounding region may then be defined as a sphere centered at the center of mass and located beyond the outer edge of the segment. The inner surface of the bounding region may then be defined as a sphere centered at the center of mass and located closer to the center of mass than the outer edge of the segment. The bounding region is then defined as the volume between the outer surface and the inner surface.

[0076] The outer face may be, for example, 30% farther away than the outer edge of the segment, and the inner face may be 30% closer to the outer edge of the segment, however, this is merely an example and the outer and inner faces may be at other distances depending on factors such as the shape of the segment and / or the number of images required for training.

[0077] Image components (voxels / pixels) can be clustered according to their values in boundary regions, for example, using known clustering techniques. Each cluster can then be compared to values in or near the central region of the segment to determine whether the cluster should be added to the segment. For example, the central region can be a sphere centered at the centroid of the segment. The central region can surround the center of mass of the segment. The central region is closer to the center of the segment than the boundary regions. Generally, the central region does not overlap with the boundary regions. The central region is sometimes called the core region of the segment.

[0078] Therefore, the method 200 may further comprise the steps of clustering the image components in the boundary regions into clusters, comparing features of each cluster with the image components in the central regions of the segments, and based on the comparison between the clusters and the central regions, selecting (206) unlabeled medical images as training data to be annotated by an annotator as part of the machine learning process of active learning.

[0079] In other words, step 206 may include selecting unlabeled medical images as training data if the values of the clusters of image components in the boundary regions are statistically similar (equivalent) to the values of the clusters of image components in the central region of the segment. Thus, the edge criterion is whether the image components in the boundary regions are statistically similar to the image components in the central region (and therefore should be added to the segment).

[0080] For example, for each cluster, a comparison can be made between the image component values in the cluster and the image component values in the central region. A comparison can be made between statistical features of the image component values in the cluster and corresponding statistical features of the image component values in the central region. This comparison can be made based on the radiomics features of each cluster. For example, the (Euclidean) distance between the feature values in the central region and the feature values in the cluster, the intensity variation in the cluster measured via, for example, the mean squared error (MSE), the entropy of the cluster, and / or a likelihood measure of the cluster compared to the central region can be determined. A score for each cluster can be determined based on one or more of the radiomics measures mentioned above. The scores for each cluster can be combined, for example, additively or multiply, to determine an overall score representing whether the clusters in the boundary regions are likely to be over- / under-segmented regions. This score can then be used as an indicator of whether the segmentation is good or bad and therefore whether an unlabeled image should be sent for annotation in step 206.

[0081] This is explained in more detail with respect to FIG. 3 , which shows an exemplary application of method 200 to segmentation of a 3D image of a liver 300 according to an embodiment of the present disclosure. In this example, during an initial pass of the model, in step 202, all image component (e.g., pixel / voxel) values in the image are scored to evaluate whether a given image component is part of the liver. The liver in this example is called the "intended segment" or region of interest (ROI). In step 204, a boundary region 302 is defined as described above. The boundary region is considered a "random error zone." This boundary region is determined according to the following steps: - Identifying the core 304 of the segment of the ROI (eg, the center of mass or centroid of the segment). - Identify a boundary region from the core of the ROI by drawing a vertical line 306 radially outward from the core of the ROI. The outer surfaces of the boundary region in Figure 3 are defined as surfaces that are 30% further out than the last identified image component in the ROI, e.g., 30% larger than the last segmentation boundary detected. This is shown in Figure 3 as the thick dashed line 308. Another inner circumference is generated that is 30% smaller than the last detected segmentation boundary and is used as the inner surface, which is shown in Figure 3 as thin dashed line 310. The boundary area 302 is the area between the outer and inner surfaces.

[0082] A heatmap zone can then be created within the boundary region to ascertain the probability of pixel values near the decision boundary. The heatmap is created by analyzing the variation in pixel intensity in the penumbra region (e.g., similar pixel intensities will get similar heat values in the heatmap).

[0083] The heatmap is further analyzed to quantitatively assess whether there is over-segmentation or under-segmentation. A scoring function is derived based on the heatmap to help arrive at a score for the boundary region. As mentioned above, the idea here is to compare extracted features in the core of the segment (where the segmentation is most certain) with the penumbra region (the region of the segment where there is high uncertainty). This comparison is performed using a distance metric (i.e., Euclidean distance) that represents the "distance" between each feature.

[0084] Score Calculation The boundary region is divided into different clusters by clustering regions with similar attributes, and a vector of radiomics features is calculated for each of these clusters, which may comprise, for example, measurements of image components in the respective clusters, such as the mean, variance, and / or kurtosis. For each cluster, we determine the Euclidean distance (ED) between the vector of radiomics features in each cluster and the (equivalent) vector of radiomics features measured in the central region of the segment. We can calculate the mean squared error (MSE), slice entropy (ENT), and slice-to-slice intensity variation via the likelihood measure (LLM) of individual slices for normal / abnormal anatomy observed near the segment centroid. This measure can be calculated as follows:

[0085] Euclidean distance

number

[0086] mean squared error

number

number

[0087] Log-likelihood ratio

number

[0088] The above measures can be combined into a single score or probability that a cluster is part of a segment.

number

[0089] Furthermore, the value of P(boundary area) can be used as a contributing factor to determine whether the area is correctly segmented or whether the image should be flagged for annotation in step 206. A higher value of P(boundary area) indicates better segmentation of the unlabeled medical image. A threshold value for P(boundary area) can be set to select cases where the model is failing. The method can include selecting unlabeled medical images for annotation that have a value of P(boundary area) lower than a P(boundary area) threshold.

[0090] Turning now to other embodiments, those skilled in the art will appreciate that combinations of the above measures may also be used. For example, a weighted combination of the fractal dimension D, the gradient coefficient G, and the sphericity S, as described above, may be expressed as, for example, a weighted sum G s =w1D+w2G+w3S By taking the coefficient G s ("geometric estimation coefficients").

[0091] G sA threshold value may be set for G. s If so, then in step 206 the medical image is sent for annotation.

[0092] The values of weights w1-w3 can be set depending on the type of segmentation. For example, w3 may be set to a higher level for lesion segmentation compared to organ segmentation (because sphericity is more relevant for lesion segmentation than for organ segmentation). w2 is set to a higher level for organ segmentation because edge shape is more important for organ segmentation compared to FD or sphericity. Another exemplary score is the "retrain confidence" measure

number

[0093] Again, this is a weighted combination of some of the above estimates, whereby these weights are set based on the particular segmentation problem: for example, for lesions, the atlas weights are set very low (near 0), while for organs, the atlas weights are set to larger values.

[0094] When retraining confidence scores for multiple unlabeled medical images are obtained (e.g., by repeating method 200 for each), the top n unlabeled medical images are selected for retraining, where the number n can be determined based on, for example, the time availability of labelers and / or the cost of obtaining ground truth annotations.

[0095] Returning to method 200, in step 206, if the segment does not satisfy the geometric and / or edge criteria, the method includes selecting unlabeled medical images as training data to be annotated by an annotator as part of the active learning machine learning process.

[0096] For example, an unlabeled medical image may be rejected if the segment does not meet one or more of the criteria described above. for example: If the fractal dimension is less than the threshold fractal dimension, If the sphericity is less than the threshold sphericity, If the gradient of the edge region is less than the threshold gradient, If the overlap between the best-fitting shape from the shape atlas and the segment is less than a threshold overlap, If the gradient at the edge region of the segment is less than the threshold gradient, It may be selected as training data to be annotated and used as training data.

[0097] The method then includes sending the unlabeled medical image to an annotator for annotation, at step 206. Those skilled in the art will appreciate that these are examples and that other geometric and / or edge criteria may be defined apart from those listed above or described elsewhere in this disclosure. Additionally, combinations of the above criteria may also be used.

[0098] Following step 206, the selected unlabeled medical image is sent to an annotator (e.g., a radiologist or other medical professional) to obtain a ground truth or "correct" segmentation for the unlabeled medical image. The medical image and the annotator's segmentation are then used as training data to (re)train the model.

[0099] In embodiments where probabilistic error zones (boundary regions) are determined, the annotator is encouraged to annotate only those regions that contain clusters where the system has low confidence, e.g., only in the error zone (the annotator does not have to annotate completely, but only annotates part of the mask, and the other part has to be verified), which is more efficient for the annotator.

[0100] The annotator checks and / or corrects errors in the segmentation (or re-segments the unlabeled medical image entirely anew), and the resulting annotated medical image is added to the training data. The model can be re-trained using both old and new updated training data to improve performance. The active learning process can be performed, for example, on an ad-hoc basis, e.g., once a week or once a month. This process helps keep the model robust in a production environment.

[0101] This is illustrated in Figure 5, which shows an example according to some embodiments herein. Figure 5 illustrates an embodiment of the method 200 described above. For example, the embodiment of Figure 5 may be performed by the apparatus 100 described above.

[0102] At 502, an unlabeled medical image is selected, and the unlabeled medical image is provided as input to a model at 504. The model outputs a (prediction of) a segmentation of the unlabeled medical image at 506. It will be understood that steps 502, 504, and 506 are performed as part of step 202 of method 200 as described above, and details thereof apply equally to steps 502, 504, and 506.

[0103] The segments of the segmentation are then evaluated by one or more of a geometry estimation module 508 , a random error zone module 510 , an atlas mapping module 512 and a context-aware rules engine module 514 .

[0104] The geometry estimation module determines whether the geometric features of a segment of the segmentation satisfy the geometric criteria of the radiological feature represented by the segment. Various exemplary geometric criteria are described in detail above with respect to step 204 (e.g., with respect to sphericity, FD, etc.), and it will be understood that the details therein apply to the geometry estimation module 508 as well.

[0105] The probabilistic error zone module 510 determines whether the edge region of the segment meets edge criteria by comparing statistical characteristics of clusters of image components in a boundary region surrounding the outer edge of the segment with those in a region closer to the center of the segment to determine over-segmentation or under-segmentation. This is as described above with respect to FIG. 3 and step 204 of method 200, and it will be understood that the details therein apply equally to the probabilistic error zone module 510.

[0106] The atlas mapping module 512 fits shapes from an atlas of anatomical feature shapes to the segments and selects unlabeled medical images as training data if the overlap between this shape and the best-fitting shape from the atlas is less than a threshold overlap. The threshold overlap is described above with respect to step 204, and it will be understood that the details therein apply to the atlas mapping module 512 as well.

[0107] The context-aware rules engine 514 uses patient-specific metadata along with the performance of the model, where patient-specific information, such as gender, age group, geographic location, and / or organ, is extracted to create a specific atlas. As described above with respect to step 204, the variation from such an atlas to the model's predicted segments (as output by the model) can be used as a distance matrix. This distance matrix is described above with respect to step 204, and it will be understood that the details therein apply equally to the context-aware rules engine 514.

[0108] If the segment does not meet the geometric and / or edge criteria, unlabeled medical data is selected for annotation by an annotator 516 and then used in retraining the model. However, if the segment meets the above criteria, annotation or retraining based on unlabeled medical images is not required and the method ends at 518.

[0109] In this way, there is a method and apparatus for selecting which unlabeled medical data should be sent for annotation and used for further training of a model for segmenting medical images.

[0110] FIG. 6 illustrates an exemplary graphical user interface (GUI) that can be used to train a model according to method 200 herein. GUI 600 is displayed on the display of device 100, described above. A user interface such as that shown in FIG. 6 can be created for active learning using method 200. In this case, a suggested retraining confidence / threshold range may be provided (602), specifying values for the thresholds described herein at which medical images should be evaluated and selected for annotation. The retraining mode may also be specified, for example, the minimum number of selected samples above which retraining should be initiated, or when training should be performed (e.g., overnight training). In the example of FIG. 6, the GUI shows three windows: two for displaying graphs 604, 606, and 608 of metrics of the model's performance on the test set (which may also be a validation set). An area 610 may be provided for displaying the selected unlabeled medical image (output in step 206 above) for annotation. Thus, a DICOM sequence can be opened in the DICOM viewer integrated in 610. After annotation, retraining can be initiated using only the UI, and the same steps are repeated for further iterations of active learning. Window 610 can display various statistical features of the unlabeled medical image.

[0111] Thus, a GUI such as that illustrated in FIG. 6 can be used to perform and customize the method 200 as described above.

[0112] In another embodiment, a computer program product is provided having a computer readable medium having computer readable code embodied therein, the computer readable code being configured, when executed by a suitable computer or processor, to cause the computer or processor to perform the methods described herein.

[0113] It will therefore be understood that the present disclosure also applies to computer programs adapted to implement the embodiments, in particular computer programs on or in a carrier, said programs being in the form of source code, object code, code intermediate sources and object code, for example in partially compiled form, or in any other form suitable for use in the implementation of the methods according to the embodiments described herein.

[0114] It will also be understood that such programs can have many different architectural designs. For example, program code implementing the functionality of the method or system may be subdivided into one or more subroutines. Many different ways of distributing the functionality among these subroutines will be apparent to those skilled in the art. The subroutines may be stored together in an executable file to form a self-contained program. Such an executable file may include computer-executable instructions, such as processor instructions and / or interpreter instructions (e.g., Java interpreter instructions). Alternatively, one or more or all of the subroutines may be stored in at least one external library file and linked with the main program statically or dynamically, e.g., at run time. The main program includes at least one call to at least one of the subroutines. The subroutines may also include function calls to each other.

[0115] The carrier of a computer program may be any entity or device capable of carrying the program. For example, the carrier may include a data storage device, for example a ROM, such as a CD-ROM or a semiconductor ROM, or a magnetic recording medium, for example a hard disk. Furthermore, the carrier may be a transmissible carrier, for example an electrical or optical signal, conveyed via an electrical or optical cable or by radio or other means. When the program is embodied in such a signal, the carrier may be constituted by such a cable or other device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, this integrated circuit being adapted to perform, or used in the performance of, the relevant method.

[0116] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in implementing the principles and techniques described herein, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, nor does a plurality of such elements or steps. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. A computer program may be stored or distributed on any suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, or in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference signs in the claims should not be construed as limiting their scope.

Claims

1. 1. A computer-implemented method for use in selecting training data for annotation by annotators as part of an active learning machine learning process, comprising: the selected training data is for use in training a model that takes medical images as input and outputs segmentations of the medical images; The method comprises: providing an unlabeled medical image as an input to the model and obtaining a segmentation of the unlabeled medical image as an output from the model; determining whether geometric features of a segment of said segmentation satisfy one or more geometric criteria of the radiological feature represented by said segment; determining whether an edge region of the segment satisfies one or more edge criteria; and If the segment does not satisfy the one or more geometric criteria and / or the one or more edge criteria, selecting the unlabeled medical image as training data to be annotated by the human as part of the active learning machine learning process.

10. A computer-implemented method comprising:

2. The segment represents a lesion. the geometric feature is the fractal dimension of the surface of the segment; and the one or more geometric criteria are based on a fractal dimension observed on the surface of the lesion; The method of claim 1.

3. The step of determining whether geometric features of a segment of the segmentation satisfy one or more geometric criteria of the radiological feature represented by the segment comprises: determining the fractal dimension of the surface of said segment; comparing the determined fractal dimension with a threshold fractal dimension; and selecting the unlabeled medical images as training data if the determined fractal dimension is less than the threshold fractal dimension.

3. The method of claim 2, comprising:

4. The segment represents a lesion. The geometric characteristics are based on the sphericity of the segments; and The one or more geometric criteria are based on the sphericity observed in the lesion. The method of claim 1, 2 or 3.

5. The step of determining whether geometric features of a segment of the segmentation satisfy one or more geometric criteria of the radiological feature represented by the segment comprises: determining a measure of sphericity of said segments; comparing the determined measure of sphericity to a threshold sphericity; and selecting the unlabeled medical image as training data if the determined measure of sphericity is less than the threshold sphericity.

5. The method of claim 4, comprising:

6. The step of determining whether geometric features of a segment of the segmentation satisfy one or more geometric criteria of the radiological feature represented by the segment comprises: fitting shapes from the atlas of shapes of radiological features to the segments and selecting the unlabeled medical images as training data if the overlap between the best-fit shape from the atlas of shapes is less than a threshold overlap.

2. The method of claim 1, comprising:

7. The radiological feature is a lesion in an organ, and the method comprises: fitting a shape from a shape atlas to a second segment representing the organ; and Using the best fitting shape from said shape atlas, the relative position of the lesion and the organ, and / or Whether the segment is a false positive labeling of a lesion because the lesion is not in the same location as the organ. determining 2. The method of claim 1, comprising:

8. The step of determining whether geometric features of a segment of the segmentation satisfy one or more geometric criteria of the radiological feature represented by the segment comprises: matching the radiological features to an atlas of shapes of the radiological features according to demographic characteristics of a subject of the medical images; obtaining predicted volumetric information for the radiological features from the matched shapes from the shape atlas; and comparing the expected volume information for the radiological feature with volume information for the segment to determine whether there is under-segmentation or over-segmentation of the segment relative to the expected volume information for the radiological feature.

2. The method of claim 1, comprising:

9. The step of comparing the predicted volume information for the radiological feature with volume information for the segment comprises: comparing the histogram of the predicted volume information with the histogram of the volume information of the segment; 9. The method of claim 8, comprising:

10. 2. The method of claim 1, wherein the one or more edge criteria is a threshold gradient between values of image components inside the segment and values of adjacent image components outside the segment for the segment to be classified as having a clear edge.

11. The method of claim 1 , wherein the one or more edge criteria are based on statistical characteristics of clusters of image components in a boundary region surrounding an outer edge of the segment.

12. The method comprises: determining the center of mass of the segment; defining the outer surface of the bounding region as a sphere centered at the center of mass and extending beyond the outer edge of the segment; defining an inner surface of the bounding region as a spherical surface centered at the center of mass and located closer to the center of mass than the outer edge of the segment; and determining the boundary region as the volume between the outer surface and the inner surface; 12. The method of claim 11, comprising:

13. clustering the image components in the boundary regions into clusters; comparing the features of each cluster with the image components in a central region of the segment; and selecting the unlabeled medical images as training data to be annotated by the human annotator as part of the active learning machine learning process based on a comparison between the clusters and the central region. The method of claim 12 further comprising:

14. 10. A computer program product having a computer readable medium having computer readable code thereon configured, when executed by a computer or processor, to cause said computer or processor to perform the method of claim 1.

15. 1. A system for use in selecting training data for annotation by an annotator as part of an active learning machine learning process, the selected training data for use in training a model that takes medical images as input and outputs a segmentation of the medical images, the system comprising: a memory having instruction data representing a sequence of instructions; a processor in communication with said memory and configured to execute said set of instructions; the set of instructions, when executed by the processor, causing the processor to: providing an unlabeled medical image as an input to the model and obtaining a segmentation of the unlabeled medical image as an output from the model; determining whether geometric features of a segment of the segmentation satisfy one or more geometric criteria of the radiological feature represented by the segment; determining whether an edge region of the segment satisfies one or more edge criteria; and If the segment does not satisfy the one or more geometric criteria and / or the one or more edge criteria, selecting the unlabeled medical image as training data to be annotated by the human annotator as part of the active learning machine learning process. system.

Images