Selecting training data for annotation

Active learning techniques in medical image segmentation selectively annotate the most uncertain data examples, addressing the challenge of frequent model updates and reducing manual effort, thereby maintaining model quality.

JP2024543956AActive Publication Date: 2024-11-26KONINKLIJKE PHILIPS NV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024532376
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-12-08
Filing Date
2022-11-17
Publication Date
2024-11-26
Estimated Expiration
2042-11-17

AI Technical Summary

Technical Problem

Existing machine learning models for medical image segmentation require frequent updates due to changing image quality and characteristics, necessitating significant manual annotation effort for new radiology images, which is time-consuming and skill-intensive.

Method used

Implementing active learning techniques to strategically select unlabeled data examples for annotation based on geometric and edge criteria, reducing the need for full dataset annotation by focusing on the most uncertain or complex examples.

Benefits of technology

This approach reduces the annotation burden on human experts while maintaining or improving the quality of segmentation models by selectively training on the most informative data, enhancing model performance efficiently.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024543956000001_ABST
    Figure 2024543956000001_ABST
Patent Text Reader

Abstract

A computer-implemented method for use in selecting training data for annotation by an annotator as part of an active learning machine learning process, where the selected training data is for use in training a model for taking a medical image as input and outputting a segmentation of the medical image. The method comprises the steps of providing an unlabeled medical image as input to the model and obtaining a segmentation of the unlabeled medical image as output from the model, a) determining whether geometric features of a segment of the segmentation satisfy geometric criteria for a radiological feature represented by the segment, and / or b) determining whether an edge region of the segment satisfies an edge criterion. The method then comprises the step of selecting the unlabeled medical image as training data to be annotated by an annotator as part of an active learning machine learning process if the segment does not satisfy the geometric and / or edge criteria.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This disclosure relates more particularly to machine learning, but non-exclusively, embodiments herein relate to selecting training data for annotation by an annotator as part of an active learning machine learning process to train a model for segmenting medical images. [Background technology]

[0002] Segmentation of anatomical structures and lesions is an essential part of treatment planning for cancer and other similar diseases. Manual segmentation of anatomical features is time-consuming and performed only by qualified expert radiologists. Therefore, various automated segmentation methods have been developed, some of which use deep learning to segment medical images into different segments or parts that correspond to different features in the respective images. With recent advances in the field of AI, machine learning (ML) models for segmentation are beginning to exceed human (individual) capabilities. ML models for segmentation are described in the paper by S. Minaee, YY Boykov, F. Porikli, AJ Plaza, N. Kehtamavaz and D. Terzopoulos, “Image Segmentation Using Deep Learning: A Survey,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109 / TPAMI.2021.3059968. Summary of the Invention [Problem to be solved by the invention]

[0003] These tools generally work well on data similar to the one on which the model was trained, but because the quality of the imaging and other image characteristics constantly change, the models need to be frequently updated and retrained as part of ongoing maintenance. Further training requires new annotated data, e.g., new radiology images that are manually segmented to provide the "ground truth" or correct annotations used to train the model. Obtaining annotations for segmentation problems requires significantly more time and skill than classification problems involving simple descriptive labels, so it is difficult to obtain datasets large enough to train AI models to segment medical images.

[0004] It is therefore an objective of the embodiments herein to reduce the annotation burden on experienced human annotators when generating training data for segmentation models, without compromising the quality of the resulting models. [Means for solving the problem]

[0005] Active learning / in-product learning is a supervised machine learning method in which certain unlabeled data examples are strategically selected for annotation and used in further training the model according to which data examples are most likely to provide the greatest benefit (e.g., provide the greatest improvement) to the training of the model. Active learning is a special case of supervised learning in which the active learning process itself can interactively query or request labels for the most complex data examples, or data examples that the model can label with the lowest confidence. For example, the model can select data examples for which the model has low confidence in its output, edge cases, or examples that are in areas of the input feature space that were previously underrepresented. In this way, the machine learning process prioritizes annotation efforts, i.e., training data is strategically selected for annotation based on which data is most uncertain / complex and / or informative to seed the model, rather than training data being provided in a more random manner. The selected training examples are then sent to annotators for annotation, and the model is then trained on the selected annotated data. This process is repeated until the model's scoring metrics are improved. In this way, models are trained only on the most important data, and full annotations need only be provided for a selected subset of data examples, rather than the full data set that would be required if training data were selected in a traditional way. By learning from the most informative examples first, active learning tends to produce results similar to, or generally better than, traditional fully supervised methods.

[0006] Active learning is often used for classification problems where it is relatively easy to select samples from a pool of unlabeled data examples that will perturb the model. For example, in a classification problem, the output probability can be used as a criterion for selecting samples from a pool of unlabeled data based on the output probability value (e.g., by selecting data examples from the pool that the model has classified with low confidence).

[0007] In the medical field, radiologists spend a lot of time segmenting images of lesions (e.g., tumors), organs at risk, and organs for treatment planning (outlining structures in the images). Deep learning-based segmentation models can help radiologists optimize their workflow. In theory, active learning provides an efficient mechanism to select appropriate training data examples for labeling from a pool of unlabeled training examples, thus maximizing the model improvement for each annotation received while minimizing the annotation effort required by the radiologist. However, in practice, the criteria for selecting data examples from a pool of unlabeled samples are not as well defined for segmentation problems as they are for simpler classification problems, and are generally more complex.

[0008] It is therefore an objective of the embodiments herein to provide an improved method for selecting unlabeled medical images for annotation and then training a model for segmenting the medical images.

[0009] Thus, according to a first aspect, there is provided a computer-implemented method for use in selecting training data for annotation by a human as part of an active learning machine learning process to train a model for taking a medical image as input and outputting a segmentation of the medical image, the method comprising the steps of: providing an unlabeled medical image as input to a model, obtaining a segmentation of the unlabeled medical image as output from the model, (a) determining whether geometric features of a segment of the segmentation satisfy one or more geometric criteria of a radiological feature represented by the segment, and / or (b) determining whether an edge region of the segment satisfies one or more edge criteria, and if the segment does not satisfy the one or more geometric criteria and / or the one or more edge criteria, selecting the unlabeled medical image as training data to be annotated by a human as part of an active learning machine learning process.

[0010] According to a second aspect there is a computer program product having a computer readable medium having computer readable code embodied therein, the computer readable code being configured, when executed by a suitable computer or processor, to cause the computer or processor to perform the method of the first aspect.

[0011] According to a third aspect, there is a system for use in selecting training data for annotation by an annotator as part of an active learning machine learning process for training a model for taking a medical image as input and outputting a segmentation of the medical image. The system includes a memory having instruction data representing a set of instructions, and a processor in communication with the memory and configured to execute the set of instructions. The set of instructions, when executed by the processor, causes the processor to provide an unlabeled medical image as an input to the model and to obtain a segmentation of the unlabeled medical image as an output from the model. The processor further determines whether (a) geometric features of a segment of the segmentation satisfy one or more geometric criteria of a radiological feature represented by the segment, and / or (b) determines whether an edge region of the segment satisfies one or more edge criteria. The processor further causes the unlabeled medical image to be selected as training data to be annotated by an annotator as part of an active learning machine learning process if the segment does not satisfy the one or more geometric criteria and / or the one or more edge criteria.

[0012] These and other aspects will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]

[0013] Exemplary embodiments will now be described, by way of example only, with reference to the following drawings, in which: [Figure 1] FIG. 1 is an apparatus according to some embodiments of the present disclosure. [Diagram 2] FIG. 2 is a method according to some embodiments of the present disclosure. [Diagram 3]FIG. 3 shows an example in which clusters of image components in the boundary region of a liver segment are compared with the central region of this segment to determine whether over-segmentation or under-segmentation has occurred. [Figure 4] FIG. 4 shows an example where volumetric information from an atlas of liver shape is compared with volumetric information of segments of the liver to determine over-segmentation or under-segmentation. [Diagram 5] FIG. 5 illustrates an exemplary method according to an embodiment of the present disclosure. [Figure 6] FIG. 6 illustrates an exemplary GUI according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] As mentioned above, an objective of the embodiments herein is to provide more efficient training of machine learning models for use in segmenting medical images. This objective is achieved herein through the use of active learning and through improved methods of selecting unlabeled medical images from a pool of unlabeled medical images for labeling by a human expert or oracle. By improving the selection of unlabeled medical images for labeling, the systems and methods herein save the time and resources of the human expert while providing models of the same or higher quality.

[0015] 1, in some embodiments, there is an apparatus 100 for use in selecting training data for annotation by an annotator as part of an active learning machine learning process, in accordance with some embodiments herein. Typically, the apparatus may form part of a computing device or system, such as, for example, a laptop, desktop computer, or other computing device. In some embodiments, the apparatus 100 may form part of a distributed computing arrangement or cloud.

[0016] The apparatus includes a memory 104 having instruction data representing a series of instructions 106, and a processor 102 (e.g., a processing circuit or logic circuit) in communication with the memory and configured to execute the series of instructions. In general, the series of instructions, when executed by a processor, causes the processor to perform any of the embodiments of a method 200, as described below.

[0017] An embodiment of the apparatus 100 is for use in selecting training data for annotation by an annotator as part of an active learning machine learning process, the selected training data being for use in training a model for taking a medical image as input and outputting a segmentation of the medical image. More specifically, the set of instructions, when executed by a processor, causes the processor to provide an unlabeled medical image as input to the model and obtain a segmentation of the unlabeled medical image as output from the model. The processor further a) determines whether geometric features of a segment of the segmentation meet geometric criteria for a radiological feature represented by the segment, and / or b) determines whether an edge region of the segment meets edge criteria, and if the segment does not meet the geometric criteria and / or the edge criteria, selects the unlabeled medical image as training data to be annotated by an annotator as part of an active learning machine learning process.

[0018] The processor 102 may have one or more processors, processing units, multi-core processors, or modules configured or programmed to control the device 100 in the manner described herein. In certain implementations, the processor 102 may have multiple software and / or hardware modules, each configured or intended to perform individual or multiple steps of the methods described herein. The processor 102 may have one or more processors, processing units, multi-core processors, and / or modules configured or programmed to control the device 100 in the manner described herein. In some implementations, for example, the processor 102 may have multiple (e.g., interoperated) processors, processing units, multi-core processors, and / or modules configured for distributed processing. It will be understood by those skilled in the art that such processors, processing units, multi-core processors, and / or modules may be located in different locations and may perform different steps of the methods described herein and / or different portions of a single step.

[0019] The memory 104 is configured to store program code executed by the processor 102 to perform the methods described herein. Alternatively or additionally, one or more memories 104 may be external to the device 100 (i.e., separate or remote from the device). For example, one or more memories 104 may be part of another device. The memory 104 is used to store medical images, segmentations, models, and / or any other information or data received, calculated, or determined by the processor 102 of the device 100 or received, calculated, or determined from any interface, memory, or device external to the device 100. The processor 102 may be configured to control the memory 104 to store the medical images, segmentations, models, and / or any other information.

[0020] In some embodiments, memory 104 may have multiple sub-memories, each capable of storing a portion of instruction data, such as at least one sub-memory capable of storing instruction data representing at least one instruction of the sequence of instructions, while at least one other sub-memory may store instruction data representing at least one other instruction of the sequence of instructions.

[0021] It is understood that Fig. 1 shows only the components required to illustrate this aspect of the disclosure, and that in a practical embodiment, the device 100 may have additional components to those shown. For example, the device 100 may further comprise a display. The display may comprise, for example, a computer screen and / or a screen on a mobile phone or tablet. The device may further comprise a user input device, for example a keyboard, mouse or other input device, that allows a user to interact with the device, for example to provide initial input parameters used in the methods described herein. The device 100 may comprise a battery or other power source for powering the device 100, or a means for connecting the device 100 to a mains power source.

[0022] The medical images referred to in this specification may be two-dimensional (2D) images or three-dimensional (3D) images. The medical images may be of any modality, such as a computed tomography (CT) image, a magnetic resonance (MR) image, an ultrasound (US) image, an X-ray image, a positron emission tomography (PET) image, a single photon emission computed tomography (SPECT) image, a nuclear medicine image, or any other medical image.

[0023] A medical image herein may be composed of (e.g., have) image components, which as used herein refer to pixels in a 2D image or voxels in a 3D image. A medical image herein may be in any suitable format, such as, for example, the Digital Imaging and Communications in Medicine (DICOM) image format.

[0024] The model is for use in segmenting medical images. Those skilled in the art will be familiar with segmentation (e.g., image segmentation), but in essence, segmentation involves extracting shape / morphological information about objects or shapes captured in an image. This is accomplished by converting the image into constituent blocks or "segments," where the pixels or voxels assigned to each segment have common attributes. For example, segmentation may include outlining (creating an outline of) structures present in the medical image, labeling image components according to their corresponding structures, and / or adding color or texture to different structures in the medical image.

[0025] The systems and methods herein relate to machine learning (ML) segmentation, where an ML model is used to convert an image into multiple constituent shapes (e.g., block shapes or block volumes) based on similar pixel / voxel values ​​and image gradients. A recent review of the use of deep learning in image segmentation is described in the paper by S. Minaee, YY Boykov, F. Porikli, AJ Plaza, N. Kehtamavaz and D. Terzopoulos, “Image Segmentation Using Deep Learning: A Survey,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109 / TPAMI.202L3059968.

[0026] The model can be any model trained using a machine learning process to take a medical image as input and output a segmentation of one or more structures in the medical image. For example, the model is trained to segment an anatomical feature (e.g., a portion of an anatomical structure) or a lesion(s) present in a particular anatomical feature. As an example, the model may be trained to segment the liver and / or lesions in the liver. However, this is merely an example and the model can be trained to segment other anatomical features and / or lesions in other anatomical features.

[0027] Examples of models trained to perform image segmentation include, but are not limited to, convolutional neural networks, U-Net, and encoder / decoder architectures (see Badrinarayanan et al., "SegNet," 2015). These are merely examples, and one of ordinary skill in the art will appreciate that the disclosure herein applies equally to any type of model trained to perform segmentation tasks on medical images that are trained using an active learning process.

[0028] The model herein may have an initial training. For example, the model may be trained on an initial training dataset with training examples, each training example having i) an example medical image, and ii) a segmentation of the example medical image. The segmentation of each training example may be provided by a radiologist. Thus, the segmentation of each training example represents a "ground truth" segmentation for that training example. The initial training dataset may be selected in any manner (e.g., randomly, based on available annotated medical images, etc.). The model may be initially trained on the initial training dataset according to a supervised machine learning process (e.g., using techniques such as backpropagation, gradient descent, etc.) in a known manner. For example, the initial training is performed using an open source library such as Scikit-learn, as described in the paper "Scikit-learn: Machine Learning in Python"; Pedregosa et al., JMLR 12, pp. 2825-2830, 2011.

[0029] Following the initial training, the model is then trained using an active learning process. Such subsequent or further training is performed, for example, to update or refine the model. The update may be for the purpose of retraining or improving the model. Further training is performed to enable the model to be able to segment a wider range of input images (e.g., to extend the model), to enable the model to keep up with changes in the input images (e.g., due to quality improvements, different manufacturers of imaging equipment, etc., new demographics of patient data, using the model at a different hospital), or for any other reason.

[0030] Typically, some initial training (as described above) is performed before active learning is undertaken, although it will be appreciated that active learning can be used directly on new models as well (e.g., using random weights and / or biases).

[0031] Those skilled in the art will be familiar with active learning, also known as query learning or in-product learning. Assuming that a large amount of unlabeled data is freely available, active learning determines, with the help of a model, which unlabeled data needs to be labeled and fed to the model for retraining. For example, it determines which examples from a pool of unlabeled data samples should be labeled and used as training data to best improve the model. In this way, training data can be selectively labeled, making better use of the expertise and limited resources of (human) labelers. Active learning techniques are outlined in an article entitled "Active Learning Literature Survey" by Settles, Burr (2009), published by the University of Wisconsin-Madison Department of Computer Sciences.

[0032] As used herein, a human expert may be referred to as a labeler, "oracle," or annotator of unlabeled data examples selected in the active learning process. An annotator is any expert qualified to segment data, e.g., typically a radiologist who can provide correct ("ground truth") segmentations for training data selected from a pool of unlabeled data samples.

[0033] 2, there is a computer-implemented method 200 for use in selecting training data for annotation by an annotator as part of an active learning machine learning process, the selected training data for use in training a model that takes medical images as input and outputs a segmentation of the medical images. For example, the model may output a segmentation of anatomical features and / or lesions in the medical images. An embodiment of method 200 may be performed by an apparatus such as apparatus 100 described above.

[0034] Briefly, in a first step 202, the method 200 comprises providing an unlabeled medical image as input to a model and obtaining a segmentation of the unlabeled medical image as output from the model. In step 204, the method comprises a) determining whether geometric features of a segment of the segmentation satisfy geometric criteria of the radiological feature represented by the segment, and / or b) determining whether an edge region of the segment satisfies an edge criterion. In step 206, if the segment does not satisfy the geometric and / or edge criteria, the method comprises selecting the unlabeled medical image as training data to be annotated by a human annotator as part of an active learning machine learning process.

[0035] The unlabeled medical images are obtained from a pool of unlabeled medical images, for example from a hospital or any other database of hospital or related data. For example, the unlabeled medical images can be obtained from a medical picture archiving and management system (PACS). "Unlabeled" in this sense means that they have not been sent to an oracle, for example a radiologist, for labeling to obtain a ground truth segmentation.

[0036] The unlabeled medical images may be selected from a pool of unlabeled medical images according to the input parameters of the model, e.g., the images are of the same modality and of the same anatomical features as those taken as input by the model, and further, the objects (e.g., patients) in the images satisfy any demographic constraints that may be applied to valid inputs to the model (e.g., the model may be trained to only take in inputs for the demographic data of a particular patient).

[0037] In step 202, an unlabeled medical image is provided (or fed) (in the usual way) as input to the model. The model processes the unlabeled medical image and provides a segmentation of the medical image as output.

[0038] In option a) of step 204, it is determined whether the geometric features of the segment of the segmentation satisfy the geometric criteria of the radiological features represented by this segment, as described above. In option b) (alternatively or in addition to option a)) it is determined whether the edge regions of the segment satisfy the edge criteria. Thus, by using the geometric and / or edge criteria, it is automatically identified whether the segment is likely to be correct or likely to be incorrect. If the segment is likely to be incorrect, in step 206, an unlabeled image is selected for annotation by an annotator (radiologist), so that this image together with its annotations (taken as ground truth) is used as training data for training the model.

[0039] As described above, the model outputs a segmentation of an unlabeled medical image, which has segments (sometimes known as masks) that correspond to particular radiological features in the unlabeled medical image.

[0040] A radiological feature is an anatomical feature (eg, an organ, the liver, the heart, a part of the vasculature, or any other anatomical feature), a lesion (eg, a tumor), or any other feature that is depicted in an unlabeled medical image.

[0041] The geometric criteria may be criteria related to a particular shape of the radiological feature to which the segment corresponds. For example, if the segment corresponds to a lesion, the geometric criteria relate to a typical (or expected) geometric shape associated with a lesion. If the segment corresponds to an anatomical feature, the geometric criteria relate to a typical (or expected) geometric shape associated with a particular anatomical feature.

[0042] As will be explained in more detail below, the geometrical feature relates to the fractal dimension of the surface of the segment, the sphericity or any other shape feature that can be related to the radiological feature to which the segment corresponds.

[0043] In general, lesions in different organs have different geometric structures and constraints, for example, liver lesions are generally spherical in shape, and therefore the inventors of the present application have realized that geometry-based inference can be used to select informative examples for annotation in an active learning framework.

[0044] In some embodiments, if an unlabeled medical image has an image of a lesion (e.g., a tumor or cancer), a segment representing this lesion is verified (e.g., as likely to correspond to a real tumor) according to the fractal dimension (FD) of the surface of this segment. In general, the FD increases with the complexity and roughness of the structure. For cancers in particular, it has been proven that the FD increases with aggressiveness. See the paper by Dobrescu entitled “Diagnosis of Breast Cancer from Mammograms by Using Fractal Measures”. Malignant tumors generally have a higher FD because they are more bumpy and spiky than benign masses. Thus, if a lower FD is obtained from a segment of a predicted lesion, it indicates that either the lesion is benign or the model has performed under segmentation. Either way, an unlabeled medical image is selected and provided to an annotator for annotation to improve the model performance. Thus, a threshold fractal dimension can be placed on the FD of the segment of the predicted lesion structure. If the FD is less than the threshold fractal dimension, it is likely not a cancer region, and therefore it is a wrong segmentation of the unlabeled medical image that should be selected in step 206 to reduce false positives. The value of the threshold fractal dimension varies depending on the task of the segmentation. For example, the value is set according to the type of anatomical structure and lesion to be detected. Therefore, the appropriate value is set by a subject matter expert, e.g., a human engineer. The appropriate value of the threshold fractal dimension may be set depending on the training requirements of the model, for example, if more training examples are required, a wider range of fractal dimension values ​​may be selected.

[0045] The fractal dimension of a complex structure such as a lesion can be calculated in a number of ways. As an example, the box counting method can be used. In the box counting method, a surface is covered with small grids. To calculate the fractal dimension, we determine how the number of grids that cover the lesion changes as we change the size of the grid. If N(ε) boxes are required to cover a surface contour S with a grid of side length ε, then the fractal dimension (D) of the contour S is given by:

number

[0046] In other words, in some embodiments of the method 200, the segment represents a lesion. In some embodiments, the lesion is a tumor. As mentioned above, a lesion is, for example, a wound, a wound, an infected area, or an abnormal area such as a tumor. For example, the model can label a segment of the segmentation as a lesion. In such an embodiment, the geometric feature is a fractal dimension of the surface of the segment, and the geometric criterion is based on a fractal dimension observed on the surface of the (real) lesion. For example, determining 204 whether the geometric feature of a segment of the segmentation satisfies the geometric criterion of the radiological feature represented by this segment comprises determining a fractal dimension of the surface of the segment, comparing the determined fractal dimension to a threshold fractal dimension, and selecting an unlabeled medical image as training data if the determined fractal dimension is less than the threshold fractal dimension.

[0047] An appropriate threshold fractal dimension can be empirically determined, for example, from the fractal dimension of actual lesions. The threshold fractal dimension may be based on the average (e.g., mode, mean or median) fractal dimension observed in the lesions. The threshold fractal dimension is set according to the number of samples required for annotation and further training of the model (e.g., a higher threshold will be selected less than a lower threshold in step 206). It will be understood that this applies to embodiments in which the medical images are 2D or 3D.

[0048] In this way, a segment of a lesion can be verified as either correctly corresponding to a real lesion or corresponding to a false positive based on the fractal dimension of the surface of this segment.

[0049] In other embodiments, a lesion segment may also be verified and / or determined to be a false positive based on its sphericity in the 3D image (or its circularity in the 2D image). Sphericity values ​​range from 0 to 1. Since lesions do not grow in a particular direction, it is expected that lesions tend to have higher sphericity values. Thus, in some embodiments, unlabeled medical images for which the model outputs lesion segments with sphericity below a threshold sphericity are selected for annotation by an annotator in step 206 according to an active learning method.

[0050] The sphericity coefficient S is

number

[0051] In other words, in some embodiments, the segment represents a lesion, the geometric feature is based on the sphericity of the segment, and the geometric criterion is based on the sphericity observed in the (real) lesion. For example, determining 204 whether the geometric feature of a segment of the segmentation satisfies the geometric criterion of the radiological feature represented by this segment comprises determining a measure of sphericity of the segment, e.g., measure S above, comparing the determined measure of sphericity to a threshold sphericity, and selecting unlabeled medical images as training data if the determined measure of sphericity is less than the threshold sphericity.

[0052] An appropriate threshold sphericity can be determined empirically, for example, from the sphericity of actual lesions. The threshold sphericity may also be based on the average (e.g., mode, mean, or median) sphericity observed in the lesions. Furthermore, the threshold sphericity is set according to the number of samples required for annotation and further training of the model (e.g., a higher threshold will be selected less than a lower threshold in step 206).

[0053] It is understood that sphericity is just one example and any other measure of sphericity may be used as well. For example, a measure of how spread out the lesion is may be used as well. For example, the ratio of minor to major axis. The criterion in such an example is that lesions with a major axis (minor / major < threshold) are likely to be mislabeled and therefore should be sent for annotation in step 206.

[0054] In this way, lesion segments can be verified as either correctly corresponding to an actual lesion or corresponding to a false positive based on the sphericity of the segment.

[0055] Turning now to other embodiments, in some embodiments, a shape atlas is used to determine whether a segment matches a radiological feature, for example, if a segment corresponds to a subject's liver, the segment is compared to a liver shape atlas.

[0056] In other words, in some embodiments, determining 204 whether the geometric features of a segment of the segmentation satisfy the geometric criteria of the radiological feature represented by this segment comprises fitting a shape from an atlas of radiological feature shapes to said segment, and selecting unlabeled medical images as training data if the overlap between the best fitting shape from the shape atlas is less than a threshold overlap.

[0057] The threshold overlap may vary depending on the particular training requirements of the model and / or the number of training examples required, for example, the threshold overlap may be increased if too many unlabeled medical images are flagged for annotation, or decreased if more unlabeled images are needed.

[0058] More specifically, imaging of the human body usually involves segmenting internal organs, and most organs tend to have similar shapes. By mapping an atlas to all segmentation outputs, incorrectly segmented samples can be identified. Mapping the atlas is done by aligning the atlas to the mask / segments. The alignment method used can be modified to obtain a better estimate. One way to measure the quality of the segmented output is by the following equation:

number

[0059] The above formula helps to calculate the amount of overlap (overlap coefficient), where the intersection of the atlas region and the segmented region is taken and then divided by the atlas region. This gives a good estimate when there is a lot of under-segmentation observed after registration.

number

number

[0060] Thus, in an embodiment where the radiological feature is a lesion or tumor associated with an organ, the method 200 comprises fitting a shape from an atlas of shapes to a second segment (in the segmentation) representing the organ. The best fitting shape from this atlas can then be used to determine which organ the lesion is associated with or forms part of. The relative positions of the lesion and organ can be established. Furthermore, a false positive label of a lesion associated with a particular organ is identified when a segment corresponding to this lesion is found not to be in the same position as another segment corresponding to said particular organ. Co-position in this sense means that these segments at least partially overlap.

[0061] In general, an appropriate atlas needs to be selected to calculate the above-mentioned atlas coefficients. The atlas selection process depends on the demographic information of the patient, such as age, sex, race, and country of origin. For a particular age group of a particular gender, the atlas may be selected to make the atlas coefficients more "context aware" and accurate. Since it has been observed that there is a large variability in the physiology of patients from different races and different countries, there may be separate atlases for each country or location.

[0062] A "context-aware" rule engine can be used to link patient-specific metadata with the performance of the model, where patient-specific information such as gender, age group, geographic location and / or organ is extracted to create an organ-specific atlas. The variation from such an atlas to the model's predicted segments (as output by the model) can be calculated using a distance matrix. The distance matrix (e.g., KL divergence) is used to select or reject images for active learning.

[0063] Figure 4 (top left) shows an example CT slice through the liver 402. The image in the middle of Figure 4 shows an accurate segmentation of the liver 404, and the image on the right shows an inaccurate segmentation of the liver 406. Histograms for the distribution of intensity values ​​for the entire image slice 408, the accurate liver segmentation 410, and the inaccurate liver segmentation 412 are shown.

[0064] From the segmented regions, statistical features such as radiomics and histogram-based features can be extracted. For example, in the case of a CT scan, histogram analysis can be used to determine the performance of the segmentation model (i.e., liver or lung) from the data distribution, as shown in FIG. 4. This allows for the determination of whether the model is performing poorly. Organ-specific volumetric information helps to determine the over- or under-segmentation behavior of the model. For example, in the example of FIG. 4, the circled region of the histogram corresponding to the incorrect segmentation of the liver 406 indicates over-segmentation and therefore contains image components that do not match the expected liver distribution, causing a second peak (circled) in the distribution of the segments, which is not present in the histogram of the correct segment 410.

[0065] Kullback-Leibler (KL) Divergence D KL allows for the calculation / quantification of the divergence between two distributions. As given in the formula below, the KL divergence can be used to quantify the divergence between, for example, the histogram of the expected liver segmentation from the atlas, “Q”, and the histogram of the expected liver segmentation from the model, “P”.

number

[0066] Also, the nth percentile (e.g., 5th, 10th, 90th, 95th) of the predicted distribution (from the atlas) is calculated and this is compared to the distribution of segment outputs from the model.

number

[0067] As an example, for illustration purposes, Condition 1: Suppose N=999 and p=10, This results in R=100, and therefore the 100th element from the sorted array S is selected. Condition 2: Suppose N=100 and p=10, This gives us R=10.1, so the integer part R I = 10 and the fractional part RF = 0.1.

[0068] In this way, the volume information of the segments and the atlas can also be compared. Thus, the method 200 can comprise, in step 204, a step of matching the radiological features to an atlas of shapes of radiological features according to the demographic characteristics of the subject of the medical image. The method can then comprise a step of obtaining expected volume information for the radiological features from the matched shapes from the atlas of shapes. CT and MR scans are generally 3D scans, and volume information in this sense means any information (features) extracted over such a 3D scan. The expected volume information for the radiological features is then compared to the expected volume information for this radiological feature, e.g., according to the D-segmentation method described above, to determine whether there is an under-segmentation or over-segmentation of the segment. KL For example, a histogram of the predicted volumetric information from the atlas can be compared to a histogram of the volumetric information of the segments.

[0069] Referring now to other embodiments, in step 204, instead of or in addition to the geometric criteria herein, an edge criterion can be used to determine whether an edge region of a segment satisfies the edge criterion. The edge criterion can be a criterion that defines a well-defined edge of the segment. Thus, the edge or edge region can be further used to determine whether the segment is likely to be an over-segmentation or under-segmentation, and therefore whether the unlabeled medical image should be sent for annotation in step 206.

[0070] An edge region may be any region that surrounds a portion of an edge of a segment. In some examples, the edge region follows the contour of the surface of the segment, e.g., the edge region is a band that surrounds the edge region and follows the contour of the edge region (e.g., with outer and inner surfaces parallel to the surface of the segment). The edge region may alternatively be defined with respect to the center of gravity, or centroid, of the segment, e.g., the edge region may be bounded by two concentric spheres centered on the centroid of the segment.

[0071] In some embodiments, the edge criterion is a threshold gradient between the values ​​of image components (e.g., pixels or voxels) inside a segment and the values ​​of adjacent image components outside the segment for the segment to be classified as having a well-defined edge.

[0072] As an example, if the sum of the gradients in the original image along the edges / surfaces of each segment (in other words, at the contour of the predicted mask) is less than a threshold gradient, the segment is labeled as an uncertain segmentation to be sent for annotation in step 206. For example, a gradient coefficient G for the contour outlining the segment can be calculated. N x , N y , N zare the coordinates of a set of points on the periphery of the predicted contour / segment S; dl are the derivatives of the segmented volume in x, y and z directions; ∇ (x,y,z) is the gradient value, e.g., the change in intensity at each point on the perimeter of the segment.

number

[0073] Then, in step 204, G can be compared to a threshold G value. The threshold value of the gradient coefficient G value may be set according to the requirements of training. In general, a lower G value indicates that the predicted segmentation is closer to the expected segmentation, and a higher G value indicates that the model is unable to predict the expected segmentation. Thus, if the G value is greater than the threshold G value, samples are selected for active learning. The value of G may be set by a human engineer, and a higher G value will result in fewer unlabeled images being selected compared to a lower G value, thus lowering the threshold will increase the number of unlabeled medical images for annotation.

[0074] Turning now to other embodiments, the edge criterion may be a criterion based on statistical characteristics of clusters of image components in a boundary region (referred to herein as a penumbra region) surrounding the outer edge of the segment.

[0075] A bounding region may be defined relative to the center of gravity or center of mass of a segment. For example, the center of mass of a segment may be determined. An outer surface of the bounding region may then be defined as a sphere centered on the center of mass and located beyond the outer edge of the segment. An inner surface of the bounding region may then be defined as a sphere centered on the center of mass and located closer to the center of mass than the outer edge of the segment. The bounding region is then defined as the volume between said outer surface and said inner surface.

[0076] The outer face may be, for example, 30% farther away than the outer edge of the segment, and the inner face may be 30% closer to the outer edge of the segment, however, this is merely one example and the outer and inner faces may be at other distances depending on factors such as, for example, the shape of the segment and / or the number of images required for training.

[0077] Image components (voxels / pixels) can be clustered according to their values ​​in the boundary regions, for example using known clustering techniques. Each cluster can then be compared to values ​​in or near the central region of the segment to determine whether the cluster should be added to the segment. For example, the central region can be a sphere centered on the centroid of the segment. The central region can surround the center of mass of the segment. The central region is closer to the center of the segment than the boundary regions. In general, the central region does not overlap with the boundary regions. The central region is sometimes called the core region of the segment.

[0078] Therefore, the method 200 may further comprise the steps of clustering the image components in the boundary regions into clusters, comparing features of each cluster with the image components in the central regions of the segments, and based on said comparison between the clusters and the central regions, selecting (206) unlabeled medical images as training data to be annotated by a human annotator as part of an active learning machine learning process.

[0079] In other words, step 206 may include selecting unlabeled medical images as training data if the values ​​of the clusters of image components in the boundary regions are statistically similar (comparable) to the values ​​of the clusters of image components in the central region of the segment. Thus, the edge criterion is whether the image components in the boundary regions are statistically similar (and therefore should be added to the segment) to the image components in the central region.

[0080] For example, for each cluster, a comparison can be made between the image component values ​​in the cluster and the image component values ​​in the central region. A comparison can be made between statistical features of the image component values ​​in the cluster and corresponding statistical features of the image component values ​​in the central region. This comparison can be made based on the radiomics features of each cluster. For example, the (Euclidean) distance between the feature values ​​in the central region and the feature values ​​in the cluster, the intensity variation in the cluster measured, for example, via the mean squared error (MSE), the entropy of the cluster and / or the likelihood measure of the cluster compared to the central region is determined. A score for each cluster is determined based on one or more of the radiomics measures mentioned above. The scores for each cluster can be combined, for example, as an addition or multiplication, to determine an overall score representing whether the clusters in the boundary regions are likely to be over / under segmented regions. This score is then used as an indication of whether the segmentation is a good or bad segmentation and therefore whether an unlabeled image should be sent for annotation in step 206.

[0081] This is explained in more detail with respect to FIG. 3, which illustrates an exemplary application of the method 200 to segmentation of a 3D image of a liver 300 according to an embodiment of the present specification. In this example, during an initial pass of the model, in step 202, all image component (e.g., pixel / voxel) values ​​in the image are scored to evaluate whether a given image component is part of the liver. The liver in this example is called the "intended segment" or region of interest (ROI). In step 204, a boundary region 302 is defined as described above. The boundary region is considered to be a "random error zone". This boundary region is determined according to the following steps: - Identify the core 304 of the segment of the ROI (eg, the center of mass or centroid of the segment). - Identify a border region from the core of the ROI by drawing a vertical line 306 radially outward from the core of the ROI. The faces outside the bounding region in Fig. 3 are defined as faces that are 30% further out than the last identified image component in the ROI, for example 30% larger than the last segmentation boundary detected, which is shown in Fig. 3 as the thick dashed line 308. - Another inner circumference is generated that is 30% smaller than the last detected segmentation boundary and this is used as the inner face, this is shown in figure 3 as thin dashed line 310. The boundary area 302 is the area between the outer surface and the inner surface.

[0082] A heatmap zone can then be created within the boundary region to ascertain the probability of pixel values ​​near the decision boundary. The heatmap is created by analyzing the variation of pixel intensities in the penumbra region (e.g., similar pixel intensities get similar heat values ​​in the heatmap).

[0083] The heatmap is further analyzed to quantitatively assess whether there is over-segmentation or under-segmentation. A scoring function is derived based on the heatmap to help arrive at a score for the boundary region. As mentioned above, the idea here is to compare the extracted features in the core of the segment (where the segmentation is most certain) with the penumbra region (area of ​​the segment where there is high uncertainty). This comparison is done using a distance metric (i.e., Euclidean distance) that represents the "distance" between each feature.

[0084] Scoring We divide the border regions into different clusters by clustering regions with similar attributes and calculate a vector of radiomics features for each of these clusters, which may comprise, for example, measurements of image content in the respective clusters, such as the mean, variance and / or kurtosis. For each cluster, we determine the Euclidean distance (ED) between the vector of radiomics features in each cluster and the (equivalent) vector of radiomics features measured in the central region of the segment. We can calculate the mean square error (MSE), slice entropy (ENT) and slice-to-slice intensity variation via the likelihood measure (LLM) of individual slices for normal / abnormal anatomy observed near the segment centroid. This measure can be calculated as follows:

[0085] Euclidean distance

number

[0086] Mean Square Error

number

number

[0087] Log-likelihood ratio

number

[0088] The above measures can be combined into a single score or probability that a cluster is part of a segment.

number

[0089] Furthermore, the value of P(boundary area) can be used as a contributing factor to determine whether the area is correctly segmented or whether the image should be flagged for annotation in step 206. A higher value of P(boundary area) indicates a better segmentation of the unlabeled medical image. A threshold value of P(boundary area) can be set to select cases where the model is failing. The method can include selecting unlabeled medical images for annotation that have a value of P(boundary area) lower than a P(boundary area) threshold.

[0090] Turning now to other embodiments, those skilled in the art will appreciate that combinations of the above measures may also be used. For example, a weighted combination of the fractal dimension D, the gradient coefficient G, and the sphericity S, as described above, may be expressed as, for example, a weighted sum G s =w1D+w2G+w3S By taking the coefficient G s ("geometric estimation coefficients").

[0091] G sA threshold may be set for the G s If so, then in step 206 the medical image is sent for annotation.

[0092] The values ​​of weights w1-w3 can be set depending on the type of segmentation. For example, w3 may be set to a higher level for lesion segmentation compared to organ segmentation (since sphericity is more relevant for lesion segmentation than for organ segmentation). w2 is set to a higher level for organ segmentation since edge shape is more important for organ segmentation compared to FD or sphericity. Another exemplary score is a measure of "retraining confidence"

number

[0093] Again, this is a weighted combination of some of the estimates mentioned above, whereby these weights are set based on the particular segmentation problem: for example, for lesions, the atlas weights are set very low (near 0), whereas for organs, the atlas weights are set to larger values.

[0094] When retraining confidence scores for multiple unlabeled medical images are obtained (e.g., by repeating method 200 for each), the top n unlabeled medical images are selected for retraining. The number n can be determined, for example, based on the time availability of labelers and / or the cost of obtaining ground truth annotations.

[0095] Returning to method 200, in step 206, if the segment does not meet the geometric and / or edge criteria, the method includes selecting an unlabeled medical image as training data to be annotated by a human annotator as part of the active learning machine learning process.

[0096] For example, an unlabeled medical image may be classified as a segment if the segment does not meet one or more of the above criteria: for example: If the fractal dimension is less than the threshold fractal dimension, If the sphericity is less than the threshold sphericity, If the gradient of the edge region is less than the threshold gradient, If the overlap between the best fitting shape from the shape atlas and the segment is less than a threshold overlap, If the gradient at the edge region of the segment is less than the threshold gradient, It may be selected as training data to be annotated and used as training data.

[0097] Then, in step 206, the method includes sending the unlabeled medical image to an annotator for annotation. Those skilled in the art will appreciate that these are examples and that other geometric and / or edge criteria may be defined apart from those listed above or described elsewhere in this disclosure. Additionally, combinations of the above criteria may also be used.

[0098] Following step 206, the selected unlabeled medical image is sent to an annotator (e.g., a radiologist or other medical professional) to obtain a ground truth or "correct" segmentation for the unlabeled medical image. The medical image and the annotator's segmentation are then used as training data to (re)train the model.

[0099] In an embodiment where probabilistic error zones (boundary regions) are determined, the annotator is prompted to annotate only those regions that contain clusters where the system has low confidence, e.g., only in the error zone (the annotator does not have to annotate completely, but only annotates parts of the mask, the other parts have to be verified), which is more efficient for the annotator.

[0100] The annotator checks and / or corrects errors in the segmentation (or re-segments the unlabeled medical image entirely anew), and the resulting annotated medical image is added to the training data. The model can be re-trained with both old and new updated training data to improve performance. The active learning process can be performed, for example, on an ad-hoc basis, for example, once a week or once a month. This process helps to keep the model robust in the production environment.

[0101] This is illustrated in Figure 5, which shows an example according to some embodiments herein. Figure 5 illustrates an embodiment of the method 200 described above. For example, the embodiment of Figure 5 may be performed by the apparatus 100 described above.

[0102] At 502, an unlabeled medical image is selected, which is provided as an input to a model at 504. The model outputs a (prediction of) a segmentation of the unlabeled medical image at 506. It will be understood that steps 502, 504 and 506 are performed as part of step 202 of method 200 as described above, details of which apply equally to steps 502, 504 and 506.

[0103] The segments of the segmentation are then evaluated by one or more of a geometry estimation module 508 , a random error zone module 510 , an atlas mapping module 512 , and a context aware rules engine module 514 .

[0104] The Geometry Estimation module determines whether the geometric features of a segment of the segmentation satisfy the geometric criteria of the radiological feature represented by this segment. Various exemplary geometric criteria are described in detail above with respect to step 204 (e.g., with respect to sphericity, FD, etc.), and it will be understood that the details therein apply to the Geometry Estimation module 508 as well.

[0105] The probabilistic error zone module 510 determines whether an edge region of a segment satisfies an edge criterion by comparing statistical characteristics of clusters of image components in a boundary region surrounding the outer edge of the segment with those in a region closer to the center of the segment to determine over-segmentation or under-segmentation. This is as described above with respect to FIG. 3 and step 204 of method 200, and it will be understood that the details therein apply equally to the probabilistic error zone module 510.

[0106] The atlas mapping module 512 fits shapes from an atlas of anatomical feature shapes to the segments and selects unlabeled medical images as training data if the overlap between the shape and the best fitting shape from the atlas is less than a threshold overlap. The threshold overlap is described above with respect to step 204, and it will be understood that the details therein apply to the atlas mapping module 512 as well.

[0107] The context aware rules engine 514 uses patient specific metadata together with the performance of the model, where patient specific information, such as gender, age group, geographic location and / or organs, is extracted to create a specific atlas. As described above with respect to step 204, the variation from such atlas to the predicted segments of the model (as output by the model) can be used as a distance matrix. This distance matrix is ​​described above with respect to step 204, and it will be understood that the details therein apply equally to the context aware rules engine 514.

[0108] If the segment does not meet the geometric and / or edge criteria, unlabeled medical data is selected for annotation by annotators 516 and then used in retraining the model. However, if the segment meets the above criteria, annotation or retraining based on unlabeled medical images is not required and the method ends at 518.

[0109] In this manner, there is a method and apparatus for selecting which unlabeled medical data should be sent for annotation and used for further training of a model for segmenting medical images.

[0110] FIG. 6 shows an exemplary graphical user interface (GUI) that can be used to train a model according to the method 200 herein. The GUI 600 is displayed on the display of the device 100 described above. A user interface such as that shown in FIG. 6 can be created to perform active learning using the method 200. In this case, a suggested retraining confidence / threshold range may be given (602), specifying values ​​for the thresholds described herein at which medical images should be evaluated and selected for annotation. The mode of retraining can also be specified, for example, a minimum number of selected samples above which retraining should be initiated, or when training should be performed (e.g. overnight training). In the example of FIG. 6, the GUI shows three windows, two for showing graphs 604, 606, 608 of metrics of the model's performance on a test set (which may also be a validation set). An area 610 may be provided for displaying the selected unlabeled medical image (output in step 206 above) for annotation. Thus, a DICOM sequence can be opened in a DICOM viewer integrated in 610. After annotation, retraining can be initiated using only the UI, and the same steps are repeated for further iterations of active learning. Window 610 can display various statistical features of the unlabeled medical image.

[0111] Thus, a GUI such as that illustrated in FIG. 6 may be used to perform and customize the method 200 as described above.

[0112] In another embodiment, a computer program product is provided having a computer readable medium having computer readable code embodied therein, which, when executed by a suitable computer or processor, is configured to cause the computer or processor to perform the methods described herein.

[0113] It will therefore be understood that the present disclosure also applies to computer programs adapted to implement the embodiments, in particular computer programs on or in a carrier, said programs being in the form of source code, object code, code intermediate sources and object code, for example in partially compiled form, or in any other form suitable for use in the implementation of the methods according to the embodiments described herein.

[0114] It will also be appreciated that such a program can have many different architectural designs. For example, the program code implementing the functionality of the method or system may be subdivided into one or more subroutines. Many different ways of distributing the functionality among these subroutines will be apparent to those skilled in the art. The subroutines can be stored together in one executable file to form a self-contained program. Such an executable file may comprise computer executable instructions, such as processor instructions and / or interpreter instructions (e.g. Java interpreter instructions). Alternatively, one or more or all of the subroutines may be stored in at least one external library file and linked statically or dynamically, e.g. at run-time, with the main program. The main program includes at least one call to at least one of the subroutines. The subroutines may also have function calls to each other.

[0115] The carrier of a computer program may be any entity or device capable of carrying the program. For example, the carrier may comprise a data storage device, for example a ROM, such as a CD-ROM or a semiconductor ROM, or a magnetic recording medium, for example a hard disk. Furthermore, the carrier is a transmissible carrier, for example an electrical or optical signal, conveyed via an electrical or optical cable or by radio or other means. When the program is embodied in such a signal, the carrier may be constituted by such a cable or other device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, the integrated circuit being adapted to perform, or used in the performance of, the relevant method.

[0116] Variations to the disclosed embodiments can be understood and implemented by those skilled in the art in implementing the principles and techniques described herein, from a study of the drawings, the disclosure and the appended claims. In the claims, the term "comprising" does not exclude other elements or steps, nor does it exclude a plurality of them. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. A computer program may be stored or distributed on a suitable medium, for example an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, for example via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be interpreted as limiting the scope thereof.

Claims

1. 1. A computer-implemented method for use in selecting training data for annotation by an annotator as part of an active learning machine learning process, comprising: the selected training data is for use in training a model that takes as input a medical image and outputs a segmentation of the medical image; The method comprises: providing an unlabeled medical image as an input to the model and obtaining a segmentation of the unlabeled medical image as an output from the model; determining whether the geometric features of a segment of the segmentation satisfy one or more geometric criteria of the radiological feature represented by the segment and / or determining whether the edge region of the segment satisfies one or more edge criteria; and if the segment does not satisfy the one or more geometric criteria and / or the one or more edge criteria, selecting the unlabeled medical image as training data to be annotated by the human as part of the active learning machine learning process.

13. A computer-implemented method comprising:

2. The segment represents a lesion. the geometric feature is the fractal dimension of the surface of the segment; and the one or more geometric criteria are based on a fractal dimension observed on a surface of the lesion; The method of claim 1.

3. The step of determining whether a geometric feature of a segment of the segmentation satisfies one or more geometric criteria of a radiological feature represented by the segment, further comprises: determining a fractal dimension of the surface of said segment; comparing the determined fractal dimension with a threshold fractal dimension; and selecting the unlabeled medical images as training data if the determined fractal dimension is less than the threshold fractal dimension. The method of claim 2, comprising:

4. The segment represents a lesion. the geometric characteristics are based on the sphericity of the segments; and The one or more geometric criteria are based on the sphericity observed in the lesion. The method according to claim 1, 2 or 3.

5. The step of determining whether a geometric feature of a segment of the segmentation satisfies one or more geometric criteria of a radiological feature represented by the segment, further comprises: determining a measure of sphericity of said segments; comparing the determined measure of sphericity with a threshold sphericity; and selecting the unlabeled medical images as training data if the determined measure of sphericity is less than the threshold sphericity.

5. The method of claim 4, comprising:

6. The step of determining whether a geometric feature of a segment of the segmentation satisfies one or more geometric criteria of a radiological feature represented by the segment, further comprises: fitting shapes from the atlas of shapes of radiological features to the segments and selecting the unlabeled medical images as training data if the overlap between the best-fit shape from the atlas of shapes is less than a threshold overlap.

6. The method according to claim 1 , further comprising:

7. The radiological feature is a lesion in an organ, and the method comprises: fitting a shape from an atlas of shapes to a second segment representing the organ; and Using the best fitting shape from the shape atlas, the relative position of the lesion and the organ, and / or Whether the segment is a false positive labeling of a lesion because the lesion is not in the same location as the organ determining 6. The method according to claim 1 , further comprising:

8. The step of determining whether a geometric feature of a segment of the segmentation satisfies one or more geometric criteria of a radiological feature represented by the segment, further comprises: matching said radiological features to an atlas of shapes of said radiological features according to demographic characteristics of a subject of said medical images; obtaining predicted volumetric information for the radiological features from the matched shapes from the shape atlas; and comparing the expected volumetric information for the radiological feature with volumetric information for the segment to determine whether there is under-segmentation or over-segmentation of the segment relative to the expected volumetric information for the radiological feature.

6. The method according to claim 1 , further comprising:

9. The step of comparing the predicted volumetric information for the radiological feature with volumetric information for the segments comprises: comparing the histogram of the predicted volumetric information with a histogram of the volumetric information of the segment.

9. The method of claim 8, having the following structure:

10. 10. The method of claim 1, wherein the one or more edge criteria is a threshold gradient between values ​​of image components inside the segment and values ​​of adjacent image components outside the segment for the segment to be classified as having a clear edge.

11. The method of claim 1 , wherein the one or more edge criteria are based on statistical characteristics of clusters of image components in a boundary region surrounding an outer edge of the segment.

12. The method comprises: determining a center of mass of said segment; defining the outer surface of the bounding region as a sphere centered at the center of mass and located beyond the outer edge of the segment; defining an inner surface of the bounding region as a sphere centered at the center of mass and located closer to the center of mass than the outer edge of the segment; and determining the boundary region as the volume between the outer surface and the inner surface; 12. The method of claim 11, having the following structure:

13. clustering image components in the boundary regions into clusters; comparing the features of each cluster with the image content in a central region of the segment; and selecting the unlabeled medical images as training data to be annotated by the human annotator as part of the active learning machine learning process based on a comparison between the clusters and the central region. The method of claim 12 further comprising:

14. 14. A computer program product having a computer readable medium having computer readable code embodied therein, the computer readable code, when executed by a suitable computer or processor, configuring the computer or processor to perform a method according to any one of claims 1 to 13.

15. 1. A system for use in selecting training data for annotation by a human annotator as part of an active learning machine learning process, the selected training data for use in training a model that takes as input medical images and outputs a segmentation of the medical images, the system comprising: a memory having instruction data representing a sequence of instructions; and A processor in communication with the memory and configured to execute the set of instructions. the sequence of instructions, when executed by the processor, causes the processor to: providing an unlabeled medical image as an input to the model and obtaining a segmentation of the unlabeled medical image as an output from the model; determining whether geometric features of a segment of the segmentation satisfy one or more geometric criteria of the radiological feature represented by the segment, and / or determining whether edge regions of the segment satisfy one or more edge criteria; and If the segment does not satisfy the one or more geometric criteria and / or the one or more edge criteria, selecting the unlabeled medical image as training data to be annotated by the human annotator as part of the active learning machine learning process. system.

Citation Information

Patent Citations

  • Ultrasound cardiac doppler study automation

    JP2020075104A

  • Medical information processing device, medical information processing model training method, and medical information processing program

    JP2021120852A