Image analysis method and device

Abstract

An image analysis method and device for detecting failures or errors in an image segmentation procedure includes comparing (14) the segmentation results for two or more images, the two or more images representing a particular anatomical region at different time points, and identifying the degree of consistency or deviation therebetween. A measure of accuracy of the segmentation procedure is determined (16) based on the derived consistency or deviation measure.

Description

[Technical field]

[0001] The present invention relates to a method and device for the analysis of images, and in particular to a method and device for assessing the correctness of a segmentation procedure applied to an image. [Background technology]

[0002] Image segmentation is a procedure in digital image processing that divides a digital image into multiple segments (i.e., collections of pixels), each covering some area or region of the image. It is most often used to identify the location of objects and boundaries (e.g., lines, curves) in an image. Different segments are typically identified and differentiated based on some similar shared graphical characteristic, e.g., color, intensity, or texture, of the pixels contained in the segmented regions. Adjacent segmented regions will differ with respect to these same attributes.

[0003] In the field of medical imaging, segmentation is useful for identifying various anatomical structures or features in a captured image, such as organs, air and fluid passages (such as blood vessels or the digestive tract), valves or chambers, etc. The resulting contours after image segmentation allow a physician to more quickly evaluate the medical image (enabling faster diagnostic or treatment decisions). Segmentation may also be beneficially fed into subsequent processing techniques. For example, a series or a large number of images of a given area may be processed using the segmentation information, for example with the aid of an interpolation algorithm such as the marching cubes algorithm, to produce a 3D reconstruction.

[0004] One important area of ​​medical imaging is ultrasound, which plays an important role in the evaluation and diagnosis of a wide range of areas of the body. In particular, the use of ultrasound in cardiac imaging is very useful.

[0005] Diagnosis, treatment and planning are often based on the use of clear delineation of relevant anatomical regions or features in ultrasound images. For example, in the field of cardiac imaging, these regions may include the ventricles, atria or surrounding blood vessels. Segmentation is frequently used to automate this delineation process.

[0006] Many organs or anatomical objects in the body undergo cyclical patterns of motion, such as the motion of the heart and its internal chambers during the cardiac cycle (cycling between systole and diastole as the heart beats), or, for example, the contraction and expansion of the lungs during the respiratory cycle.

[0007] Model-based image segmentation is often used for medical imaging. Model-based segmentation uses, for example, a predefined model of a particular anatomical region to model the typical shape, size, texture, or other graphical features characteristic of a particular anatomical structure or feature. These model properties can be imposed as constraints on the segmentation, for example, the model is fitted to the image data, thereby enabling segmentation of the anatomical feature.

[0008] Despite improvements, image segmentation is frequently unreliable. Automatic delineation of structures can frequently fail, for example due to poor image quality, anatomical anomalies, or unexpected imaging fields. There can be a number of reasons for failure of the segmentation process. For example, the algorithm may misinterpret the input image content, or the image may lack sufficient resolution or signal content to allow accurate delineation of borders and lines in certain areas. For example, model-based segmentation relies on evaluating image features such as gradients or textures. Poor ultrasound signal quality or loss of signal data in certain anatomical regions can lead to treatment failure. This can lead to inaccurate delineation of image features and therefore unreliable segmentation results.

[0009] More specifically, failure to accurately identify the properties or features (i.e., pixel characteristics) of the underlying image can lead to an incorrect fitting (i.e., adaptation) of the model to the image, which in turn results in inaccurate and inconsistent segmentation results.

[0010] Failures in the segmentation procedure can manifest themselves in the form of unrealistic distortions of model shapes in the segmentation product, or they can manifest as model shapes that appear reliable but in reality deviate significantly from the true underlying anatomical structures.

[0011] Detecting when segmentation fails is crucial to avoid clinicians making medical decisions based on inaccurate segmentation results, which is most important when the segmentation effort appears reliable in shape despite failing.

[0012] A typical approach to detect segmentation failures in current technology is to compare the segmentation result with the original image and look for characteristic irregularities that indicate failures. However, this approach is very difficult. In particular, many segmentation failures occur due to poor quality or missing signal data in certain areas. These same areas of low signal often lead to similar errors being introduced by the checking algorithm itself, leading to ambiguity about which represent successful results and which represent failed results.

[0013] Therefore, an improved technique for detecting segmentation errors would be useful.

[0014] The paper "An Improved Algorithm for Coronary Bypass Anastomosis Segmentation in Epicardial Ultrasound Sequences" by JORGENSEN ALEX SKOVSBO et al., ULTRASOUND IN MEDICINE AND BIOLOGY, vol. 42, no. 12, pages 3010-3021, describes a method for automatically segmenting coronary artery bypass anastomoses in in vivo endoscopic ultrasound (EUS) sequences. In particular, an automatic anastomosis segmentation algorithm is used to extract areas of anastomotic structures from in vivo EUS sequences obtained from 16 porcine anastomoses. The algorithm consists of four main components: vessel detection, vessel segmentation, segmentation quality control, and frame-to-frame contour alignment. The accuracy of the segmentation is evaluated using m-fold cross-validation based on 830 manual segmentations of anastomotic structures.

[0015] The document "Evaluation of a multi-atlas based method for segmentation of cardiac CTA data: a large-scale, multicenter, and multivendor study" by Kl RIAA LI H et al., MEDICAL PHYSICS, AIP, MELVILLE, NY, US, vol. 37, no. 12, November 18, 2010, pages 6279-6291, concerns a method for segmenting cardiac computed tomography angiography (CTA) data. In particular, an automatic, multi-atlas based method for segmenting the whole heart (i.e., the outer surface of the pericardium) and the ventricles from CTA data is discussed. In this segmentation approach, eight atlas images are registered to a new patient's CTA scan. The eight corresponding manually labeled images are then propagated and combined using a voxel-wise majority voting procedure to obtain the cardiac segmentation.

[0016] The document "Learning image context for segmentation of the prostate in CT-guided radiotherapy" by WEI LI et al., PHYSICS IN MEDICINE AND BIOLOGY, INSTITUTE OF PHYSICS PUBLISHING, BRISTOL, GB, vol. 57, no. 5, February 17, 2012, pp. 1283-1308, relates to a method for segmenting the prostate in CT images based on the use of a classifier algorithm, where the classifier is updated or trained for each newly derived segmentation result. In particular, an online learning and patient-specific classification method is presented. Two sets of location-adaptive classifiers are respectively positioned along two coordinate directions in the planning image space of the patient, and are further trained by the planning images and also by previously segmented treatment images of the same patient. Each location-adaptive classifier is recursively trained by both static image appearance features and iteratively updated image context features. Summary of the Invention

[0017] The invention is defined by the claims.

[0018] According to an embodiment in accordance with an aspect of the present invention, there is provided an image analysis method for determining failures in an image segmentation procedure for identifying and segmenting one or more anatomical features in an image, the image analysis method comprising: obtaining an image segmentation of two or more images of an anatomical region, the two or more images representing the anatomical region at different respective time points, each segmented using an image segmentation procedure; comparing one or more similar segmented anatomical features segmented in each of the image segmentations and deriving one or more measures of consistency between the anatomical features in the different image segmentations; determining a measure of accuracy of the image segmentation procedure based on one or more consistency measures; An image analysis method is provided comprising:

[0019] The method may, for example, be a computer-implemented method.

[0020] The invention is based on the insight that failures in segmentation algorithms are very sensitive to small changes in the input data. This is found to be manifested as large variations in the segmentation outcome for images obtained at different times (where imaging conditions, e.g. probe position, are slightly different). It is therefore realized that failed segmentations can be detected by looking for such variations. The invention therefore proposes to compare segmentation results for images of the same anatomical region obtained at several different time points, thereby allowing the detection of errors in the segmentation algorithm.

[0021] This approach is perfectly suited to medical imaging, where it is very common to capture sequential images of the same area as part of a consultation, and therefore the accuracy check procedure of the present invention is a natural fit into the normal workflow of medical imaging.

[0022] This approach differs from known approaches in that it is based on comparing the results of segmentation algorithms with each other, rather than only comparing each segmentation result with the original image on which it is based, thus avoiding the problems associated with known accuracy checking procedures.

[0023] Image segmentation may be, for example, the segmentation of two or more images of an anatomical region of a single patient.

[0024] The segmentation procedure may, for example, be an automated segmentation procedure, for example involving one or more segmentation algorithms.

[0025] As mentioned, the large variation over time occurs due to the high sensitivity of typical segmentation errors to very small changes in the input data. Some exemplary input data changes that can lead to large changes in the output segmentation results include, for example, the motion of the ultrasound transducer unit (e.g., probe) when acquiring the data, changes in the (speckle) noise pattern, or anatomical movement over time.

[0026] The different time points are some minimum time separation from each other, which helps ensure that imaging conditions have changed sufficiently to lead to variations in segmentation outcomes.

[0027] Variations in segmentation results are detected via measuring consistency between segmented anatomical features in the segmentation, which may be based on determining a measure of consistency in one or more (e.g., spatial or geometric) characteristics of the segmented anatomical features, such as size, shape, position, etc.

[0028] The segmentations obtained collectively and compared result in a single consistency measure that represents the overall consistency between two or more segmentations.

[0029] In other embodiments, the consistency measure is derived based on evaluating the consistency between individual segmented (anatomical) features of the segmentation. In some embodiments, multiple consistency measures are derived corresponding to different compared features. In other embodiments, a single consistency measure is derived that represents the overall, e.g. average, consistency of the entire set of segmented anatomical features. In other embodiments, a respective consistency measure is derived for each segmented anatomical feature, i.e. a separate consistency measure is derived for each segmented feature.

[0030] The consistency measures can be defined in various ways: in some embodiments, they are defined in terms of a measure of deviation between one or more segmented features (e.g., between one or more characteristics of the segmented features).

[0031] In practice, segmented features refer to, for example, segmentation contours or boundaries that are intended to correspond to anatomical features of the underlying image.

[0032] Image segmentation is the outcome or result of the application of a segmentation procedure. By segmentation procedure we mean an automated segmentation procedure, for example based on one or more segmentation algorithms. Segmentation can take various forms.

[0033] In certain cases, image segmentation involves a segmentation mesh for application or applied to the image. In an accurately segmented image, the feature contours of the segmentation mesh should match actual anatomical features in the image.

[0034] Alternatively, the segmentation may take the form of a set of boundary contours of various anatomical features. The segmentation may also take the form of a set of data that classifies each pixel in the image with respect to the anatomical structure in which it is located in the image. Thus, the segmentation may generally be represented graphically, or simply by (non-graphical) segmentation data.

[0035] In some cases, the segmentation involves both the segmentation product and the original image, for example superimposed one on top of the other.

[0036] In an embodiment, one segmentation of each of the two or more images is obtained, ie a respective segmentation of each of the two or more images is obtained.

[0037] The accuracy measure indicates how well the segmentation reflects the true anatomy. In an embodiment, it may be a quantitative measure, e.g., indicating a degree of accuracy. It may be a binary measure, e.g., correct or inaccurate. It may be a probabilistic measure, i.e., indicating the likelihood that the segmentation is correct.

[0038] According to an embodiment, the method further comprises generating a data output representative of the measure of accuracy. The method further comprises communicating the data output to an external device, such as a remote computer or server.

[0039] In an embodiment, any of the embodiments of the method are implemented by a computer, processor, or controller.

[0040] As described above, according to one or more embodiments, comparing the image segmentations and / or deriving one or more consistency measures comprises comparing one or more similar segmented anatomical features segmented in each of the image segmentations and deriving one or more consistency measures between the anatomical features in the different image segmentations.

[0041] In some embodiments, the accuracy measure is determined based on applying one or more predetermined minimum thresholds for the consistency measure.

[0042] A measure of accuracy is determined based on one or more derived measures of consistency and reference to a predetermined normal degree of inconsistency between image segmentations for images captured at different times.

[0043] For example, when minimum thresholds are applied as described above, these thresholds represent some expected normal level of variation in the segmented features due to changes in the underlying anatomical structures over time. Thus, the minimum threshold or thresholds represent a predetermined expected (or normal) degree of inconsistency between respective segmented features for images captured at different time points.

[0044] More generally, reference to a normal degree of inconsistency allows the inconsistency measured or derived during segmentation to be calibrated against expected inconsistencies due to anatomical variations.

[0045] The objective is to detect anomalous changes in the segmentation that deviate from expected or reliable variations in the actual anatomical structures, ensuring that variations due to actual anatomical motion are not mistaken for segmentation errors.

[0046] These expected levels of inconsistency may be due to variations or motion of the underlying anatomical structures, or simply due to expected (but tolerable) random variations in the segmentation results.

[0047] Here, the procedure for determining the accuracy measure takes into account, for example, expected (normal) changes in the size, shape, position of one or more features in the underlying anatomical structure, for example over time.

[0048] If thresholds are used, different pre-defined thresholds for the consistency measure or measures are used depending on the particular segmented feature to which the consistency measure pertains, i.e., the thresholds applied vary depending on the anatomical feature in question.

[0049] According to an advantageous set of embodiments, the step of deriving one or more consistency measures and / or the step of deriving the accuracy measure are based on the use of machine learning algorithms.

[0050] In an embodiment, a machine learning algorithm is trained to distinguish between normal and abnormal levels of inconsistency between image segmentations of an anatomical region representing the anatomical region at different time points. The algorithm is configured to compare the image segmentations taking into account the normal expected level of inconsistency and derive one or more measures of consistency. Preferably, the algorithm also determines an overall accuracy measure. In some embodiments, the steps of deriving one or more measures of consistency and accuracy may be performed by the machine learning algorithm as a single integrated procedure.

[0051] Where thresholds are used to derive the consistency measure, one or more predefined thresholds for the consistency measure are embodied by the machine learning algorithm, and thus in some cases the machine learning algorithm may be understood as acting as a set of thresholds through the algorithmic processing it applies.

[0052] One approach to taking into account expected normal variations due to anatomical changes is to include a pre-stored set of minimum thresholds for the consistency measures, as discussed above, where these levels are set above the levels of normal anatomical variation, e.g., one threshold for each feature or each part of a feature.

[0053] However, it is computationally more efficient to apply new developments in the field of machine learning that allow the use of classifier algorithms, which are trained using prior data representing both accurate and inaccurate segmentations of anatomical regions.

[0054] Machine learning algorithms can also take a more holistic approach to assessment by considering segmentations as a whole and comparing them against each other, rather than applying individual thresholds to individual features, which can lead to more accurate results and where certain nuances that arise when considering, for example, interactions over time between segmented anatomical features are more effectively taken into account.

[0055] The machine learning algorithm performs both the comparison step and the accuracy determination step, and both the process of deriving a measure of consistency between segmentations and the process of deriving a measure of accuracy are integrated and embodied in the overall classification procedure performed by the algorithm.

[0056] In an embodiment, the machine learning algorithm uses a neural network.

[0057] More broadly, a classifier algorithm is adapted and used to determine a consistency measure based on comparing multiple segmentations to one another (and then classifying the segmentation outcomes as accurate or inaccurate depending on the comparison). The classifier algorithm employs machine learning or deep learning using training data, as discussed above.

[0058] According to one or more embodiments, the accuracy measure includes a probabilistic accuracy measure, i.e., represents a probability of being correct.

[0059] An embodiment of the invention comprises determining a consistency measure between different segmentations, which means for example a consistency measure between one or more segmented anatomical features in the image segmentation, for example between one or more properties of the different segmented features, which for example include shape, position and / or size properties.

[0060] Thus, according to one or more embodiments, the step of determining one or more consistency measures for the segmented anatomical features comprises comparing the position, shape, and / or size of at least a portion of the contour of each of the anatomical features (e.g., within the image segmentation).

[0061] According to an advantageous set of embodiments, the different points in time correspond to defined points in a periodic or cyclical motion pattern of the imaged anatomical region, for example different points in the cardiac cycle (e.g., end systole and end diastole), or different points in the respiratory cycle.

[0062] This allows for uniformity or consistency in the consistency or accuracy measures derived, since the segmentations being compared represent the same set of anatomical time points, at which the anatomical structures can be expected to have changed by some standard, definable amount.

[0063] In a particular embodiment, the method comprises both the steps of obtaining an image segmentation and obtaining an image on which the segmentation is based.

[0064] According to one or more sets of embodiments, the image analysis method is performed in real-time with the step of obtaining the image segmentation.

[0065] In some embodiments, the method is performed in real time with the acquisition of image data representative of the region, for example the method comprises processing the received image data thereby to obtain two or more image segmentations.

[0066] The method comprises, for example, the steps of receiving or acquiring a sequence of images or segmented images and performing the method in real time with their acquisition. The method comprises the steps of controlling an associated ultrasound transducer unit to acquire one or more images and performing the method in real time with their acquisition. This comprises the step of applying a segmentation algorithm to the acquired images.

[0067] In a particular set of embodiments, the image analysis method is an ultrasound image analysis method, where the method is for use with (i.e. for analysis of) ultrasound images and the images to be analysed are ultrasound images.

[0068] The image segmentation may be obtained in various ways: In some embodiments, the image segmentation is received, for example, from an external ultrasound imaging unit or from an ultrasound transducer unit.

[0069] Obtaining one or more image segmentations includes receiving one or more images and processing the images with an image segmentation procedure.

[0070] Obtaining one or more image segmentations includes receiving ultrasound data, processing the ultrasound data to derive a set of two or more images, the two or more images representing an anatomical region at different respective time points, and processing the derived one or more images with an image segmentation procedure.

[0071] An embodiment according to a further aspect of the present invention is an image analysis device for determining failures in an image segmentation procedure to identify and segment one or more anatomical features in an image, the image analysis device comprising: an acquisition module configured to obtain image segmentations of two or more images of an anatomical region, the two or more images representing the anatomical region at different time points, each segmented using an image segmentation procedure; a comparison module configured to compare the image segmentations and derive one or more measures of consistency between the different image segmentations; an analysis module configured to determine a measure of accuracy of the image segmentation procedure based on one or more consistency measures; The present invention provides an image analysis device having an

[0072] The implementation options and details of each of the above features are understood and interpreted according to the descriptions and explanations provided above with respect to the method aspects of the present invention.

[0073] Any of the examples, options or embodiment features or details described above in relation to the method aspect of the invention may be applied to, combined with or incorporated in the apparatus aspect of the invention, mutatis mutandis.

[0074] An embodiment according to a further aspect of the present invention provides an ultrasound system comprising an ultrasound transducer unit; and an image analysis device according to any embodiment or embodiment outlined above or described below, or as described in any claim of the present application, operably coupled to the ultrasound transducer unit for use in obtaining two or more image segmentations.

[0075] In an embodiment, the image analysis device is configured to control the ultrasound transducer unit to acquire two or more images of the anatomical region for segmentation.

[0076] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0077] For a better understanding of the present invention, and to show more clearly how the same may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief description of the drawings]

[0078] [Figure 1] 1 illustrates an exemplary method in block diagram form according to one or more embodiments. [Diagram 2] 1 illustrates the determination of an exemplary consistency measure between exemplary segmented features in different segmentations. [Diagram 3] 1 illustrates, in block diagram form, an exemplary image analysis device in accordance with one or more embodiments. [Figure 4] 1 illustrates, in block diagram form, an exemplary ultrasound system including an exemplary image analysis device in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0079] The present invention will now be described with reference to the drawings.

[0080] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to denote the same or similar parts.

[0081] The present invention provides an image analysis method and device for detecting failures or errors in an image segmentation procedure. The method comprises the steps of comparing the segmentation outcomes for two or more images, the two or more images representing a particular anatomical region at different respective time points, and identifying the degree of consistency or deviation therebetween. Based on the derived consistency or deviation measure, a measure of accuracy of the segmentation procedure can be determined.

[0082] For example, image segmentation sets that fail to meet some minimum similarity threshold or multiple thresholds (eg, for different segmented features) may be determined to be inaccurate or to have a low level of accuracy.

[0083] Therefore, the invention proposes to identify failures in the segmentation algorithm based on a consistency check between segmentations on different images obtained at different times from the same anatomical region.

[0084] A typical feature of segmentation failure is high sensitivity to small changes in the input image data, especially to background patterns or features that do not represent changes in the actual object being imaged. As a result, different segmentation efforts all fail, but the manifestation of failure in the final segmentation is very different. Common variations in the input data during acquisition that lead to these segmentation variations include, for example, probe motion, changes in the (speckle) noise pattern, etc.

[0085] Natural variations in the data over a sequence of acquired images have a substantial effect on their appearance, as segmentation failures are sensitive to these changes in the input data.

[0086] Given at least two image segmentations, consistency measures can be calculated. These consistency measures can be quantitative, e.g., defined according to a spectrum or scale of consistency. The consistency measures can be binary, e.g., classifying the compared segmentations into normal (corresponding to successful segmentation) and abnormal (corresponding to unsuccessful segmentation). The method is not limited to using only two segmentations, and using a larger number of segmentations increases the accuracy of the results.

[0087] In some embodiments, a corresponding image is obtained for each segmentation and used in addition to the image segmentation itself, and in some embodiments the two are combined or superimposed on one another (e.g., by superimposing the segmentation boundary on each respective image).

[0088] Having an image in addition to a segmentation is an optional additional feature that allows a greater number of features or details to be extracted in the comparison procedure than with segmentation alone. For example, image data allows image intensity to be taken into account (e.g., with respect to different segmented features) in the comparison between segmentations.

[0089] A first, simple example of an image analysis method according to one or more embodiments is illustrated in block diagram form in Figure 1. The method 10 is for determining failures in an image segmentation procedure, which is for identifying and segmenting one or more anatomical features in an image.

[0090] Briefly, the method 10 comprises step 12 of obtaining an image segmentation of two or more images of an anatomical region, the two or more images representing the region at different respective time points, each of which is segmented using a segmentation procedure.

[0091] The method further comprises the step of comparing the image segmentations and deriving a measure of consistency between the different segmentations 14. For example, in some embodiments, one or more similar segmented features segmented in each of the image segmentations are compared and a measure of consistency between the features in the different segmentations is derived.

[0092] The method further comprises a step 16 of determining a measure of accuracy of the segmentation procedure based on the one or more consistency measures.

[0093] According to one particular embodiment, when imaging a moving body (e.g., an object such as the heart or lungs that fluctuates cyclically), the step of deriving a consistency measure takes into account the degree of expected normal variation in the segmentation due to underlying changes in the anatomical structure.

[0094] In particular, abnormal variations can be detected if normal or reliable changes in the segmented anatomical structure, such as typical probe or cardiac motion, can be described and encoded. As an example, with respect to cardiac segmentation, knowledge of the particular point in the cardiac cycle at which an image was taken allows a great deal of information to be inferred about the current state of the anatomical structure. The relative change in the anatomical structure between two known points can also be inferred. For example, in the context of cardiac imaging, it is common to capture ultrasound images representing the heart at the end diastole (ED) and end systole (ES) points of the cardiac cycle.

[0095] The method comprises the step 12 of obtaining a segmentation of at least two images (i.e. a plurality of images) representing the anatomical region at different respective times. The two images are either chronologically captured images during a single imaging session or are captured in different imaging sessions. Preferably, the images are recorded in the same coordinate system.

[0096] The step 12 of acquiring one or more image segmentations comprises receiving pre-formed or pre-calculated segmentations from an external source, such as an external ultrasound imaging unit, or from an ultrasound transducer unit, or for example from a memory or data storage device (when, for example, the method is used for post-processing and the images and segmentations have already been derived).

[0097] For example, in some embodiments, imaging data from multiple previous visits for a given patient is accessed or acquired from a data store, a current or recent set of acquired image data is compared to the stored historical data for the previous visits, and one or more consistency measures are derived for the segmentations of images from the previous visits.

[0098] In these embodiments, it may be advantageous to store and utilize additional image intensity information (e.g., based on storing the image data and the derived segmentation) because comparisons outside of the same time series (i.e., outside of the same single imaging session) may show variations due to varying transducer positions or different patient health conditions (i.e., variations unrelated to the accuracy of the segmentation).

[0099] In another embodiment, obtaining 12 one or more image segmentations comprises receiving one or more images and processing the images by a segmentation procedure.

[0100] For example, in an embodiment, the method comprises receiving a sequence of input images representing a given anatomical region at different time points, the method comprises processing the images by a segmentation procedure, for example achieved using a dedicated segmentation module or in other ways.

[0101] The segmentation procedure provides as output a segmentation of the anatomical structure of interest for each image.

[0102] Image segmentation is preferably anatomical model-based segmentation, which uses a model of the anatomical region being imaged to aid in identifying important anatomical features or objects.

[0103] Model-based image segmentation in the field of medical imaging is a well-known procedure, and those skilled in the art will be aware of various exemplary segmentation algorithms that may be used to achieve image segmentation.

[0104] As an example, a suitable exemplary segmentation algorithm for use in the embodiments is described in detail in the document "Automatic Model-Based Segmentation of the Heart in CT Images" by Ecabert, O. et al., Medical Imaging, IEEE Transactions on, 2008, No. 27, pp. 1189-1201.

[0105] Different segmentation algorithms are used to segment the images of different anatomical regions, such as the heart, lungs, gastrointestinal tract, neck, groin or any other region.

[0106] According to certain embodiments, in some cases a reduced or simplified version of the segmentation procedure is used, which can be useful, for example, in time-sensitive applications (e.g., when the method is performed in real-time with image acquisition during a consultation) where there is not enough time to apply the full model to all of the received image sequence (e.g., to both end-diastolic and end-systolic image frames when the sequence consists of end-diastolic and end-systolic time points of the cardiac cycle) and where constraints are imposed on processing power.

[0107] A simplified or reduced version of the segmentation procedure is a faster auxiliary model that is reduced in certain respects. This auxiliary model produces lower resolution results, but the distinctions derived between different segments (e.g., the geometric structure and location of the derived borders) remain unchanged compared to the full model. Thus, the auxiliary model still shares the underlying image feature recognition properties with the full segmentation model, ensuring that failures occur in the same way as the full model.

[0108] As an example, considering a model-based segmentation (such as that outlined in the exemplary document above), a reduced version of the model is obtained by reducing the resolution of the triangles of the mesh representation of the model, reducing the number of iterations in the fitting process, reducing the analysis of more detailed anatomical structures such as pulmonary veins, or any other suitable fitting, for example involving non-essential features of the segmentation.

[0109] In deep learning based segmentation techniques, a reduced deep learning model is achieved, for example, by reducing the network depth, reducing the input image size (e.g., by downsampling), or any other suitable modification.

[0110] In certain cases, the image segmentation product includes a segmentation mesh that describes or represents anatomical information contained in the input image. In some cases, the mesh is applied to the image. In an accurately segmented image, the feature contours of the segmentation mesh should match the actual anatomical features in the image.

[0111] Alternatively, the segmentation may take the form of a set of boundary contours of various anatomical features. The segmentation may also take the form of a set of data that classifies each pixel in the image with respect to the anatomical structure in which it is located in the image. Thus, the segmentation may generally be represented graphically, or simply by (non-graphical) segmentation data.

[0112] In some cases, the segmentation involves both the segmentation product and the original image, for example superimposed one on top of the other.

[0113] The output segmentation includes identification of various anatomical features, including anatomical objects such as organs, chambers, valves, blood or air passages, or any portion or section of such objects, such as the apex of the left ventricle.

[0114] Segmentation identifies, for example, the contours or boundaries of such features, or, for example, the areas over which they extend.

[0115] The segmentation output, e.g., meshes, for each of two or more images (e.g., in the case of cardiac imaging, from the ED and ES cardiac phases) provides indications of characteristic segmented anatomical features for each segmentation. Changes in these features between the different image segmentations are detected and based thereon a measure of consistency between the respective features or between the overall segmentations is derived.

[0116] By way of example, the consistency measure between segmentations may correspond to a measure of consistency between one or more properties of one or more different segmented features, for example these may include shape, position and / or size properties.

[0117] Thus, the step of determining a measure of consistency comprises a comparison of the position, shape and / or size of at least a portion of the contour of each of the set of one or more segmented anatomical features.

[0118] Examples of characteristics of segmented anatomical features that may be compared to derive one or more consistency measures include, for example, the overall height of the left lateral heart, the height and width of the left ventricle, the motion of the upper tip of the left atrium, the apex, the right ventricular lateral wall and the interventricular septum during the ED and ES phases, etc. These merely represent illustrative examples, and there are many other potential characteristics whose variations are used as a basis for deriving a consistency measure during a segmentation or between segmented features of a segmentation.

[0119] FIG. 2 shows the changes in some example anatomical feature properties that are used to determine a measure of consistency between anatomical features in different segmentations.

[0120] FIG. 2(a) illustrates an ultrasound image of a subject's heart. For illustrative purposes, two segmentations are shown superimposed and correspond to images of the heart at different times. The white arrows indicate the height and width of the left ventricle. The changes in height and / or width are used to determine a consistency measure. An additional characteristic whose variations are detected is, for example, the overall height of the heart region.

[0121] Figure 2(b) shows the location of the upper tip of the left atrium (representing the top of the heart) and the location of the cardiac apex (representing the bottom of the heart) as another feature that can be used to determine a consistency measure. The white arrows indicate diagrammatically the movement of these points that occurs during different segmentations for images corresponding to different time points.

[0122] Figure 2(c) shows an example of the motion of the right ventricular lateral wall. Figure 2(c) illustrates the view of the heart across a plane perpendicular to the level of the heart. The black arrows indicate the exemplary motions in a schematic manner.

[0123] In some embodiments, the derived changes in one or more of the position, size or shape characteristics of the segmented features are directly used as a consistency measure during segmentation, e.g., they are fed into an algorithm (e.g., a classifier algorithm) to determine a measure of accuracy of the segmentation procedure.

[0124] In other embodiments, the derived changes in one or more of the position, size or shape properties of the segmented features are further processed to derive one or more consistency measures, for example by taking an average of one or more of the property changes or by combining the property changes in any other manner.

[0125] Using the derived consistency measure or measures, a measure of segmentation accuracy is determined (16).

[0126] According to one or more embodiments, this involves applying a set of minimum consistency thresholds to the consistency measures, which represent the maximum expected levels of normal background variation.

[0127] In the case of imaging moving objects (eg, those that vary cyclically), these thresholds take into account the expected degree of normal variation in segmentation characteristics due to underlying changes in the anatomy.

[0128] Thus, in an embodiment, the one or more minimum thresholds represent a predetermined normal degree of inconsistency between segmentations relating to images captured at different times, e.g., between respective segmented features (or different characteristics of these features).

[0129] According to an advantageous set of embodiments, a classification algorithm is used to determine the accuracy measure, which is advantageously a machine learning or deep learning algorithm, for example using neural networks.

[0130] A machine learning algorithm is trained to distinguish between normal and abnormal levels of inconsistency between image segmentations of an anatomical region representing the anatomical region at different time points.

[0131] Based on the derived set of one or more consistency measures, the classification algorithm determines an accuracy measure, for example, to distinguish between accurate and inaccurate segmentations.

[0132] The accuracy measure may be quantitative, e.g., a value that follows a spectrum or scale of accuracy. The accuracy measure may be binary, e.g., accurate or inaccurate. According to one or more embodiments, the accuracy measure may be a probabilistic accuracy measure.

[0133] As mentioned, the classification algorithm is a machine learning algorithm. One exemplary type of machine learning algorithm used is a support vector machine. Details for the implementation of such algorithms are outlined, for example, in the document "Support-vector networks" by C. Cortes, V. Vapnik, Machine Learning, No. 20, pp. 273-297, (1995).

[0134] Another exemplary type of machine learning algorithm used is an algorithm based on neural networks. Further details on the implementation of such algorithms are outlined, for example, in the document "Neural Networks for Pattern Recognition" by Christopher M. Bishop, Oxford University Press, Inc., New York, NY, USA, 1995.

[0135] According to an embodiment, when a machine learning classifier algorithm is used to determine the accuracy measure, the algorithm is first trained using the sample consistency measure extracted from a set of reference data. In a particular embodiment, this training is included in the method as an optional initial step. The training step is performed only once, and then the remaining steps 12, 14, 16 are performed as many times as necessary for different sets of image segmentations.

[0136] By way of example, the algorithm is trained using example segmentations produced by both accurate and inaccurate segmentation procedures, so that the algorithm models typical patterns for subsequent discrimination between successful and unsuccessful segmentations.

[0137] For example, the algorithm learns to distinguish between normal and abnormal levels and types of variation during a segmentation, e.g., between segmented features in a segmentation.

[0138] Next, an exemplary procedure for training a machine learning classifier algorithm to determine a measure of accuracy of the segmentation procedure is described.

[0139] A set of training data is prepared or provided, and each example in the training data is labeled (eg, by an expert human) as representing a failed or successful segmentation (correct or inaccurate segmentation procedure).

[0140] During training of the classifier algorithm, the classifier algorithm parameters are optimized with respect to the training data so that it successfully recognizes and distinguishes between normal variations (i.e., those that represent accurate segmentation results) and unusual variations (i.e., those that represent inaccurate segmentation results).

[0141] In some embodiments, the algorithm is trained to provide results as a two-class output, meaning that it can identify correct and incorrect segmentation results separately. In this embodiment, the algorithm is trained with examples of both correct and incorrect (i.e., successful and failed) segmentation results. When making an accuracy determination, the algorithm looks for features in the segmentation that it has learned to be indicative of correct segmentation, and features that it has learned to be indicative of incorrect segmentation.

[0142] Alternatively, the algorithm can be trained in terms of a one-class problem, where the algorithm is trained simply to detect anomalies or deviations from some normal population of sample data. Thus, here, the algorithm is trained using only segmentation examples generated by the correct segmentation procedure. The algorithm then detects whether the new segmentation is inside or outside the modeled population of learned correct examples.

[0143] As mentioned above, one exemplary machine learning classifier algorithm that may be used is a support vector machine. Such an algorithm learns weights for a kernel evaluation (similarity) between test and training samples.

[0144] Another example is a neural network, which receives inputs / features of the inputs and calculates a weighted sum of these, which is then passed through a function, which is typically a non-linear function. This processing is performed at each of the neural network's multiple layers, the number of layers being typically referred to as the depth of the network.

[0145] A simple illustrative example is logistic regression, where the output of the algorithm is probabilistic, outputting a value between 0 and 1 indicating the likelihood that the segmentation procedure is accurate.

[0146] In an embodiment, the training procedure also includes providing the underlying image data to the algorithm as part of the training data.

[0147] According to one or more embodiments, the method comprises the further step of generating a data output representative of a derived measure of accuracy of the segmentation procedure.

[0148] In a particular embodiment, the method comprises the step of generating an output for providing to an associated user interface for communicating the derived accuracy measure to the user interface, e.g. a control output for controlling the user interface to generate a sensory output representative of the derived accuracy measure, e.g. for controlling the display of the user interface to display said measure.

[0149] Thus, the user interface is configured to inform the user of problems with the segmentation algorithm and determine recommendations regarding changes in the image acquisition procedure.

[0150] For example, the interface may display information indicating characteristics or aspects of the input data that lead to poor quality segmentation results and provide suggestions regarding changes in data acquisition methods to improve segmentation errors, hi some embodiments, the system is adapted to automatically switch to a semi-automatic mode of data acquisition to avoid consecutive data acquisition errors.

[0151] According to a particular embodiment, the method applies a filtering process, whereby only segmentations having a rating that leads to an acceptable accuracy measure are displayed or used.

[0152] According to a further aspect of the present invention there is provided an image analysis device configured to carry out any example or embodiment of the above described method outlined in this disclosure or any claim.

[0153] Thus, the embodiment provides an image analysis device for determining failures in an image segmentation procedure.An exemplary image analysis device 30 according to one or more embodiments is illustrated in block diagram form in FIG.

[0154] The device comprises an acquisition module 32 configured to acquire image segmentations of two or more images of an anatomical region, the two or more images representing the region at different time points, each of which is segmented using a segmentation procedure.

[0155] The device further comprises a comparison module 34 configured to compare the image segmentations and to derive a measure of consistency between the different segmentations.

[0156] The device further comprises an analysis module 36 configured to determine a measure of accuracy of the segmentation procedure based on the one or more consistency measures.

[0157] The implementation options and details of each of the above device features are understood and interpreted in accordance with the descriptions and explanations provided above with respect to the method aspects of the present invention.

[0158] Any of the features or details of the examples, options or embodiments described above with respect to the method may be applied to, combined with or incorporated in the device aspect of the present invention, mutatis mutandis.

[0159] It is noted that, in the above examples, the acquisition module 32, the comparison module 34, and the analysis module 36 are illustrated as separate components within the image analysis device 30, but this is not required. Their associated functions may be distributed in different ways among one or more components. For example, in some examples, the functions of the different modules may be integrated and performed by a single element, e.g., by a single controller or processor, or the functions may be otherwise distributed among one or more elements or components. Alternatively, the functions of each module may be performed by a separate controller or processor within the image analysis device.

[0160] According to one or more embodiments, the device further comprises a user interface for communicating the derived accuracy measure to a user, for example by generating a sensory output representative of the accuracy measure. The user interface comprises a display for generating a visual output representative of the derived accuracy measure.

[0161] The functionality of one or more of the above modules is realized by a controller or processor. A separate controller or processor may be provided to realize the functionality of each respective module, or the functionality of all the modules may be performed by a single controller or processor.

[0162] The controller may be implemented in many ways by software and / or hardware to perform the various functions required. A processor is one example of a controller, which performs the functions required using one or more microprocessors programmed using software (e.g., microcode). However, the controller may be implemented with or without a processor, or as a combination of dedicated hardware to perform some functions and processors (e.g., one or more programmed microprocessors and associated circuitry) to perform other functions.

[0163] Examples of controller components used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0164] In various embodiments, the processor or controller is associated with one or more storage media, such as volatile or non-volatile computer memory, such as RAM, PROM, EPROM, and EEPROM. The storage media is encoded with one or more programs that, when executed on the one or more processors and / or controllers, perform the required functions. The various storage media may be fixed within the processor or controller or may be portable, such that the one or more programs stored in the storage media can be loaded into the processor or controller.

[0165] A further aspect of the invention provides an ultrasound system. An exemplary ultrasound system is illustrated in block diagram form in Figure 4. The system 40 comprises an ultrasound transducer unit 38; and an image analysis device 30 according to any example or embodiment outlined above or described below, or as described in any claim of the present application, operably coupled to the ultrasound transducer unit for use in obtaining two or more image segmentations.

[0166] The ultrasound transducer unit 38 provides ultrasound image data to the image analysis device 30, which is adapted to process this data by means of a segmentation algorithm in order to derive two or more segmentations.

[0167] The image analysis device 30 is adapted to control the ultrasound transducer unit 38 to acquire ultrasound image data.

[0168] The ultrasonic transducer unit 38 may take any suitable form, for example an ultrasonic probe or an ultrasonic sensor having a flat form factor. The transducer unit comprises one or more ultrasonic transducers, for example an ultrasonic transducer array.

[0169] Although embodiments of the present invention apply in any context where segmentation is applied to medical images, a particularly useful application is ultrasound imaging, for example ultrasound imaging of the cardiac area.

[0170] The systems, methods or devices of the present invention are configured for real-time assessment of segmentation accuracy, advantageously implemented as part of an ultrasound unit, such as an ultrasound workstation, providing image data acquisition and analysis results in real-time.

[0171] As mentioned, although several embodiments have been described above with reference to the application of the invention to imaging of the cardiac region, the invention is not limited to use for cardiac imaging and may be used for segmentation of images of any anatomical region.

[0172] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the words "comprises, includes, has" do not exclude other elements or steps, and the singular does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. Where computer programs are discussed above, the computer programs may be stored / distributed on a suitable medium, such as an optical storage medium or a solid medium, provided together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless remote communication systems. It is noted that where the term "adapted" is used in the claims and description, the term "adapted" is intended to be equivalent to the term "configured". Any reference signs in the claims should not be interpreted as limiting the scope.

Claims

1. 1. A computer-implemented image analysis method for determining failures in an image segmentation procedure to identify and segment one or more anatomical features in an image, the image analysis method comprising: obtaining an image segmentation of two or more images of an anatomical region of a patient, the two or more images representing the anatomical region at different respective time points, each segmented using the image segmentation procedure; comparing the image segmentations and deriving one or more measures of consistency between the different image segmentations; determining a measure of accuracy of the image segmentation procedure based on the one or more measures of consistency; 20. An image analysis method comprising: determining a measure of accuracy based on one or more of the derived measures of consistency and reference to a predetermined normal degree of inconsistency between the image segmentations for images captured at each of the different times.

2. 2. The image analysis method of claim 1, wherein the step of comparing the image segmentations and / or the step of deriving one or more measures of consistency comprises the steps of comparing one or more similar segmented anatomical features segmented in each of the image segmentations and deriving one or more measures of consistency between the anatomical features in different image segmentations.

3. 3. The image analysis method according to claim 1 or 2, wherein the measure of accuracy is determined based on applying one or more predetermined minimum thresholds for the measure of consistency.

4. 4. A method of image analysis according to claim 1, wherein the step of deriving the one or more measures of consistency and / or the step of deriving the measure of accuracy comprises the use of a machine learning algorithm.

5. 5. The image analysis method of claim 4, wherein the machine learning algorithm is trained to distinguish between normal and abnormal levels of inconsistency between image segmentations of the anatomical region representing the anatomical region at each of the different time points.

6. The image analysis method of claim 1 , wherein the accuracy measure comprises a probabilistic accuracy measure.

7. 7. The image analysis method of claim 2, wherein the step of determining one or more consistency measures comprises a comparison of a position, a shape and / or a size of at least a portion of a contour of each of the segmented anatomical features.

8. 8. The method of claim 1, wherein each of the different time points corresponds to a defined time point in a periodic or cyclical motion pattern of the imaged anatomical region.

9. 9. The image analysis method according to claim 1, wherein the image analysis method is performed in real time with the acquisition of image data of the two or more images.

10. 10. The image analysis method according to claim 1, wherein the image analysis method is an ultrasound image analysis method.

11. 11. The image analysis method of claim 1, wherein the step of obtaining one or more of the image segmentations comprises the steps of receiving one or more images and processing the images by the image segmentation procedure.

12. 11. The image analysis method of claim 1, wherein the step of obtaining one or more of the image segmentations comprises the steps of receiving ultrasound data, processing the ultrasound data to derive a set of two or more images, the two or more images representing the anatomical region at different respective time points, and processing the derived one or more images by the image segmentation procedure.

13. 1. An image analysis device for determining failures in an image segmentation procedure to identify and segment one or more anatomical features in an image, the image analysis device comprising: an acquisition module that obtains an image segmentation of two or more images of an anatomical region of a patient, the two or more images representing the anatomical region at different respective time points, each segmented using the image segmentation procedure; a comparison module for comparing the image segmentations and deriving a measure of consistency between different image segmentations; and an analysis module that determines a measure of accuracy of the image segmentation procedure based on one or more of the measures of consistency, an image analysis device, wherein the measure of accuracy is determined based on one or more of the derived measures of consistency and reference to a predetermined normal degree of inconsistency between the image segmentations for images captured at each of the different times.

14. An ultrasonic transducer unit; an image analysis device according to claim 13 operatively coupled to said ultrasound transducer unit for use in obtaining two or more of said image segmentations; An ultrasound system comprising:

Images