Detecting anatomical abnormalities through segmentation results with and without shape priors

Abstract

A system and related method for image processing. The system comprises an input IN interface for receiving two segmentation maps for an input image. The two segmentation maps I1, I2 are obtained by respective segmenters, a first segmenter SEG1 and a second segmenter SEG2. The first segmenter SEG1 implements a shape prior-based segmentation algorithm. The second segmenter SEG2 implements a segmentation algorithm not based on shape priors, or at least the second segmenter SEG2 considers one or more shape priors with lower weight compared to the first segmenter SEG1. A differentiator DIF is configured to identify differences between the two segmentation maps. The system enables anomaly detection.

Description

[Technical Field]

[0001] The present invention relates to a system for image processing, an imaging system, an image processing method, a method for training a machine learning model, a computer program element, and a computer readable medium. [Background technology]

[0002] Medical images are often used for diagnostic purposes. One such diagnostic purpose is the examination of such images by an observer for anatomical abnormalities. Examples of such anatomical abnormalities include fractures (as opposed to healthy, unbroken bones) and neoplastic tissue (as opposed to undiseased organs).

[0003] However, inspection of potentially hundreds of image slices in, for example, a 3D image volume is time-consuming and error-prone because it depends on the observer's skill. This is exacerbated by the fact that abnormalities are reflected in the image as subtle image structures that are not easily discernible even to the trained eye. Furthermore, such inspections are often required under time pressure in stressful environments, such as a busy trauma clinic. Summary of the Invention [Problem to be solved by the invention]

[0004] Therefore, there is a need for improved image segmentation. [Means for solving the problem]

[0005] The object of the present invention is achieved by the subject matter of the independent claims, further embodiments of which are incorporated in the dependent claims. It is noted that the below-described aspects of the invention apply equally to imaging systems, image processing methods, methods for training machine learning models, computer program elements and computer-readable media.

[0006] According to a first aspect of the present invention, at least one input interface for receiving two segmentation maps for an input (medical) image, the two segmentation maps being obtained by respective segmenters, a first segmenter and a second segmenter, the first segmenter implementing a shape-prior based segmentation algorithm and the second segmenter implementing a segmentation algorithm not based on shape priors, or at least the second segmenter considering one or more shape priors with a lower weight compared to the first segmenter; a differentiator configured to identify differences between the two segmentation maps to facilitate detection of anatomical abnormalities; A system for image processing is provided, comprising:

[0007] In an embodiment, the system comprises a visualizer configured to output an indication of said difference.

[0008] In an embodiment, the visualizer is for displaying the indication and i) the input image and / or ii) the first or second segmentation map on a display device (DD).

[0009] In embodiments, the indicia are coded to represent the magnitude of the difference. Visualization of the difference, while preferred herein, is not necessarily required in all embodiments. For example, it may be sufficient to cause a message to be sent via a communication system, to generate a sound, to activate a lamp, or to generate any other alarm signal when such an abnormality is detected. The difference is thresholded to conclude that such an abnormality is present, as is the medical input image related to the patient.

[0010] In an embodiment, the second segmenter is based on a machine learning model and related algorithms.

[0011] In an embodiment, the machine learning algorithm is based on an artificial neural network, although other machine learning models capable of classification / segmentation are also contemplated herein.

[0012] In an embodiment, the segmentations in the two segmentation maps represent any one or more of: i) bone tissue; ii) cancer tissue.

[0013] The system can be used to detect, for example, bone fractures, due to the difference in how the two segmentation algorithms operate. Any identified differences are therefore indicative of a bone fracture.

[0014] What is proposed herein is to detect anatomical abnormalities by applying two different segmentation algorithms, one with an explicitly coded shape prior and the other without, and comparing the segmentation results provided by both algorithms. For example, considering the detection of spinal fractures in CT (computed tomography), in an embodiment, a model-based segmentation (MBS) technique is applied. MBS algorithms are a class of algorithms that incorporate a shape prior. For example, a shape prior is used to segment vertebrae visible in the field of view and for different vertebrae. Furthermore, a deep learning (DL)-based segmentation technique is used, which purely (or at least mostly) considers image intensity and does not code a shape prior. Here, we utilize different operating modes of two different segmentation algorithms, with and without a shape prior. By comparing the results, any fractures can be easily detected. Shape prior-based algorithms, such as MBS-type segmentation techniques, tend to favor results that preserve the topological connectivity of the shape prior, but there is no such bias in non-shape prior-based approaches, such as in ML algorithms more generally. While ML approaches are preferred for the second segmenter, this is not a requirement for all embodiments. In some embodiments, such a shape prior is still used in the second segmentation algorithm, but it should be configured so that less weight is given to the shape prior relative to in-image information.

[0015] Shape prior-based algorithms, such as MBS-type algorithms or other algorithms contemplated herein, use a set of virtual geometric models, such as mesh models. The shape priors are deformed during the segmentation operation to calculate the segmentation result. The shape priors represent shape prototypes. They model the shape of, for example, an organ or anatomical structure based on references (generated before segmentation). The shape priors are topologically connected. They are implemented as mesh models, kept in a library, and accessed by the first segmenter that uses the shape priors. The shape priors encode shape prior knowledge, such as clinical knowledge of anatomical structures and their shapes, and the deformations that such anatomical structures are capable of. Such mesh models are not used to calculate the segmentation result, for example, in most ML (machine learning)-based segmentation algorithms. The segmentation result is a segmented image. The segmented image includes the original image and a segmentation map that classifies image elements. In an embodiment, the segmentation result includes the segmentation map without the original image. The segmentation can be binary or multi-label.

[0016] In an embodiment, the input image is one of: i) an X-ray image, ii) an emission image, iii) a magnetic resonance MR image. Any other imaging modality may also be used, such as ultrasound or emission / nuclear imaging.

[0017] In another aspect, there is provided an imaging system comprising an imaging device and a system according to any of the above embodiments.

[0018] In another aspect, further comprising: receiving two segmentation maps for the input image, the two segmentation maps being obtained by respective segmenters, a first segmenter and a second segmenter, the first segmenter implementing a shape prior based segmentation algorithm and the second segmenter implementing a segmentation algorithm not based on shape priors, or at least the second segmenter considering one or more shape priors with a lower weight compared to the first segmenter; Checking the differences between the two segmentation maps; An image processing method is provided, including:

[0019] In another aspect, a method for training a machine learning model is provided.

[0020] In another aspect, there is provided a computer program element comprising instructions adapted, when executed by at least one processing unit, to cause the processing unit to perform the method.

[0021] In another aspect, at least one computer-readable medium having stored thereon a program element or having stored thereon a machine learning model is further provided.

[0022] definition "User" refers to a person, such as a medical professional, who operates an imaging device or oversees an imaging procedure. In other words, the user is generally not the patient.

[0023] "Imaging," as used herein, refers to an inanimate "object" or image of a human or animal patient, or anatomical part thereof, or of biological significance, such as a microscopic image of a microorganism, virus, etc. Inanimate objects include items of baggage in security screening, or products in non-destructive testing. However, the proposed system is described herein primarily in the medical field, and therefore references herein are primarily to a "patient," or a part of a patient, such as an anatomical structure or organ, to be imaged, or a group of the patient's anatomical structures or organs.

[0024] In general, "machine learning" includes computerized configurations that implement machine learning ("ML") algorithms. Machine learning algorithms are configured to learn from training data to perform a task, such as classification. Some machine learning algorithms are model-based. Model-based ML algorithms operate by adjusting the parameters of a machine learning model. This adjustment procedure is called "training." The model is thus configured to perform a task by training. ML algorithms also include instance-based learning. Task performance by an ML algorithm measurably improves as more (newer) training data is used during training. Performance is measured by objective testing when test data is provided to the system. Performance is defined in terms of a certain error rate that should be achieved for given test data. See, e.g., T.M. Mitchell, "Machine Learning," p. 2, Section 1.1, McGraw-Hill, 1997.

[0025] Illustrative embodiments of the present invention will now be described with reference to the following drawings, which are not to scale unless otherwise noted: [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 illustrates an imaging system including a segmentation function. [Figure 2]FIG. 10 is a diagram showing a segmentation result. [Figure 3] FIG. 1 shows a flowchart of a method for image processing. [Figure 4] FIG. 1 illustrates a block diagram of a machine learning model. [Figure 5] FIG. 1 illustrates a block diagram of a computer-implemented training system for training a machine learning model. [Figure 6] FIG. 1 illustrates a flowchart of a method for training a machine learning model based on training data. DETAILED DESCRIPTION OF THE INVENTION

[0027] Referring to FIG. 1, a computer-aided imaging system AR is shown.

[0028] The arrangement AR comprises in an embodiment an imaging device IA for providing an image and an image processing system IPS for processing said image.

[0029] Very briefly, the image enhancement processor IPS includes segmentation functionality to facilitate medical diagnosis of medical abnormalities. Image-based detection of bone fractures is one application contemplated herein. Other medical applications are also contemplated herein, as well as applications outside the medical field, such as non-destructive materials testing. In particular, as discussed more fully below, the system uses two-channel segmentation to enhance the detection of even fine structural details, such as hairline fractures due to stress or osteoporosis, and early cancers, such as breast cancer. Other applications of the proposed image processing system for segmentation include organ contouring in radiation therapy planning.

[0030] An imaging device IA (sometimes referred to herein simply as an "imager") is herein preferably envisioned for medical purposes and is operable to collect one or more images of a patient PAT or other object of interest. The imaging device may be fixed, such as set up in an examination room at a medical facility, or mobile / portable.

[0031] In general, the imaging device comprises a signal source SS for generating an interrogation signal XB. The interrogation signal XB interacts with tissue in the patient PAT and is thereby modified (i.e., attenuated, redirected, or otherwise produces a measurable result). The modified signal is then detected by a signal detection unit DT. The detected signals, such as intensity values, form detector projection raw data or projection images. The projection images are further processed by a reconstruction module RECON to generate a reconstructed image I.

[0032] The imager IA contemplated herein is configured for structural or functional imaging. Various imaging modalities are contemplated herein, including transmission imaging and emission imaging, or other imaging modalities such as ultrasound (US) imaging. For example, in transmission imaging, such as X-ray-based imaging, the signal source SS is an X-ray tube, and the query signal is an X-ray beam XB generated by the tube SS. In this embodiment, the modified X-ray beam is incident on X-ray-sensitive pixels of the detection unit DT. The X-ray-sensitive detector DT registers the incident radiation as a distribution of intensity values. The registered intensity values form a projection image λ. While the X-ray projection images λ may be useful on their own, such as in X-ray radiography, they are then converted by the reconstruction module RECON into cross-sectional images in CT imaging. In particular, the reconstruction module RECON applies a reconstruction algorithm, such as a filtered backprojection algorithm or other algorithm, to the projection images. The cross-sectional images form 2D images in 3D space. In CT, multiple such cross-sectional images are reconstructed from different sets of projection images to obtain a 3D image volume.

[0033] In an MRI imager, the detection unit is formed by a coil capable of picking up radio frequency signals representative of the projection images, from which MRI cross-sectional images are reconstructed by means of MRI reconstruction algorithms.

[0034] The x-ray images in CT or radiography reveal structural details of a patient's anatomy, as does MRI.

[0035] In emission imaging, such as PET or SPECT, a signal source SS resides within the patient's body in the form of a previously administered radioactive tracer substance. Nuclear events caused by the tracer substance are then registered as projection images in a PET / SPECT detector DT configured around the patient. A PET / SPECT reconstruction algorithm is then applied to obtain a reconstructed PET / SPECT image that represents functional details of processes within the patient's body, such as metabolic processes.

[0036] The imaging device IA is controlled by an operator from an operator console CC, who can set some imaging parameters IP or start or stop projection data acquisition.

[0037] Unless otherwise specified, in the following, no distinction is made between reconstructed and projection images; both will simply be referred to as the "input image" or "image" I to be processed by the image enhancement processor system IPS. In other words, the image processing system IPS may be configured to operate on images from the projection domain or on images reconstructed in the imaging domain. Consistent with what has been explained above, the input image I is either 2D, with pixel values at location (i, j), or 3D, with voxel values organized at location (i, j, k). Therefore, the input image I is organized into a 2D or 3D data structure, such as a 2D or 3D matrix. The pixel / voxel values represent quantities of interest, such as attenuation values, phase contrast values, or dark-field signals, in X-ray imaging. X-ray imaging with spectral imaging is also considered. In other modalities, such as MRI, the values represent RF frequency signal intensities or gamma photon number rates, as in emission imaging.

[0038] An input image I produced by device IA is received in the imaging processor IPS at its interface IN, for example by wired or wireless communication means. The image I is either received directly from the imaging device IA or is retrieved from a memory, such as an image repository IRP or a buffer in which the image is originally stored.

[0039] The image processor IPS runs on one or more general-purpose computing units or on one or more dedicated computing units. The (at least one) computing unit PU is communicatively coupled to the imaging device IA or to the imaging repository IRP. In other embodiments, the image processor IP is integrated into the imaging device IA. The imaging processor IP is configured in hardware or software, or a combination of both. A hardware implementation includes a suitably programmed circuit, such as a microprocessor, an FPGA, or other general-purpose circuit. Hard-coded circuits, such as ASICS or on-chip systems, are also contemplated herein. Dedicated processors, such as a GPU (graphical processing unit), are also used.

[0040] Referring now to operation in more detail and with continued reference to FIG. 1, the computer-implemented image processing system IPS includes a segmentation function for segmenting an image I provided by an imaging device IA or retrieved, for example, from a medical image repository IRP.

[0041] Broadly, segmentation is the task of classifying an image into labels according to the pixel or voxel level. Alternatively, the classification may be according to larger image regions as a whole, such as contours or surfaces. The labels represent the semantics of each pixel, voxel, or image region. In this way, a classification operation may identify, for example, bone pixels, i.e., pixels with the label "bone." The pixels may then be considered to represent bone tissue, etc., for other tissue types. Thus, the footprints of organs in an image may be identified by segmentation.

[0042] A segmented image consists of a segmentation map that spatially indicates the respective label for each pixel, voxel, or image region. One or more of these segmentation maps thus indicate to the user the different tissue types in the field of view captured by the image. The labeling is binary (“bone” / “non-bone”), but multi-label segmentation into multiple organs / tissues is also envisioned herein. Segmentation can aid in diagnosis. Medical applications envisioned herein and that can be supported by the segmentation function include inspecting images for medical abnormalities, such as cancerous tissue or trauma. An application particularly envisioned herein is inspecting images for fractures. Some fractures, such as hairline fractures, are difficult to detect without assistance. If such fractures are unrecognized and left untreated, this can lead to future medical complications. The proposed segmentation function is particularly tuned to identify even microstructures in medical images that may be evidence of medical abnormalities, particularly fractures.

[0043] This improved resolution capability arises in part from the two-channel nature of the segmentation function. In particular, the input image I to be segmented is processed in two ways or "channels" by different segmentation algorithms. More specifically, the image processing system includes two segmentation modules SEG1 and SEG2, each implemented by a different respective segmentation algorithm or at least configured differently. Each operates differently on the input image I to calculate a segmented image, specifically a segmentation map associated with the input image I. Each segmentation map is the primary result of some segmentation operation by SEG1 and SEG2. A segmentation map is a function that assigns a value to each element (pixel, voxel, or region) representing an object of interest, such as an anatomical structure, a group of anatomical structures, or a tissue type. Bone is one example herein, but cancer tissue, brain tissue, etc. are also contemplated herein. The values are probability values. As a special case, the segmentation map may be a binary map with "1" / "0" entries. The segmentation map is represented as a data structure, such as a matrix in a spatial registry with the image to be segmented, where each entry is a normalized value, such as a probability estimate. In some embodiments, as envisioned herein, one of the segmentation modules SEG1 is configured to operate based on a model-based segmentation ("MBS") type algorithm. MBS algorithms have been reported elsewhere, such as in "Automatic model-based segmentation of the heart in CT images" by O. Ecabert et al., published in IEEE Trans. Med. Imaging, vol. 27(9), pp. 1189-11201 (2008). MBS type algorithms work by using one or more prototype shape priors that the MBS algorithm attempts to fit to the image to perform or facilitate segmentation.However, MBS-type algorithms are not the only algorithms contemplated herein for the shape prior-based segmenter SEG1. Other segmentation algorithms that incorporate shape priors are also contemplated herein, such as atlas-based segmentation techniques that register shape priors on the image. Some machine learning techniques that incorporate shape prior knowledge in the form of shape priors are also contemplated herein.

[0044] A shape prior is a virtual geometric shape model. One or more shape priors are used. The shape priors are stored in computer memory. The shape priors are implemented as mesh models. The mesh model is composed of interconnected mesh elements, e.g., polygons, that share edges and vertices with neighboring mesh elements. The mesh model can be stored in memory in a suitable data structure, such as a matrix structure or a pointer structure. The shape prior represents an approximation of the shape of the expected structure recorded in the medical image and is therefore informed by prior medical knowledge. The shape prior is an ideal prototype shape, such as a sphere, cylinder, pyramid, or ellipsoid in the 3D case. For processing 2D images, such as in radiography, 2D shape priors are used. However, actual structures recorded in medical images generally do not have such ideal shapes but are understood as deformed or perturbed instances of the shape prior. Alternatively, instead of using shape priors as idealized geometric primitives, the present specification also contemplates, in a preferred embodiment, "statistical" shape priors based on average measurements of a population. Such statistical shape priors are obtained as follows: Images of a reference cohort of patients are collected. The images are annotated, for example, by experts, possibly assisted by semi-automated segmentation tools. The images thus contain, for example, cardiac annotations. A shape prior is generated as the average cardiac shape for the cohort. In this way, a statistical shape prior is obtained, i.e., an average model representing the average geometric appearance, e.g., of a heart, for the reference cohort.

[0045] During operation of the shape prior-based segmenter SEG1, one or more such shape priors are geometrically deformed to adapt to image structures in the input image I, thus performing or at least facilitating segmentation. Image elements (voxels, pixels of a larger image region) that lie inside or on the surface of the fitted shape model S are considered to represent the segmented object of interest (tissue type, organ, or group of organs), while image elements outside the fitted shape model S do not. Thus, the surface-based representation of the shape prior-based segmentation can be converted into a binary segmentation result, such as a binary map. In addition to shape prior deformations, other factors are taken into account by some shape prior algorithms, such as the MBS algorithm, as described in more detail below.

[0046] In sharp contrast, in some embodiments, the segmentation algorithm implemented by the other segmenter module SEG2 does not use such shape priors. In particular, the second segmenter module SEG2 does not operate on deforming such shape priors. Shape prior-based algorithms implemented by SEG1 have a natural tendency to favor their library of shape priors when deforming them to perform segmentation. Thus, there is an inherent bias toward assumed prototypical shapes implemented by shape priors. Only segmentation results that are derivable by deformation from one or more shape priors can be generated as output by the segmenter SEG1. In particular, large deviations from shape priors are suppressed to allow only realistic / expected anatomical shape deformations. For example, deformations that do not lead to topologically connected results are not assumed by the segmenter SEG1.

[0047] Generally, such non-shape prior-based algorithms implemented by the second segmenter SEG2 have no such bias. Preferably, as contemplated herein, the second segmentation model SEG2 is implemented as a machine learning-based segmenter using a machine learning model that is suitably pre-tuned based on previous training data. A non-exclusive example contemplated herein is an artificial neural network-type machine learning model. Other models, such as support vector machines (SVMs), or other machine learning models capable of classification tasks, are also contemplated herein. For example, in the artificial neural networks (NNs) contemplated herein in some embodiments of the second segmenter SEG2, there is no preconceived shape prior model explicitly adapted to the image to achieve segmentation. The neural network model itself is completely generic and therefore does not favor a specific shape model type from the start. Rather, it is the entire initial set of any initialized parameters of the machine learning model, sometimes called weights, as described in more detail below, that are typically adjusted during training based on the training dataset to improve the cost function. The parameter adjustment is iterative and guided by an optimization algorithm, such as a backpropagation algorithm or other gradient-based algorithm, or even other algorithms. It should also be noted that the model-based segmentation algorithm of segmenter SEG1 is not based on such training data.

[0048] To identify some abnormalities such as bone fractures or cancerous tissue in the segmented images, we utilize different operating modes of two segmentation algorithms managed by two modules SEG1, SEG2.

[0049] The basic mode of operation is illustrated in FIG. 2. FIGS. 2A and 2B show the segmentation results of the respective segmentation algorithms SEG1 and SEG2. More specifically, FIGS. 2A and 2B show the segmentation of a fractured vertebra. FIG. 2A shows the segmentation results from a segmentation using the shape prior used by segmenter SEG1, whereas FIG. 2B shows the segmentation without a shape prior by segmenter SEG2. As can be seen, segmenter SEG1 includes anomalies (fractured areas) in the segmentation results from the non-fractured vertebra shape prior used. Segmenter SEG2 does not use such a shape prior and therefore labels all bony components as vertebrae but excludes the soft tissue discernible in the space Δ left between the fractured components. Therefore, the difference between the segmentation results produced by the two segmenters SEG1 and SEG2 allows for the detection of fractured regions.

[0050] Non-MBS algorithms, such as machine learning algorithms, are not biased towards particular shape priors and their variants, and such biases therefore cannot prevent them from segmenting a given structure into topologically disconnected regions, as shown in FIG. 2B.

[0051] Model-based segmentation, such as in Figure 2A, however, does not allow for topologically disconnected segmentation. MBS maintains the integrity of its shape priors due to their inherent bias toward shape. Contrary to Figure 2B, no gaps or discontinuities appear in the MBS-based segmentation of Figure 2A.

[0052] In other words, as shown in FIG. 2B, the non-model-based segmentation / segmenter SEG2 is guided purely by image structures in the processed image, such as intensity values. In contrast, the model-based segmentation algorithm / segmenter SEG1 in FIG. 2A does not consider image values, or does so only by attempting to fit a library of predefined shape priors. More specifically, in model-based segmentation, the enabled deformations operate continuously, thus eliminating topologically truncated results, or at least those with discontinuities or gaps. In an MBS-type algorithm, starting from topologically connected shape priors, the resulting segmentation is deformed but remains topologically connected, as in FIG. 2A. A truncated segmentation with a gap Δ, as in the example of FIG. 2B, is simply not possible using the purely model-based approach used by the segmentation module SEG1. That said, it is not necessarily the case that the second segmentation algorithm SEG2 alternatively does not include a shape prior at all: rather, in this case, the second segmentation algorithm is configured to give higher weight to image structures or image values recorded in the image rather than to its shape prior, as will be explained in more detail below.

[0053] Looking first in more detail at MBS-type algorithms as an embodiment of shape prior-based segmentation, some such techniques use the concepts of external and internal energy. The segmentation is formulated in terms of optimizing an objective function, F, which can be divided into two parts: the internal energy E int One is the external energy E extThe internal / external energies are conceptualized for this purpose from elasticity theory. The deformation D is assumed to incur a cost due to the inherent elasticity of the shape prior. The lower the assumed elasticity of the shape prior, the higher the cost. The elasticity correlates with the material or anatomical properties of the organ / tissue to be segmented, as this is assumed to be known a priori.

[0054] Internal energy term E int represents how the shape prior should be deformed, while the second term E ext represents the processing of image data itself. The internal energy term adjusts how the model can deform the image, but only within realistic limits. The external energy "pushes" the model towards the data in the image. The contribution of each of the two terms to the total cost F can be controlled by several weighting parameters α, β ≥ 0.

number

[0055] In the embodiment, it is assumed that the second segmentation algorithm SEG2 is not necessarily of machine learning type. It can still be of MBS type, and therefore uses the above-described concept of internal / external energy, but with a shape prior deformation term E int The weight β, which is higher than the weight α of the image information term E ext The shape prior-based segmentation algorithm is constructed in a contrasting manner: it uses the image information term E ext A higher weight α is assigned to the shape prior deformation term E compared to the weight β for int In particular, if both segmentation algorithms SEG1 and SEG2 use MBS, the weight α for the internal energy is higher for segmentation algorithm SEG1 than for segmentation algorithm SEG2.

[0056] Preferably, however, SEG2 is not of the form (1) at all, but instead is based on a trained machine learning model M without shape priors, while SEG1 implements an MBS-type algorithm configured to improve the cost function (1). ext β>2, but α>β, in particular α>β, e.g., α / β>2. However, again, the MBS-type algorithm (1) is only one embodiment of a shape prior-based algorithm, and any other shape prior-based algorithm for the segmenter SEG1 is also contemplated, such as atlas-based or other algorithms.

[0057] The operation of the image processing system IPS with two-channel segmentation functionality will now be described in more detail, with continued reference to Figure 1. An input image I is received at an input port IN. The input image I is processed by two segmentation algorithms SEG1, SEG2 to generate respective different segmentation results I1 and I2. The results include, in particular, respective segmentation map(s).

[0058] The two segmentation results I1, I2, natively spatially aligned with one another, are then compared to generate an indication of a medical abnormality, such as an anatomical abnormality like a fracture. More specifically, in embodiments, a differencer DIF is used to perform a point-wise subtraction of the two segmentation maps I2 and I1, or segmented images, to generate a difference image / map. An example of such a difference image / map is described above in FIG. 2C.

[0059] A fixed or user-defined threshold is used to establish which image elements in the difference image represent relevant deviations between the two segmentation results. Image elements whose difference exceeds this threshold are considered to be part of an indicator region that may be formed in this manner. An anomaly locator map may be compiled in this manner, indicating for each image element whether or not there is such a segmentation result deviation. The collection of all image elements that represent such deviations may be referred to herein as the deviation set DS. The deviation set DS may therefore indicate the sought-after anatomical abnormality, such as a fracture.

[0060] The deviation set is processed by the visualizer VIZ into a color or gray value coded overlay widget that can be displayed overlaid with the input image or any one of the two segmentations I1, I2, where the color or gray value varies with magnitude across the deviation set.

[0061] Some or all of the images / maps I1, I2, DS may be displayed superimposed or simultaneously displayed, for example, in different image planes on a display device DD. More specifically, in an optional visualization aid embodiment for computer-aided reading, regions of significant deviation SD between the output segmentations I1, I2 may be displayed using a color- or gray-value-coded overlay of the image. Where applicable, the color / gray-value-coded overlay varies in color or gray value with the magnitude of deviation between the segmentations. For example, this may indicate the difference between the voxel / pixel-level probability response of a neural network (e.g., a convolutional neural network CNN) and the classification results provided by a shape prior-based segmenter SEG1, such as MBS segmentation.

[0062] Two-channel segmentation can therefore facilitate anomaly detection by indicating deviations from healthy organ shape. The proposed IPS system can be used to detect anatomical anomalies based on medical images. For example, in the case of CT or radiography images, the IPS system allows for the detection of fractures resulting from trauma. Instead of or in addition to visualizing the differences between the two maps, an alarm signal or message can be generated, for example, based on thresholding the difference values.

[0063] Reference is now made to Figure 3, which shows a flow chart of a method for image processing. In particular, the steps described relate to the two-channel segmentation described above, although it will be understood that the following steps are not necessarily tied to the architecture described above, but may be understood as teachings in their own right.

[0064] In steps S310a and S310b, an input image I, such as an X-ray image, is processed by two different segmentation algorithms or by at least two differently configured versions of the same segmentation algorithm.

[0065] Preferably, in step S310a, the input image I is processed into a segmented image I1 (shown as I1 in FIG. 3) based on an algorithm that uses a shape prior to compute the segmentation. A model-based segmentation type algorithm is one example envisioned herein in embodiments.

[0066] In step S310b, the input image I is processed by a segmentation algorithm that is not based on any shape prior model to produce a segmentation output I2 (shown as I2 in FIG. 3). In an embodiment, at least, no explicit shape prior is used.

[0067] Preferably, step S310b is performed by a machine learning model without a shape prior pre-trained on training data. Alternatively, as described above with respect to equation (1), the segmentation algorithm in step S310b is also a model-based algorithm, but is configured to weigh the contribution from a reduced shape prior adaptation minus the contribution from the structure in the image.

[0068] The order of steps S310a and S310b is not important.

[0069] In step S320, the two segmentation results I1, I2 are received and passed to step S330, where the two images are compared. In particular, the point-wise difference between the two segmentation results / images I1, I2 is formed to generate a deviation map or difference image Δ.

[0070] In step S340, the difference image is analyzed, for example, by comparing the difference values encoded in the difference image / map Δ with a preset significance threshold.

[0071] Image locations whose difference value exceeds a given threshold are considered to be part of the deviation set. The difference set may be color coded or otherwise processed.

[0072] In step S350, the possibly processed color- or gray-value-encoded deviation set is displayed. This may include displaying the deviation set as an overlay along with one of the segmented input images I1, I2. Additionally or alternatively, the deviation set is displayed in combination with the original input image, such as as an overlay on the original input image I. Other display options are contemplated herein, such as simultaneously displaying the deviation set with one or more of images I1, I2, I in any combination in different panes of a graphics display on the display device DD. In addition to or instead of such a display, the thresholded difference map Δ is used to generate an alarm signal or message to notify the user that a medical abnormality has been detected.

[0073] Reference is now made to Figure 4, which illustrates an embodiment of a machine-based implementation of the second non-shape prior-based segmentation algorithm SEG2. Figure 4 shows a schematic block diagram of a machine learning model M of the artificial neural network type. In particular, the embodiment uses an at least partially convolutional neural network type ("CNN") that includes one or more layers that are non-fully connected layers. The neural network M in Figure 4 is of the feedforward type, although recurrent architectures are not excluded herein.

[0074] The model M is trained by a computerized training system TS, which is described more fully below in FIG. 5. During training, the training system TS adapts an initial set of (model) parameters θ of the model M. In the context of neural network models, the parameters are sometimes referred to herein as network weights (these weights are unrelated to the weights α, β in (1) above). The training data is generated by simulation or obtained from existing historical images or other data, such as those found in a medical image database such as a PACS (Picture Archiving and Communication System) or similar, as described in more detail below with respect to FIG. 5.

[0075] Therefore, two processing phases are defined for a machine learning model NN: a training phase and a deployment (or inference) phase. In the training phase, prior to the deployment phase, the model is trained by adapting the parameters of the model based on training data. Once trained, the model is used in the deployment phase, for example, to segment new images encountered in daily clinical practice. Training can be a one-off operation or can be repeated as new training data becomes available.

[0076] The machine learning model M is stored in one (or more) computer memories MEM'. The pre-trained model M is deployed as a machine learning component running on a computing device PU, such as a desktop computer, workstation, laptop, etc., or multiple such devices in a distributed computing architecture. Preferably, to achieve good throughput, the computing device PU includes one or more processors (CPUs) that support parallel computing, such as processors with a multi-core design. In one embodiment, a GPU (graphical processing unit) is used.

[0077] Referring now more particularly to Figure 4, the network M comprises a plurality of computational nodes arranged in layers in a cascaded fashion, with data flow proceeding from left to right, thus from layer to layer. Recurrent networks are not excluded herein. Convolutional networks have been shown to provide good results when processing image data.

[0078] In unfolding, input data, such as an input image I to be segmented, is applied to an input layer IL, optionally supplemented with contextual non-image data CXD, which describes or relates to the image data I. The contextual non-image data CXD includes biological characteristics of the patient imaged in the image I or the imaging parameters Ip used to acquire the input image. The input data I is then applied to hidden layers L1-L N (only two are shown, but there could be only one or more) and then emerge in the output layer OL as an estimated output M(I). According to the embodiment described above, the output M(x) is the segmented image I', or at least the associated segmentation map.

[0079] Because the model network M has two or more hidden layers, it is sometimes said to have a deep architecture. In feedforward networks, the "depth" is the number of hidden layers between the input layer IL and the output layer OL, while in recurrent networks, the depth is the number of hidden layers times the number of paths.

[0080] The layers of the network, and indeed the input and output images and the inputs and outputs between hidden layers (herein referred to as feature maps), can be represented as two-dimensional or higher-dimensional matrices ("tensors") for efficiency of computation and memory allocation, with the dimensions and number of entries representing the sizes mentioned above.

[0081] Preferably, hidden layers L1-L Ncontains a sequence of convolutional layers using the convolution operator CV. The number of convolutional layers is at least one, such as two to five, or any other number. m The layer and the input layer IL implement one or more convolution operators CV. Each layer L m implement the same number of convolution operators CV, or the number is different for some or all layers. Optionally, zero padding P is used.

[0082] In an embodiment, downstream of the sequence of convolutional layers are one or more fully connected layers FC (only one is shown) to generate the classification result, i.e., the segmented image I′ or segmentation map.

[0083] Each layer L i The activation layer processes the input feature maps from the previous layer into intermediate outputs sometimes called logits. An optional bias term is applied, for example by summation. The activation layer processes the logits in a nonlinear manner into next-generation feature maps, which are then output and passed as input to the next layer, and so on. The activation layer is implemented as a rectified linear unit (RELU) as shown, or as a softmax function, a sigmoid function, a hyperbolic function, or any other suitable nonlinear function. Optionally, there are other function layers, such as pooling layers or dropout layers, to promote more robust learning. Pooling layers reduce the dimensionality of the output, while dropout layers sever connections between nodes from different layers.

[0084] Convolutional layer L 1-NA convolutional layer is distinguished from a fully connected layer (FC) in that the entries in the output feature map of a convolutional layer are not a combination of all nodes received as inputs to that layer. In other words, the convolution kernel is applied to only a subset of the input image to the feature map received from the previous convolutional layer. The subset is different for each entry in the output feature map. The operation of the convolution operator can therefore be conceptualized as "sliding" over the input, similar to the discrete filter kernel in a typical convolution operation known from typical signal processing. Hence the name "convolutional layer." In a fully connected layer (FC), the output node is generally obtained by processing all nodes of the previous layer.

[0085] In an embodiment, the output layer OL is configured as a combiner layer, such as a softmax function layer or similar computational node, where feature maps from previous layers are combined into normalized counts to represent classification probabilities per class. The classification results provided by the network M include, for example, a binary classification into two labels if only one organ type is important. For multi-organ / tissue classification, the output layer OL provides multi-label classification results.

[0086] If, in addition to the image data I, patient biological characteristics or other non-image context data CXD are to be co-processed, the non-image context data CXD suffers from a sparse representation that may be undesirable for efficient processing together with the image data, and so it must first be converted into a suitable representation or encoding. One-hot encoding and processing by an autoencoder network is used to convert the non-image context data CXD into a more suitable denser representation, which is then processed together with the image data by the model M.

[0087] It will be understood that the model M described above in FIG. 4 is merely an example embodiment and does not limit the present disclosure. Other neural network architectures with more, fewer, or different capabilities than those described herein are also contemplated herein. Multi-mode networks, such as GANs (generative adversarial networks), were reported by I. Goodfellow et al. in "Generative Adversarial Networks," arXiv:1406.2661, submitted in June 2014. Furthermore, the model M contemplated herein is not necessarily a neural network type, such as a support vector machine, a decision tree, or a random forest. Other model-based or instance-based ML techniques are also contemplated herein. Model-based approaches can be discriminative, such as the NN in FIG. 4, or generative. Instance-based approaches for segmentation include, for example, k-NN ("nearest neighbor") approaches. In alternative embodiments, typical statistical classification methods based on sampling from training data are also contemplated herein. Still other techniques include Bayesian networks, or random fields such as Markov-type random fields.

[0088] 5 illustrates a training system for training a machine learning model for use as a second segmentation module that is not based on a shape prior. In the exemplary model-based approach described above, such as the NN-type model M, the sum of weights for all convolutional filter kernels, fully connected layers, output layers, etc., defines the configuration of the machine learning model. The weights are different for each layer. In particular, it is these weights that are learned in the training phase. Once the training phase is over, the fully learned weights, along with the node-configured architecture, can be stored in one or more data memories MEM′ and used for deployment.

[0089] Reference is now made to FIG. 5, which shows a training system TS for training parameters in a model-based ML algorithm, such as a neural network type model, or for training a non-neural network type ML model.

[0090] In supervised learning, the training data is the data (x k , y k ), where k can be in the hundreds or even thousands. For each pair k (index k is unrelated to the index used above to specify the generation of the feature map), the training data is k and the associated target y k The training data is therefore organized in pairs k, particularly for the supervised learning schemes primarily envisaged herein, although it should be noted that this specification does not exclude unsupervised learning schemes.

[0091] training input data x k is obtained from historical image data collected in a laboratory or previously in a clinic and maintained in an image repository, for example, a PACS in a HIS (hospital information system). k Or "ground truth" is the (unsegmented) image x k The segmentation map represents a segmented image associated with the segmented image. Such labels may include expert segmentation, annotations, or contouring. Such labels may be retrieved from a PACS database or other medical data repository. Historical images include previously segmented images. Also, in some embodiments, pairs of unsegmented vs. segmented training data may be generated from simulations.

[0092] If the training is to include context data CXD, then in general, the target y kThere is no context data included in the pair. In other words, for learning with context data, the pair is generally ((x k ,c),y k ) and the non-image context data c is the training input x k Only relevant to the target k is not related to.

[0093] In the training phase, the architecture of a machine learning model M, such as the NN network in Figure 4, is pre-populated with an initial set of weights. The weights θ of the model NN are parameterized M θ The purpose of the training system TS is to generate training data (x k , y k ) pair to optimize, and therefore adapt, the parameters θ. In other words, learning can be mathematically formulated as an optimization scheme in which a cost function F is minimized, although a dual formulation that maximizes a utility function is used instead.

[0094] For the moment, assuming the paradigm of a cost function F, this measures the aggregated residual, i.e., the error incurred between the data estimated by the model M and the target from some or all of the training data pairs k.

number

[0095] In equation (2) and below, the function M() is k Figure 1 shows the results of a model M applied to a sparse matrix. For the classifiers primarily considered here, the distance measure ||·|| is preferably formulated as one of the following: cross-entropy or Kullback-Leibler divergence. Any suitable distance measure may be used.

[0096] In training, the training input data x of the training pair kis propagated through the initialized network M. In particular, the training input x for the kth pair k is received at the input IL of the model M, passed through the model, and then output training data M at the output OL. θ (x). The measure |||| is the actual training output M generated by the model M. θ (x k ) and the desired target y k The method is configured to measure the difference, also referred to herein as the residual, between

[0097] The output training data M(x k ) is the applied input training image data x k Target y related to k In general, this output M(x k ) and the associated target y of the kth pair currently under consideration. k In that case, there is an error between the complete training dataset pair (x k , y k An optimization scheme, such as backward / forward propagation or other gradient descent-based method, is used to adapt the parameters θ of the model M so as to reduce the aggregated residual (the sum in (2)) for all or a subset of

[0098] The model parameters θ are the current pair (x k , y k After one or more iterations in the first inner loop, where the updater UP updates the next training data pair x k+1 , y k+1The second, outer loop, is then entered, where the corresponding training data pairs are processed accordingly. The structure of the updater UP depends on the optimization scheme used. For example, the inner loop managed by the updater UP is implemented by one or more forward and backward passes in a forward / backward propagation algorithm. While adapting the parameters, the aggregated, e.g., summed (2), residuals of all training pairs are considered to be reached for the current pair in order to improve the objective function. The aggregated residuals can be formed by constructing the objective function F as a sum-of-squares residual, such as in equation (2), of some or all possible pairwise residuals. Instead of sums of squares, other algebraic combinations are also contemplated. The process flow alternates through the inner and outer loops until the desired minimum number of training data pairs has been processed.

[0099] The training system shown in Fig. 5 may be considered for all model-based learning methods, particularly supervised methods. In alternative embodiments, this specification also contemplates unsupervised learning methods. While Fig. 5 primarily relates to model-based learning, alternative embodiments contemplate instant-based learning methods instead. Also, while Figs. 5 and 6 relate to parameterized models, this specification does not exclude non-parametric ML algorithms. A GPU is used to implement the training system TS.

[0100] The fully trained machine learning module M may be stored in one or more memories MEM′ or databases and made available as a pre-trained machine learning model M for the segmenter SEG2. The trained model is made available in a cloud service; access may be provided free of charge, or their use may be authorized via payment authorization or a pay-per-use scheme.

[0101] FIG. 6 shows a flowchart of a method for training a machine learning module, such as the method described above in FIG.

[0102] Suitable training data must be collated. Preferably, a supervised learning scheme is envisaged herein, although this is not essential as unsupervised or at least semi-unsupervised learning setups are also envisaged herein.

[0103] In supervised learning, the training data includes suitable pairs of data items, each pair including training input data and associated target training output data. The image or projection data may be paired by retrieving the image or projection data from a database or other image repository, as described above.

[0104] Continuing to refer to FIG. 6, in step S610, the training data is divided into pairs (x k , y k ) as defined above in Figure 5. k and the associated target y k , x k Includes.

[0105] In step S620, the training input x is used to generate the training output. k is applied to the initialized machine learning model NN.

[0106] In step S630, the associated target y k Training output M(x k ) is quantified by a cost function F. To improve the cost function, in step S640, one or more parameters of the model are adapted in one or more iterations in the inner loop. For example, the model parameters are adapted to reduce the residual as measured by the cost function. The parameters include, in particular, the weights W of the convolution operator when a convolutional NN model M is used.

[0107] The training method then returns to step S610 in the outer loop, where the next pair of training data is supplied. In step S620, the parameters of the model are adapted to reduce, and in particular minimize, the aggregated residual for all pairs considered. A cost function quantifies the aggregated residual. Forward-backward propagation or a similar gradient-based technique is used in the inner loop.

[0108] More generally, the parameters of the model M are adjusted to improve an objective function F, which may be either a cost function or a utility function. In an embodiment, the cost function is configured to measure an aggregated residual. In an embodiment, the aggregation of residuals is performed by summing all or some of the residuals for all pairs considered. The method is preferably implemented on one or more general-purpose processing units TS, having processors capable of parallel processing, to speed up training.

[0109] While FIG. 6 relates to parameterized model-based training, embodiments herein also contemplate non-parameterized and / or instance-based ML training approaches.

[0110] The components of the image processing system IPS or the training system TS are implemented as one or more software modules running on one or more general-purpose processing units PU, such as workstations associated with the imagers IA, or on a server computer associated with a group of imagers.

[0111] Alternatively or additionally, some or all components of the image processing system IPS or the training system TS are configured in hardware, such as a suitably programmed microcontroller or microprocessor, such as an FPGA (Field Programmable Gate Array), or as a hardwired IC chip, an application specific integrated circuit (ASIC) integrated into the imaging system image processing system IPS. In further embodiments, the image processing system IPS or the training system TS are also implemented partly in software and partly in hardware.

[0112] The different components of the image processing system IPS or the training system TS are implemented on a single data processing unit PU. Alternatively, some or several components are implemented on different processing units PU, possibly arranged remotely in a distributed architecture and connectable over a suitable communication network, such as in a cloud setting or a client-server setup. In particular, the two segmenters SEG1, SEG2 run on the same or different computing systems, while the differencer DIF runs on another system, possibly remote from at least one of the systems on which the segmenters SEG1, SEG2 run. In the case of the process described herein, the segmentation map is transmitted to the differencer DIF by a communication network or retrieved by the differencer DIF from a data storage or the like.

[0113] One or more features described herein may be configured or implemented as or with circuitry encoded in a computer-readable medium, and / or combinations thereof, which may include discrete and / or integrated circuits, systems-on-chips (SOCs), and combinations thereof, machines, computer systems, processors and memories, and computer programs.

[0114] In another exemplary embodiment of the invention, a computer program or a computer program element is provided, characterized in that it is adapted to perform, on a suitable system, the method steps of the method according to one of the previous embodiments.

[0115] The computer program element is therefore stored on a computing unit, which is also part of an embodiment of the present invention. This computing unit is adapted to perform or cause the performance of the method steps described above. Furthermore, it is adapted to operate the apparatus components described above. The computing unit may be adapted to operate automatically and / or to execute user instructions. The computer program is loaded into the working memory of a data processor. The data processor is therefore equipped to perform the method of the present invention.

[0116] This exemplary embodiment of the invention covers both computer programs that use the invention from the beginning and computer programs that, through updates, turn existing programs into programs that use the invention.

[0117] Furthermore, the computer program element may provide all the necessary steps to implement the procedures of the exemplary embodiments of the methods described above.

[0118] According to a further exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is presented, having stored thereon a computer program element, the computer program element being as described in the previous section.

[0119] The computer program may be stored and / or distributed on a suitable medium (particularly, but not necessarily, a non-transitory medium), such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0120] However, the computer program may also be presented over a network such as the World Wide Web and downloaded from such a network into the working memory of a data processor. According to a further exemplary embodiment of the present invention, a medium for making a computer program element available for downloading is provided, the computer program element being configured to perform a method according to one of the previously described embodiments of the present invention.

[0121] It should be noted that embodiments of the present invention have been described with respect to different subject matters. In particular, some embodiments have been described with respect to method-type claims, while other embodiments have been described with respect to device-type claims. However, those skilled in the art will infer from the above and following description that, unless otherwise notified, any combination of features belonging to one type of subject matter, as well as any combination between features relating to different subject matters, is considered to be disclosed in the present application. However, all features can be combined to provide a synergistic effect that exceeds the mere sum of the features.

[0122] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description is to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the dependent claims.

[0123] In the claims, the word "comprises" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single processor or other unit fulfills the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be interpreted as limiting the scope.

Claims

1. at least one input interface for receiving two segmentation maps for an input image, the two segmentation maps being obtained by respective segmenters, a first segmenter and a second segmenter, the first segmenter implementing a shape prior-based segmentation algorithm and the second segmenter implementing a segmentation algorithm not based on shape priors, or the second segmenter considering one or more shape priors with a lower weight compared to the first segmenter; a differencer for identifying differences between the two segmentation maps and detecting anatomical abnormalities from the differences; A system for image processing comprising:

2. The system described in claim 1, wherein the differencer detects the anatomical abnormality based on the difference between the two segmentation maps exceeding a fixed or user-defined threshold.

3. The system of claim 1 or 2, comprising a visualizer that outputs an indication of the differences.

4. The system of claim 3 , wherein the visualizer displays the indication and i) the input image or ii) the first or second segmentation map on a display device.

5. A system as described in claim 3 or 4, wherein the visualizer encodes the indication to represent the magnitude of the difference.

6. The system of claim 1 , wherein the second segmenter is based on a machine learning model.

7. The system of claim 6 , wherein the machine learning model is based on an artificial neural network.

8. The system of claim 1 , wherein the segmentations in the two segmentation maps represent one or more of: i) bone tissue; ii) cancer tissue.

9. The system of claim 8 , wherein the anatomical abnormality is one of a bone fracture, and the difference is indicative of the bone fracture.

10. 10. The system of claim 1, wherein the input image is one of: i) an X-ray image; ii) an emission image; iii) a magnetic resonance image; iv) an ultrasound image.

11. An imaging system comprising an imaging device and a system according to any one of claims 1 to 10.

12. receiving two segmentation maps for an input image, the two segmentation maps being obtained by respective segmenters, a first segmenter and a second segmenter, the first segmenter implementing a shape prior-based segmentation algorithm and the second segmenter implementing a segmentation algorithm not based on shape priors, or the second segmenter considering one or more shape priors with lower weight compared to the first segmenter; determining differences between the two segmentation maps; detecting anatomical abnormalities from the differences; 1. A computer-implemented method of image processing, comprising:

13. A computer program comprising instructions which, when executed by at least one processing unit, cause said processing unit to perform the method of claim 12.

14. At least one computer readable medium having stored thereon a computer program according to claim 13.

Images