Medical Image Segmentation and Atlas Image Selection

Abstract

Some embodiments relate to segmenting medical images. For example, a medical image is registered to multiple atlas images, after which a segmentation function is applied. The multiple segments are fused into a final overall segment. Atlas images are selected based on high segmentation quality or low registration quality.

Description

[Technical field]

[0001] The subject matter disclosed herein relates to a method for medical image segmentation, a method for selecting an atlas image for use in a medical image segmentation method, a medical image segmentation system, an atlas image selection system, and a computer readable medium. [Background technology]

[0002] Accurate and robust medical image segmentation remains a challenging task.

[0003] One approach to medical image segmentation is multi-atlas-based image segmentation methods. These methods can be constructed using a limited set of segmented reference images, known as atlases. Unfortunately, atlas-based methods often lack the desired accuracy. Examples of registration methods and their application to medical image segmentation are described in "Deformable medical image registration: a survey" by A. Sotiras et al., or "Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain" by BBA Vants et al., both papers are incorporated herein by reference.

[0004] Another approach to medical image segmentation is neural networks, such as convolutional neural networks. Examples of neural networks suitable for medical image segmentation are described in the articles "Comparison of deep learning-based techniques for organ segmentation in abdominal CT images" by Groza et al., or "Foveal fully convolutional nets for multi-organ segmentation" by T. Brosch and A. Saalbach, both of which are incorporated herein by reference. Summary of the Invention [Problem to be solved by the invention]

[0005] Neural network-based approaches allow for the efficient construction of accurate segmentation algorithms. Unfortunately, neural networks do not generalize well to new data sets with slightly different characteristics. When a trained neural network is presented with new images that deviate even slightly from the images on which it was trained, the resulting segmentation will not be completely correct.

[0006] There is a need for a medical image segmentation method that on the one hand allows for accurate segmentation, but on the other hand generalizes better to unfamiliar images. [Means for solving the problem]

[0007] To address these and other challenges, a medical segmentation method is provided that uses both a segmentation function and a plurality of atlas images. Additionally, a method is provided for selecting atlas images for use in the medical image segmentation method. Additionally, corresponding devices and software are provided.

[0008] A machine learning based segmentation algorithm is obtained that produces accurate and robust segments of medical images while also generalizing well to new data sets with slightly different characteristics. The improved generalization is used to reduce the amount of training data.

[0009] By aligning an image to an atlas image with which the segmentation function is familiar, e.g., where the segmentation function performs well, accurate segmentation is more likely to be achieved. For example, it is avoided to apply the segmentation function to images for which it produces failure mode segments indicating a breakdown in segmentation ability. By aligning an image to multiple atlas images, the probability that a large proportion of them will show poor segmentation output is reduced, improving the accuracy of the process. For example, it is avoided that accidental anomalies in the atlas or input images dominate the results. Combining a set of multiple predictors with an appropriate voting scheme improves their predictive performance.

[0010] An advantage of the atlas image selection embodiment is that the atlas image selection is performed after the training of the segmentation function is completed. In fact, the training of the segmentation function does not depend on the atlas image selection. This allows the embodiment to enhance an existing segmentation function.

[0011] For example, in one embodiment, the segmentation function is a machine learning segmentation function. For example, training images are used to train the segmentation function. For example, the segmentation function is, for example, a neural network, for example, a convolutional neural network (CNN). Alternatively to a neural network, the machine learning segmentation function is a decision forest. The machine learning segmentation function is an ensemble of neural networks. For example, the final result is assembled from the results of the individual network responses in the ensemble. For example, the ensemble of neural networks is trained using different parameters.

[0012] A subset of the n training images is selected as atlas images. To segment a new image, the new image is aligned with the selected n atlas images, and a segmentation function is applied to the aligned images to obtain n segments, e.g., pixel or voxel segments. After applying the corresponding inverse transform to each of the n segments, the resulting segments are fused, e.g., using a majority vote, to obtain the final segment.

[0013] The methods for segmentation and / or the methods for selecting atlas images are implemented in an electronic device, e.g., a computer. The segmentation methods described herein are applied to a wide range of practical applications, e.g., clinical workstations or web / cloud based clinical applications for diagnosis, quantification, or treatment planning.

[0014] Those skilled in the art will appreciate that the method applies to multidimensional image data, e.g., two-dimensional (2D), three-dimensional (3D), or four-dimensional (4D) images, acquired by a variety of acquisition modalities, such as, but not limited to, standard X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), positron emission tomography (PET), single photon emission computed tomography (SPECT), and nuclear medicine (NM).

[0015] An embodiment of the method may be implemented in a computer as a computer-implemented method, or in special hardware, or as a combination of both. Executable code for an embodiment of the method may be stored in a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product includes non-transitory program code stored in a computer readable medium for performing an embodiment of the method when the program product is executed in a computer.

[0016] In one embodiment, the computer program comprises computer program code adapted to perform all or part of the steps of the method embodiments when the computer program is run on a computer. Preferably, the computer program is embodied in a computer readable medium.

[0017] Further details, aspects and embodiments are described, by way of example only, with reference to the drawings, in which elements are illustrated for simplicity and clarity and are not necessarily drawn to scale, and in which elements corresponding to elements already described may be labeled with the same reference numerals. [Brief description of the drawings]

[0018] [Figure 1a] FIG. 1 illustrates a schematic diagram of an example embodiment of a medical image segmentation system. [Figure 1b] FIG. 1 is a schematic diagram illustrating an example embodiment of an atlas image selection system. [Figure 2a] FIG. 2 shows a schematic diagram of an example embodiment of a segmentation function; [Figure 2b] FIG. 2 shows a schematic example of an embodiment of image segmentation; [Figure 2c] FIG. 2 shows a schematic diagram of an example embodiment of a segmentation quality determination; [Diagram 3] FIG. 1 illustrates a schematic example of an embodiment of a method for medical image segmentation. [Figure 4a] FIG. 1 illustrates a schematic example of an embodiment of a method for selecting an atlas image for use in a medical image segmentation method. [Figure 4b] FIG. 1 illustrates a schematic example of an embodiment of a method for selecting an atlas image for use in a medical image segmentation method. [Figure 5a] FIG. 1 illustrates a schematic example of an embodiment of a method for medical image segmentation. [Figure 5b] FIG. 1 illustrates a schematic example of an embodiment of a method for selecting an atlas image for use in a medical image segmentation method. [Figure 6a] FIG. 1 illustrates a schematic example of a medical imaging embodiment; [Figure 6b] FIG. 1 illustrates a schematic example of a medical imaging embodiment; [Figure 6c] FIG. 10 is a schematic diagram illustrating dice scores for an example embodiment of a correctly working segmentation function. [Figure 6d] FIG. 10 is a schematic diagram illustrating dice scores for an example embodiment of a segmentation function indicative of failure modes. [Figure 7a] 1 is a schematic diagram of a computer readable medium including a writeable portion containing a computer program according to an embodiment; [Figure 7b] FIG. 2 is a schematic diagram illustrating a representation of a processor system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] While the subject matter disclosed herein is susceptible to embodiments taking many different forms, one or more specific embodiments have been shown in the drawings and are described in detail herein, with the understanding that the disclosure is to be considered as an example of the principles of the subject matter disclosed herein and is not intended to limit the subject matter disclosed herein to the specific embodiments shown and described.

[0020] In the following, for purposes of understanding, elements of the embodiments are described in operation, but it will be apparent that each element is configured to perform the functions described as being performed by that element.

[0021] Furthermore, features described in this specification or recited in mutually different dependent claims may be combined, so that the subject matter disclosed herein is not limited to the embodiments.

[0022] Fig. 1a shows a schematic example of an embodiment of a medical image segmentation system 110. For example, the image segmentation system 110 is configured for the image segmentation method, for example, as presented herein. Fig. 1b shows a schematic example of an embodiment of an atlas image selection system 160. For example, the system 160 is configured for the atlas image selection method, for example, as presented herein. For example, the atlas image selected in the system 160 is used in the system 110 for image segmentation.

[0023] The medical image segmentation system 110 and the atlas selection system 160 may be separate systems. The medical image segmentation system 110 and the atlas selection system 160 may be combined in the machine learning system 100. The system 100 is configured for atlas selection and for image segmentation, which is part of the segmentation function training and / or part of the segmentation in new images.

[0024] For example, the medical image segmentation system 110 is used in a production environment. For example, the medical image segmentation system 110 is used by a medical professional, a medical imaging system operator, etc., to generate segments of new images. The segmentation function in the medical image segmentation system 110 is typically a machine learning segmentation function, e.g., the segmentation function is trained or has been trained based on a number of training images. For convenience, it is assumed that the segmentation function is of machine learning type, but this is not strictly required. For example, a hand-crafted segmentation function, e.g., an expert system, etc., may be further improved by the embodiment. The machine learning segmentation function may, for example, include a neural network configured to receive an image as an input and to generate a segment as an output, but this is not required. For example, the segmentation function may comprise other machine learning functions, e.g., a deep forest, or may comprise an ensemble of one or more machine learning functions.

[0025] For example, the atlas image selection system 160 is used when preparing the image segmentation system 110. For example, the system 160 is used together with the system to train the segmentation function of the system 110. For example, the system 160 is further configured to train the parameters of the segmentation function used in the image segmentation system 110. Although it is optional, for example, the training of the segmentation function may be performed by a different device. In fact, it is an advantage that the atlas image can be selected after the training of the segmentation function is completed. Interestingly, a fully trained segmentation function can be obtained from any source, for example from some third party, and then an atlas image is selected for it. This approach subsequently enhances the accuracy and robustness of any existing segmentation function for image segmentation. This is maintained even if the use of the atlas image was not intended or expected during the design or training of the segmentation function.

[0026] The image segmentation system 110 comprises a processor system 130, a memory 140, and a communication interface 150. The image segmentation system 110 is configured to communicate with a segmentation function storage 112 and an atlas image storage 114.

[0027] The segmentation function storage 112 is configured to store a segmentation function, which is configured to receive a medical input image and to generate image segments. The atlas storage 114 is configured to store a plurality of atlas images.

[0028] The atlas image selection system 160 includes a processor system 170, a memory 180, and a communication interface 190. The system 160 includes a segmentation function storage 112, for example similar to that in the system 110, and a test image storage 164.

[0029] The segmentation function storage 112 is configured to store a segmentation function configured to receive a medical input image and to generate image segments.

[0030] Test image storage 164 is configured to store a plurality of test images and corresponding test image segments.

[0031] Storages 112, 114, and 164 may be local storage to system 110 or 160, such as a local hard drive or memory. Storages 112, 114, and 164 may also be non-local storage, such as cloud storage, in which case storages 112, 114, and 164 may be implemented as storage interfaces to the non-local storage.

[0032] The systems 110 and / or 160 communicate with each other, external storage, input devices, output devices, and / or one or more sensors via a computer network. The computer network may be the Internet, an intranet, a LAN, a WLAN, etc. The computer network may be the Internet. The systems include connection interfaces configured to communicate within the system or, as needed, with the outside of the system. For example, the connection interfaces include connectors, such as wired connectors, such as Ethernet connectors, optical connectors, etc., or wireless connectors, such as antennas, such as Wi-Fi, 4G, or 5G antennas.

[0033] The communication interface is used to transmit or receive input images for segmentation, atlas images, segmentation function parameters, output images, etc.

[0034] The implementation of the systems 110 and 160 may be realized in a processor system, e.g., one or more processor circuits, e.g., microprocessors, examples of which are shown herein. The system 110 may be realized in a device that may or may not include a storage unit. The system 160 may be realized in a device that may or may not include a storage unit. The systems 110 and 160 may be realized in a system or device, etc. The systems 110 and / or 160 comprise functional units configured for elements, e.g., steps, or portions, etc. of the image segmentation method and / or atlas selection method embodiments. The functional units are realized in whole or in part by computer instructions stored in the systems 110 and 160, e.g., in electronic memories of the systems 110 and 160, and executable by the microprocessors of the systems 110 and 160. In hybrid embodiments, the functional units are realized in part by hardware, e.g., coprocessors, e.g., segmentation function coprocessors, graphics coprocessors, etc., and in part by software stored and executed in the systems 110 and 160. The segmentation function parameters and / or training data may be stored locally in the systems 110 and 160 or in cloud storage.

[0035] 2a illustrates in a schematic manner an example of an embodiment of a segmentation function 210 for use, for example, in an embodiment of an image segmentation method and / or in an embodiment of an atlas image selection method. For example, the segmentation function 210 is configured to receive a medical input image 212 and to generate image segments 214. For example, the segmentation function is configured to receive a 2D or 3D image I and to generate annotations, for example segments A. The segments are binary images. For example, the image is an image of lungs and the segments are binary images indicating whether a pixel / voxel of the input image corresponds to a lung or not. The segmentation function 210 generates corresponding annotations A1, A2, ..., A. m With a set of images I1,I2,…,Im The segmentation function is trained or has been trained based on . Training of the segmentation function is done in a conventional manner. For example, a neural network is trained, for example using backpropagation. The training uses the Adam optimizer.

[0036] Fig. 2b shows a schematic example of an embodiment of image segmentation. A medical image 213 is shown in Fig. 2b. The medical image 213 shows objects of interest, e.g. lungs, kidneys, heart, etc., as well as objects of no interest. In the image 213, the objects of interest are indicated diagrammatically by squares. The segment 215 is in this case a binary image where black pixels indicate the presence of an object of interest.

[0037] Instead of a binary image, a segment may contain more than two different classifications, e.g. to indicate more than one object of interest. For example, a segment is a two- or three-dimensional array indicating, for pixels or voxels in the input, which object they belong to. The size of the segment along that dimension may be the same as that of the input image, but the size may also be smaller, e.g. one output element classifies multiple input elements, e.g. pixels / voxels. The size of the segment along that dimension may also be larger, e.g. shape-based interpolation is used to upsample the result.

[0038] In one embodiment, the input image is an abdominal CT and the objects of interest are the liver, spleen, left kidney, and right kidney. The segmentation function is a neural network, e.g., U-type or F-type, e.g., a deep convolutional neural network (CNN). The segmentation function is further trained for other objects and image modalities, e.g., arteries, bones, etc., or MRI, x-ray, etc. The segmentation method indicates common anatomical features of medical images, e.g., anatomical features present for anatomically normal, e.g., average humans. The segmentation method also or alternatively indicates medical abnormalities, e.g., tumors, fractures, etc.

[0039] Fig. 3 shows a schematic example of an embodiment of a method for medical image segmentation. Fig. 5a shows a schematic example of an embodiment of a method 400 for medical image segmentation. Fig. 3 shows data structures and data items used in the method 400. However, data structures and data items such as those shown in Fig. 3 are manipulated using an embodiment of the method 400.

[0040] The method 400 includes obtaining 410 a segmentation function 210, the segmentation function configured to receive a medical input image and to generate an image segment.

[0041] For example, the segmentation function 210 may be a convolutional network, or a ResNet type architecture, etc. Obtaining the segmentation function may include obtaining the segmentation function from storage, or receiving the segmentation function from a computer network, etc. The segmentation function may be expressed as a collection of segmentation function parameters.

[0042] Typically, the segmentation function 210 is fully trained before atlas image selection, but this is not required. For example, after atlas selection, the segmentation function is further trained while including input image alignment. For example, this is fine-tuning. In fact, during training, we even perform multiple segmentation function applications on multiple aligned input images, and, for example, in one embodiment, fuse the resulting segments and compute an error signal from the final fused segment. The error signal is used for further training, for example fine-tuning. The advantage of this approach is that the segmentation function is optimized for use with the selected atlas image. However, this fine-tuning is not required, and for example, atlas selection may be used as a way to boost the performance of the segmentation function without the need to further train it.

[0043] The method 400 includes obtaining 420 a plurality of atlas images 320. Atlas images 321, 322, and 323 are shown in FIG. 3. There may be more or fewer atlas images. For example, there may be four or more, eight or more, sixteen or more atlas images. The atlas images are selected according to a method for atlas image selection. For example, the atlas images are selected for various positive characteristics. For example, the atlas images segment well. For example, the atlas images represent a good cross-section of the image population.

[0044] The method 400 includes receiving 430 a medical image 312. For example, the medical image may be acquired from a medical imaging device, such as a CT device, an MRI device, an X-ray device, etc. For example, the image may be represented as a 2D array of pixels or a 3D array of voxels. The image may be compressed or may be in raw format, etc.

[0045] The method 400 comprises registering 440 the received image 312 to the multiple atlas images, thus obtaining multiple registered images 330 and multiple corresponding registration transformations configured to register the received image to the multiple atlas images. Figure 3 shows registered images 331, 332 and 333. For clarity, the corresponding transformations are not shown in Figure 3. Figure 3 shows a registration function 325 configured to register the image 312 to the atlas image 320.

[0046] Image registration is a process in which a source image is transformed to better align with a target image. For function 325, image 312 is the source image and then the atlas image serves as the target image. The registration function is configured to select an allowed transformation from a predefined class of transformations. For example, the classes of transformations may be, for example, translation registration, rigid registration, similarity registration, affine registration, or non-rigid registration. Rigid transformations include translation and rotation. Similarity transformations include translation, rotation, and scaling. The selection of the registration is through optimizing a loss function.

[0047] Elastic registration is used for the registration function 325. For example, a diffeomorphism is selected by the registration function 325. Elastic transformation allows for close alignment between the source and target images. Therefore, it is expected that the segmentation quality of the atlas image can be matched by the transformed image.

[0048] After alignment of an input 312, multiple aligned images are obtained, e.g., one for each atlas image. When aligning an image 312 with an atlas image, function 325 generates not only the aligned image, but also a transformation that maps the image 312 to the atlas image. For example, given an input image I and an atlas image I, i Given,the resulting transformation is chosen from the transformation class,T i (I) is at least to some extent image I i and align.

[0049] The method 400 includes applying 450 the segmentation function 210 to the plurality of aligned images 330, thus obtaining a plurality of aligned image segments 340. For example, the segmentation function NN may be NN(T i(I)), where i takes on values ​​ranging from the number of atlas images. The output of the segmentation function is a two- or three-dimensional array of the same or smaller dimensions as the images 312. The elements of the array indicate the segments determined by the segmentation function. A segment element is a value indicating the type of the corresponding pixel / voxel, but can also be a vector. For example, if segmentation into p objects is desired. The elements in the output array of the segmentation function are p-dimensional vectors. The elements in the vector indicate the observed objects. The sum of the vectors is 1, or scaled to 1 etc.

[0050] The method 400 includes applying 460 inverses of the alignment transformations to the image segments 340, thus obtaining the image segments 350. For example, the method may include:

number

number

[0051] The method 400 includes determining 470 an output segment 361 from the plurality of image segments. For example, determining includes majority voting. For example, the image segment 350 includes a binary image and the segment 316 further includes a binary image, in which the pixel value is determined by the most frequently occurring value at the corresponding pixel of the segment 350. The same approach may be used for a multi-value image instead of a binary image. If the segment 350 is an array, e.g., a 2D or 3D array, containing p-dimensional vectors, the corresponding vector elements are averaged according to some averaging function. For example, vectors v each corresponding to the same element, e.g., pixel, of the segment 350 are averaged 470 according to some averaging function.i Given a set of 1 / nΣv i , the corresponding vector in segment 361 may be determined as follows. Interestingly, the majority function may be approximated by using a power average, for example the root mean square. For example,

number

[0052] As mentioned above, the segmentation function trained for image segmentation is extended by aligning the images to the atlas image. This has the advantage that instead of one segment, the segmentation function is forced to provide multiple segments, which are essentially for the same image, but this information is not available to the segmentation function.

[0053] Some types of segmentation functions are invariant to some transformations. For example, if the segmentation function includes a convolutional neural network, the segmentation function is invariant under translation, at least to some extent, but this does not hold for slightly more complicated transformations. In fact, convolutional networks are typically not invariant under rigid body transformations, let alone elastic transformations, e.g., diffeomorphic transformations. Thus, if it occurs that the image 312 falls into a failure mode of the segmentation function 210, this may not be correct for the registered version of the image 312, and even less likely for most of the registered images 330.

[0054] In one embodiment, the segmentation function is not invariant under a class of transformations used to determine the registration quality and / or for image segmentation. For example, in one embodiment, the segmentation function is invariant under a first set of transformations and a second set of transformations used to determine the registration quality and / or for image segmentation, where the second set is larger than the first set, e.g., the first set is a subset of the second set. For example, in one embodiment, the segmentation function includes a convolutional neural network and the transformations used to determine the registration quality and / or for image segmentation are larger than the set of transformations, e.g., further include rotations.

[0055] For example, if we assume that the probability that image 312 is mis-segmented is about 5%, and that 10 atlas images are used, then the probability that the majority will be mis-segmented is approximately

number

[0056] The atlas images may be selected from the images on which the segmentation function was trained. For example, one may select the atlas images randomly. Selecting the atlas images from the training images has the advantage that the segmentation function is likely to be familiar with those images. However, better results are achieved with more careful selection, for example as described herein.

[0057] Figure 4a shows a schematic example of an embodiment of a method for selecting an atlas image for use in a medical image segmentation method. Figure 5b shows a schematic example of an embodiment of a method 500 for selecting an atlas image for use in a medical image segmentation method. An atlas image is selected for a method, such as method 400.

[0058] For example, the method 500 uses the data items and structures shown in Figure 4a, and vice versa. The method 500 further uses the data shown in Figure 4b, which is described below.

[0059] The method 500 includes obtaining 510 a segmentation function 210 configured to receive an input image and generate image segments. The segmentation function 210 is the same as the segmentation function used for generation purposes. As mentioned above, the segmentation function includes a neural network, for example U or F net, CNN, etc. are used.

[0060] The method 500 comprises obtaining 520 a number of test images 610 and corresponding test image segments 620. For example, the test images 610 are obtained from training the segmentation function 210. For example, the segmentation function 210 is trained based on a number of pairs of training images and training image segments. For example, a backpropagation method is used to train the neural network based on the training set. Typically, some image set is left for testing purposes, for example to test the convergence of the segmentation function. The test images 610 are obtained from the images used for training and / or from those used to test the segmentation function. The test images 610 are preferably obtained from the same or similar image distribution from which the images on the basis of which the segmentation function is trained are drawn.

[0061] The test images 610 are all images on which the segmentation function 210 was trained and / or tested, particularly if the total number of images was relatively small, e.g., less than about 100. The test images 610 may also be a subset, e.g., a random selection of images used for training and / or testing, which is useful when the number of training / test images is large.

[0062] The method 500 includes determining 530 a segmentation quality 630 for a number of test images by comparing associated test image segments 620 with the image segments generated by the segmentation function. For example, the test image 610 includes associated segments, such as ground truth segments 620, which may also have been used in training or testing. There are several ways to compute the segmentation quality. One way to do so is to compute a Dice score between the ground truth image segments and the segments generated by the segmentation function. For example, the segmentation quality is obtained for the image 611 by computing a Dice score between segments 621 and 631. A high Dice score indicates good overlap and thus a high segmentation quality.

[0063] The method 500 includes selecting 540 one or more of the test images as atlas images with segmentation quality above a threshold. For example, a Dice score above a threshold is required, e.g., a Dice score above 0.8. The threshold is predefined. The threshold may be dynamic. For example, the atlas images are selected from the best performing images. For example, a percentage of the worst scoring images are discarded.

[0064] The inventors have found that in practice most images perform well in segmentation, but a certain percentage, e.g., about 5% of the images, exhibit worse segmentation than most images. By selecting the atlas images from the best performing 95%, or 90%, or 80%, etc. of the images, one avoids aligning the input images 312 based on images on which the segmentation function performs poorly, thus increasing the likelihood that the segmentation function will perform poorly on further aligned images.

[0065] For example, in one embodiment, the worst performing portions of images 610, e.g., the bottom 5%, are discarded. Then, atlas images are selected from the remaining images, which is performed randomly, e.g., randomly selecting 10 images.

[0066] For example, in addition to, or possibly instead of, segmentation quality, further criteria are imposed for selecting the atlas images.

[0067] 2c shows a schematic example of an embodiment of the segmentation quality determination. An image 212 is shown with associated true image segments, e.g. ground truth segments. The image 212 is segmented by a segmentation function 210 to obtain a generated image segment 214. The generated image segment 214 and the true image segment 216 are then compared using, e.g., a comparator 220 to obtain a segmentation quality 217. For example, the comparator 220 computes a Dice score.

[0068] Fig. 4b shows a schematic example of an embodiment of a method for selecting an atlas image for use in a medical image segmentation method. An image 610' is shown in Fig. 4b that has not yet been selected as an atlas image and has not yet been discarded for other reasons, e.g. because it has too bad segmentation quality. Also shown in Fig. 4b is an image 651 that has already been selected as an atlas image. For example, image 651 has been selected as having the best segmentation quality, or it has been selected randomly from images with good segmentation quality.

[0069] The alignment function is applied to image 610' in selected image 651. The alignment quality is obtained by comparing the aligned test image with selected test image 651. In this way, the alignment quality 640 is obtained. The comparison uses a comparator 220, e.g., a Dice score. Other similarity measures, e.g., correlation, may be used instead of the Dice score.

[0070] Interestingly, and different from the segmentation quality, atlas images with low registration quality are selected. A low registration quality indicates that the image is very different from the images already selected as atlas images. For example, the image has the lowest registration score. For example, the image with the best registration quality is discarded and one or more random images are then selected. For example, the image is selected from the 50% of images with the lowest registration quality.

[0071] In one embodiment, the registration performed in method 400, e.g., in registration function 325, selects a registration from the same class of transformations as the registration performed to compute the registration quality. In particular, the registration types for atlas image selection and form image segmentation are translation-only transformations, e.g., shift transformations, rigid transformations, similarity transformations, affine transformations, or non-rigid transformations, e.g., elastic transformations, e.g., diffeomorphic transformations. The transformations are inelastic, or at least much less elastic.

[0072] Interestingly, in the method 400, for example for the registration performed in the registration function 325, it is possible to select a different registration, for example from a larger class of transformations than the registration performed to compute the registration quality. In particular, the former are elastic registrations, for example diffeomorphic transformations, while the latter transformations are inelastic or at least much less elastic. For example, a rigid registration may be used to compute the registration quality. Rigid transformations only allow translation and rotation. A slightly larger class is allowed, for example translation, rotation and scaling, or affine transformations.

[0073] The elastic registration in method 400 allows the images to be matched as closely as possible to the atlas image, and thus allows obtaining a segmentation quality comparable to that of the atlas image, whereas the elastic registration makes the images too similar, and the rigid transformation allows a better diversity of the images to be observed.

[0074] In Fig. 4b, multiple test images are compared to one atlas image, e.g., after a first atlas image is selected based solely on segmentation quality, it is used to select a second atlas image, e.g., used to add together an atlas image that has different alignment terms from a particular previously selected atlas image, which may be selected randomly from the already selected atlas images, or, e.g., by repeatedly using each of the selected images, e.g., one at a time.

[0075] Another approach is to compare the test image 610' not with one selected atlas image 651, but with multiple atlas images, and even with all currently selected atlas images. For example, calculate the registration quality between the test image 610' and the currently selected atlas image. Then, an overall registration quality for the test image 610' can be determined, which is for example the average of the registration qualities. A high average indicates that the registration quality is often high. Instead of the arithmetic mean, an exponentiation technique may be used, which highlights some very different atlas images, even if they are similar to most of the others. Selecting a further atlas image with a low overall registration quality is more likely to give independent segments to the segmentation function. On the other hand, if the further atlas image has a high segmentation quality, the segmentation function may give a suitable performance for it.

[0076] Selection based on global alignment is performed by discarding images that are too similar and randomly selecting from the remaining ones, although other selection methods are possible as well, for example as shown herein.

[0077] For example, the following steps are used:

[0078] 1. Select a random pool of test images, say 100 or 1000 training images, that contain relevant true segments.

[0079] 2. Discard a% of the pool that have the worst segmentation quality, say a is 5%.

[0080] 3. Select a random atlas image from the current pool and remove the selected image from the pool.

[0081] 4. Compute the overall registration quality for the pool of test images compared to the selected atlas images.

[0082] 5. Discard images from the pool that have excessively high overall registration quality.

[0083] 6. Unless the pool is empty or enough atlas images have been selected, proceed to Part 3.

[0084] Step 2 uses absolute scores instead of relative scores, e.g., images with a dice score of less than 0.8 between the generated segment and the true segment are discarded.

[0085] Step 4 is performed efficiently because the registration quality between each test image and the atlas image is stored and does not need to be recomputed, so if one atlas image is added, only the registration quality between the test image and the added atlas image needs to be computed.

[0086] Step 5 removes b% of the pool, for example as low as 5%, or slightly higher, for example 25%. However, instead of a fixed percentage an absolute threshold may also be used. The absolute threshold is determined empirically.

[0087] Instead of a random selection, for example in step 3, one may select the image with the best segmentation quality and / or the image with the worst registration quality, or a combination thereof, e.g., summation, etc. The above-described atlas image selection algorithm is an example, different implementations are possible.

[0088] In general, the limits on segmentation quality and / or registration quality, whether absolute or relative, depend on the difficulty of the segmentation problem, the quality of the segmentation function, e.g., the amount of training data, the tolerance for false positives, etc. If the segmentation function includes a neural network, its quality depends on its depth. The limits are established empirically by keeping a separate set of training images as test data and evaluating the resulting segmentation system.

[0089] In various embodiments of systems 110 and 160, the communications interface may be selected from a variety of alternatives. For example, the interface may be a network interface to a local or wide area network, such as the Internet, a storage interface to internal or external data storage, a keyboard, an application program interface (API), etc.

[0090] Systems 110 and 160 may include a user interface that encompasses familiar elements such as, for example, one or more buttons, a keyboard, a display, a touch screen, etc. The user interface may be configured to accommodate user interaction for configuring the system, training a segmentation function based on a training set, or applying the system to new image data, selecting an atlas image, etc.

[0091] The storage may be implemented as an electronic memory, such as a flash memory, or a magnetic memory, such as a hard disk. The storage may comprise multiple independent memories that together constitute the storage 140, 180. The storage may include a temporary memory, such as a RAM. The storage may be a cloud storage.

[0092] The system 110 may be implemented in one device. The system 160 may be implemented in one device. Typically, each of the systems 110 and 160 comprises a microprocessor that executes appropriate software stored in the system, for example, the software may be downloaded and / or stored in a corresponding memory, for example a volatile memory, for example a RAM, or a non-volatile memory, for example a flash. Alternatively, the systems may be implemented wholly or partly in programmable logic, for example a field programmable gate array (FPGA). The systems may be implemented wholly or partly as so-called application specific integrated circuits (ASICs), for example integrated circuits (ICs) customized for their specific use. For example, the circuits may be implemented in CMOS, for example using hardware description languages, for example Verilog, VHDL, etc. In particular, the systems 110 and 160 may comprise a circuit for the evaluation of a segmentation function.

[0093] The processor circuitry may be implemented in a distributed manner, for example as multiple sub-processor circuits. The storage may be distributed across multiple distributed sub-storages. Some or all of the memory may be electronic memory, magnetic memory, etc. For example, the storage may include volatile and non-volatile portions. Part of the storage may be read-only.

[0094] In one embodiment, test time data augmentation is included in, for example, method 400. The segmentation function, for example segmentation function 210, uses training of a conventional segmentation function, for example, the segmentation function is configured for a conventional machine learning algorithm. For example, the machine learning algorithm is a decision forest. For example, the machine learning algorithm is a neural network. For example, the segmentation function includes a neural network, for example a convolutional neural network (CNN), such as U-net or F-net. The atlas images are selected from the training images. Below, some further optional modifications, details, and embodiments are presented.

[0095] In one embodiment, n≧2, e.g., n=10, atlas images are used. A new image for segmentation is aligned to the n atlases, resulting in n reversible transformations. A segmentation function, e.g., a CNN, is applied to each of the n transformed images, resulting in n annotated images, e.g., segmented images. The inverse transformation resulting from the alignment is applied to the annotated images, and the labels of the transformed annotated images are fused to obtain the final segmentation.

[0096] The basic idea also applies more generally to other tasks such as image classification (e.g., using ResNet-type neural network architectures), in which case the inverse transformation step after application of the segmentation function can be skipped.

[0097] The segmentation function training is performed on the corresponding annotations A1,…,A m with m 2D or 3D images I1,…,I m The training of the 2D / 3D segmentation function for image segmentation can be done conventionally. For example, a neural network is trained using, for example, an Adams optimizer, for example, using backpropagation. The m images with corresponding annotations are used as a whole for training the segmentation function, or are subdivided into training and test sets.

[0098] Atlas selection selects a subset of m images, n (≦m), as the atlas. Individual images to be used as the atlas are selected using various criteria. One criterion may be the segmentation quality observed during training (e.g., Dice score of the image after training when it is part of the training data). Another criterion may be the segmentation quality observed during testing (e.g., Dice score of the image after training when it is part of the test data). Another criterion may be image difference. For example, a rigid body registration is applied to the training images (e.g., to the atlas candidates), and then the candidate with the smallest mutual annotation overlap in terms of Dice score after registration to the already selected candidate is selected, and this process is repeated iteratively to select n atlas images.

[0099] A large number of algorithms have been presented for image-to-atlas registration and image transformation and have been applied to medical image registration. Recently, registration algorithms that utilize deep learning and neural networks have been used more and more. Algorithms that result in diffeomorphic transformations have the advantage that the inverse transformation can be easily obtained. Depending on the images handled, similarity measures can be used that are suitable for registering images of the same modality, such as cross-correlation, or for registering images of different types, such as mutual information. Image I new and atlas images I1,…,I n The application of a (non-rigid) registration algorithm to, e.g., T i (I new ), we use n transformations T1,…,T n results.

[0100] If the atlas image has a larger field of view than the new image being segmented, the atlas image can be used to complement the field of view of the new image.

[0101] In addition to spatial image transformation, a new image I newThe image grey values ​​of T may be transformed to better represent the atlas image. This transformation is called the transformation T i For example, a parametric transformation is applied to the intensities, and the parameters are determined such that the resulting histogram of the new image after the intensity transformation corresponds to the atlas image.

[0102] The transformed image T1(I new ),…,T n (I new ), e.g., multiple aligned images are segmented using a previously trained segmentation function and annotated with annotation A 1,new ,…,A n,new After segmentation, the annotations are transformed back to the space of the new image, resulting in the annotation

number

number

[0103] Label Fusion: Annotation

number

number

[0104] An embodiment is shown using the example of rib and spine segments in a 3D CT image. Figure 6a shows a visual representation of such a CT-scan. In particular, an F-net neural network has been trained to solve this task. The image in Figure 6a has been segmented using a segmentation function that includes a neural network, and a Dice score between the output of the segmentation function and the expert's segments has been computed. The diagram in Figure 6c shows the Dice score for the spine at 652 and the Dice score for the ribs at 662. Note that both scores are high, indicating that the segmentation function is performing well on the image in Figure 6a.

[0105] During training of the neural network on the segmentation function, data augmentation was used, in this case a rotation of the CT data set up to 7°. Accurate output is expected from the segmentation function over this range. Fig. 6b shows the same image as Fig. 6a but with a significantly different orientation, i.e. a rotation of 45° in this example. Note that apart from this rotation, the images are the same. When the segmentation function is applied to Fig. 6b, the resulting Dice scores are substantially smaller. The diagram in Fig. 6d shows the Dice scores for the spine at 654 and for the ribs at 664. Note that both scores are smaller, indicating that the segmentation function does not perform well on the image in Fig. 6b. Note that the score for the spine is about half of the previous value, indicating a collapse of the predictive ability for spine objects in this case.

[0106] Even a rigid registration to an atlas image similar to that in Fig. 6a, Fig. 6b, returns the segmentation ability of the segmentation function. More subtle prediction collapse, e.g., due to unique anatomical variations, can be addressed by non-rigid registration. Applying the segmentation function to a transformed version of the test image label fusion improves the segmentation.

[0107] For example, the segmentation function, atlas images, or test images are obtained from an electronic storage system, either internal or external, addressed, for example, via a computer cable or computer network. The segmentation function is trained based on images, e.g., medical images, obtained from a medical imaging device, e.g., a CT scanner. Receiving medical images for segmentation is performed in the same manner as obtaining, described above. Receiving is internal or external, via an API or other interface. Images are received from an imaging device at a medical terminal. Registering images, determining registration or segmentation quality, etc. is performed at an electronic device, such as a computer. The neural network for the segmentation function includes multiple layers, including, for example, convolutional layers, during training and / or application. For example, the neural network in the segmentation function includes at least 2, 5, 10, 15, 20, or 40, or more hidden layers, etc. The number of neurons in the neural network may be, for example, at least 10, 100, 1000, 10000, 100000, 1 million,000,000, or even greater.

[0108] As will be apparent to one of ordinary skill in the art, many different ways of performing the methods herein, such as methods 400 and / or 500, are possible. For example, the steps may be performed in the order shown, but the order of the steps may be changed or some steps may be performed in parallel. Furthermore, steps of other methods may be inserted between steps. The inserted steps may represent, for example, improvements to the methods described herein or may be unrelated to the method. Some portions may be performed at least partially in parallel. Furthermore, a given portion may not be completely finished before the next step is begun.

[0109] The method embodiments may be implemented using software that includes instructions for causing a processor system to perform the method 400 and / or 500. The software may only comprise steps performed by a particular sub-entity of the system. The software may be stored on a suitable storage medium, such as a hard disk, floppy, memory, optical disk, etc. The software may be transmitted wired, wirelessly, or as a signal using a data network, such as the Internet. The software may be made available for download and / or for remote use at a server. The method embodiments may be implemented using a bitstream configured to form a programmable logic unit, such as a field programmable gate array (FPGA), to perform the method.

[0110] It is understood that the subject matter disclosed herein also extends to a computer program, in particular a computer program on or in a medium, adapted to implement the subject matter disclosed herein. The program may be in the form of source code, object code, code intermediate source, and object code, for example in partially compiled form, or in any other form suitable for use in implementing the embodiments of the method. An embodiment of a computer program product includes computer executable instructions corresponding to each of the processing steps of at least one of the described methods. These instructions may be subdivided into subroutines and / or stored in one or more files that are statically or dynamically linked. Another embodiment of a computer program product includes computer executable instructions corresponding to each of the devices, units and / or parts of at least one of the described systems and / or products.

[0111] Fig. 7a shows a computer readable medium 1000 including a writeable portion 1010 including a computer program 1020, which includes instructions for causing a processor system to implement the segmentation method and / or the atlas selection method according to the embodiment. The computer program 1020 may be embodied in the computer readable medium 1000 as a physical mark or by magnetization of the computer readable medium 1000. However, any other suitable embodiment is also contemplated. Furthermore, although the computer readable medium 1000 is shown in this example as an optical disk, it is understood that the computer readable medium 1000 may be any suitable computer readable medium, such as a hard disk, a solid state memory, a flash memory, etc., and may be non-writable or writable. The computer program 1020 includes instructions for causing a processor system to implement the segmentation method and / or the atlas selection method.

[0112] Fig. 7b shows a schematic diagram of a processor system 1140 according to an embodiment of a segmentation device and / or system and / or an atlas selection device and / or system. The processor system comprises one or more integrated circuits 1110. The architecture of the one or more integrated circuits 1110 is shown in a schematic diagram in Fig. 7b. The circuit 1110 comprises a processing unit 1120, e.g. a CPU, for operating computer program components to execute a method according to an embodiment and / or to realize a module or unit thereof. The circuit 1110 comprises a memory 1122 for storing programming code, data, etc. Part of the memory 1122 may be read-only. The circuit 1110 comprises a communication element 1126, e.g. an antenna, a connector, or both. The circuit 1110 comprises a dedicated integrated circuit 1124 for performing part or all of the processing defined in the method. The processor 1120, memory 1122, special purpose ICs 1124, and communication elements 1126 may be connected to each other via an interconnect 1130, such as a bus. The processor system 1110 may be configured for contact and / or contactless communication using antennas and / or connectors, respectively.

[0113] For example, in one embodiment, the processor system 1140, e.g., segmentation device and / or system and / or atlas selection device and / or system, comprises a processor circuit and a memory circuit. The processor is configured to execute software stored in the memory circuit. For example, the processor circuit is an Intel Core i7 processor, an ARM Cortex-R8, etc. In one embodiment, the processor circuit is an ARM Cortex M0. The memory circuit may be a ROM circuit, or a non-volatile memory, e.g., a flash memory. The memory circuit may be a volatile memory, e.g., an SRAM memory. In the latter case, the device may comprise a non-volatile software interface, e.g., a hard drive, a network interface, etc., configured to provide the software.

[0114] Although device 1100 is shown to include one of each of the described components, various components may be duplicated in various embodiments. For example, processor 1120 may include multiple microprocessors configured to independently execute the methods described herein or to perform steps or subroutines of the methods described herein, such that multiple processors cooperate to achieve the functionality described herein. Furthermore, when device 1100 is implemented by a cloud computing system, various hardware components may reside in separate physical systems. For example, processor 1120 may include a first processor in a first server and a second processor in a second server.

[0115] It should be noted that the above-described embodiments are illustrative rather than limiting of the subject matter disclosed herein, and that those skilled in the art will be able to design many alternative embodiments.

[0116] In the claims, any reference signs placed in parentheses shall not be construed as limiting the claim. Use of the verb "comprise" and its conjugations does not negate the presence of elements and steps other than those stated in a claim. The reference of an element in the singular does not negate the presence of a plurality of such elements. A phrase such as "at least one of", when preceded by a list of elements, denotes the selection of all or any subset of the elements from the list. For example, the phrase "at least one of A, B, and C" should be understood to include A only, B only, C only, both A and B, both A and C, both B and C, or all of A, B, and C. The subject matter disclosed herein may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a device claim listing several parts, several of these parts may be embodied by one and the same hardware item. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0117] In the claims, reference signs in parentheses represent reference signs in drawings of exemplary embodiments or represent formulas of embodiments to facilitate the understanding of the claims, and these reference signs should not be construed as limiting the claims. [Explanation of symbols]

[0118] 100 Machine Learning Systems 110 Medical Image Segmentation System 112 Segmented Function Storage 114 Atlas Image Storage 130 Processor System 140 Memory 150 Communication Interface 160 Atlas Image Selection System 164 test image storage 170 Processor System 180 Memory 190 Communication Interface 210 Segmentation Functions 212, 213 Medical input images 214, 215 Image segments 216 True Image Segments 220 Comparator 217 Segmentation Quality 320 Multiple Atlas Images 321~323 Atlas Images 312 Medical Imaging 325 Alignment Functions 330 Multiple Aligned Images 331~333 Aligned images 340 Multiple Aligned Image Segments 341-343 Aligned image segments 350 Multiple Deregistered Image Segments 351~353 Deregistered image segments 361 Output Segments 610 Multiple Test Images 610' Test image not yet selected 611~613 Test images 620 Multiple Test Image Segments 621~623 Test image segment 630 Multiple Segmentation Qualities 631~633 Segmentation Quality 651 Atlas Images 640 Multiple Alignment Quality 641~643 Alignment Quality 652, 654 Dice score for spine 662, 664 Dice score for ribs 400 Method for medical image segmentation Get Segmentation Function 410. The segmentation function is configured to receive the medical input image and to generate image segments. 420 Acquire multiple atlas images. 430 Receive medical images. 440 registers the received image to the multiple atlas images, thus obtaining multiple registered images and multiple corresponding registration transformations configured to register the received image to the multiple atlas images. 450 A segmentation function is applied to the multiple aligned images to obtain multiple aligned image segments. 460 Applying the inverse of the multiple alignment transforms to the multiple aligned image segments to obtain multiple image segments. 470 An output segment is determined from the multiple image segments. 500 Method for selecting an atlas image for use in a medical image segmentation method 510 Obtain a segmentation function configured to receive an input image and to generate image segments. 520 A plurality of test images and corresponding test image segments are obtained. 530 Determine segmentation quality for the plurality of test images by comparing associated test image segments to the image segments generated by the segmentation function. 540 Select one or more of the test images as the atlas images having a segmentation quality above a threshold. 1000 Computer Readable Medium 1010 Writable part 1020 Computer Programs 1110 Integrated circuits 1120 Processing Unit 1122 Memory 1124 Special Purpose Integrated Circuit 1126 Communication Elements 1130 Interconnection 1140 Processor System

Claims

1. receiving a medical input image and obtaining a segmentation function that generates image segments; acquiring a plurality of atlas images; receiving a medical image; registering the plurality of atlas images with the received medical images to obtain a plurality of registered images and a plurality of corresponding registration transformations that register the received medical images to the plurality of atlas images; applying the segmentation function to the plurality of registered images to obtain a plurality of registered image segments; applying an inverse of the corresponding alignment transforms to the aligned image segments to obtain a plurality of image segments; determining an output segment from the plurality of image segments; 23. A method for medical image segmentation comprising:

2. the segmentation function is a machine learning function trained based on a plurality of training images and corresponding training image segments, and the atlas image is selected from the plurality of training images. The method of claim 1.

3. 1. A method for selecting an atlas image for use in a method for medical image segmentation, the method comprising: receiving an input image and obtaining a segmentation function for generating image segments; obtaining a plurality of test images and corresponding test image segments; determining a segmentation quality for the plurality of test images by comparing associated test image segments with image segments generated by the segmentation function; selecting one or more of the test images as the atlas images having a segmentation quality above a threshold; The method comprising:

4. For the test images that have not yet been selected, applying an alignment from the unselected test images to a selected test image and determining an alignment quality by comparing the aligned test images to the selected test images; selecting a test image as a further atlas image having a registration quality below a threshold; 4. The method of claim 3, comprising:

5. For the test images that have not yet been selected, applying an alignment to the test images selected from the test images and determining a number of alignment qualities; determining an overall registration quality from the determined plurality of registration qualities; selecting a test image as a further atlas image having an overall registration quality below a threshold; 5. The method of claim 4, comprising:

6. the registration is one of a translational registration, a rigid registration, a similarity registration, an affine registration, or a non-rigid registration; The method according to claim 4 or claim 5.

7. determining the segmentation quality and / or the alignment quality comprises determining a dice score; The method according to claim 5 or claim 6.

8. the segmentation function is trained on a plurality of training images, the test image being included in the training images; 8. The method according to any one of claims 3 to 7.

9. the segmentation function is a neural network; 9. The method according to any one of claims 1 to 8.

10. A method for selecting an atlas image according to any one of claims 3 to 9, The method for medical image segmentation according to claim 1.

11. a segmentation function storage for storing a segmentation function, the segmentation function receiving a medical input image and generating image segments; an atlas storage that stores a plurality of atlas images; a communication interface for receiving medical images; 1. A processor system comprising: registering the received medical images to the plurality of atlas images, thus obtaining a plurality of registered images and a plurality of corresponding registration transformations that register the received medical images to the plurality of atlas images; applying the segmentation function to the plurality of registered images to obtain a plurality of registered image segments; applying an inverse of the corresponding alignment transforms to the aligned image segments to obtain an image segment; determining an output segment from the plurality of image segments; A processor system for performing A medical image segmentation system comprising:

12. a segmentation function storage for storing a segmentation function, the segmentation function receiving a medical input image and generating image segments; a test image storage for storing a plurality of test images and corresponding test image segments; 1. A processor system comprising: determining a segmentation quality for the plurality of test images by comparing associated test image segments with image segments generated by the segmentation function; selecting one or more of the test images as atlas images having a segmentation quality above a threshold; A processor system for performing 1. An atlas image selection system for use in a medical image segmentation method or device, comprising:

13. comprising data representing instructions which, when executed by a processor system, cause said processor system to perform the method of any one of claims 1 to 10; Transient or non-transient computer readable media.

Images