Method and system for segmenting an image

Abstract

In a method for segmenting features in an image, image products associated with the image are provided to a model trained using a machine learning process 102. Shape descriptor indications for features in the image are received from the model based on the image products 104. A model-based segmentation MBS is then initialized using the indicated shape descriptors to segment the features 106.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION Embodiments herein relate to image processing, particularly, but not exclusively, to segmenting one or more features of an image. [Background technology]

[0002] The present disclosure is in the field of image segmentation. The disclosure herein can be applied to a diverse range of images, such as medical images. Image segmentation involves extracting shape / morphological information about objects or shapes captured in an image. This can be achieved by converting the image into building blocks or "segments" that represent different features in the image. In some methods, image segmentation can involve fitting a model to one or more features in the image. Summary of the Invention [Problem to be solved by the invention]

[0003] One method of image segmentation is model-based segmentation (MBS), whereby a triangular mesh of a target structure (e.g., heart, brain, lungs, etc.) is iteratively adapted to features in the image. Segmentation models typically encode population-based appearance features and shape information. Such information describes the allowable shape variations based on the actual shape of the target structure in members of the population. Shape variations can be encoded, for example, in the form of eigenmodes that describe how changes to one part of the model are constrained or dependent on the shape of other parts of the model.

[0004] Model-based segmentation has been used in various applications to segment one or more target organs from medical images; see, for example, Ecabert, O., et al. 2008 entitled "Automatic Model-Based Segmentation of the Heart in CT Images"; IEEE Trans. Med. Imaging 27 (9), 1189-1201. Due to the use of triangular surface meshes, MBS generally provides smooth segmentation results. Furthermore, MBS is generally considered robust to image artifacts such as variations in image quality.

[0005] Another segmentation method uses machine learning (ML) models to convert an image into multiple constituent shapes (e.g., block shapes or block volumes) based on similar pixel / voxel values ​​and image gradients, as described, for example, in Long et al. "Fully Convolutional Networks for Semantic Segmentation."

[0006] The purpose of the embodiments herein is to improve upon these methods. [Means for solving the problem]

[0007] As mentioned previously, two common methods for image segmentation are model-based segmentation (MBS) and machine learning (ML) segmentation. MBS involves including prior shape knowledge in the mesh (e.g., shape constraints) because only certain eigenmodes of mesh deformation are allowed during the fitting process. This can generally result in an accurate fit that matches the typical shape associated with the object being segmented. For example, if the MBS model is a heart, only allowable deformations that result in the shape of the heart observed in real human populations can be allowed. However, edge cases that require the model to deform toward the limits of the model's tolerance can result in spurious fits.

[0008] In contrast, machine learning (ML) approaches operate voxel-wise and are therefore generally not subject to shape constraints. As a result, they can fit arbitrary shapes, but the lack of prior shape knowledge can lead to lower accuracy. This can also result in non-smooth or leaky segmentation results.

[0009] The aim of the embodiments herein is to improve upon such methods to provide improved segmentation, especially for edge cases that may not be well handled by MBS models.

[0010] Thus, according to a first aspect herein, there is a method for segmenting features in an image, the method comprising providing an image product associated with the image to a model trained using a machine learning process, receiving from the model, based on the image product, an indication of shape descriptors for features in the image, and using the indicated shape descriptors in a model-based segmentation MBS to initialize the MBS and segment the features.

[0011] In this manner, a model trained using a machine learning process can be used to predict one or more shape descriptors used to initialize an MBS model. In this manner, for example, the coarse segmentation results of an ML (machine learning) model can be used to initiate a more accurate MBS segmentation, as described in more detail below.

[0012] According to a second aspect, there is a method of training a machine learning model for use in a segmentation process for segmenting features in an image, the method comprising: obtaining training data comprising: i) example image products, each example image product associated with a respective image, and ii) corresponding ground truth shape descriptors for features in each example image product; and training the model to predict new shape descriptors for new image products based on the training data.

[0013] According to a third aspect, there is provided a system for segmenting features in an image. The system includes a memory including instruction data representing an instruction set, and a processor in communication with the memory and configured to execute the instruction set. When executed by the processor, the instruction set causes the processor to provide image products associated with the image to a model trained using a machine learning process, receive from the model, based on the image products, indications of shape descriptors for features in the image, and use the indicated shape descriptors in a model-based segmentation MBS to initialize the MBS and to initialize one or more features.

[0014] According to a fourth aspect, there is provided a system for training a machine learning model for use in a segmentation process for segmenting features in an image. The system includes a memory including instruction data representing an instruction set, and a processor in communication with the memory and configured to execute the instruction set. When executed by the processor, the instruction set causes the processor to: obtain training data including: i) example image products, each example image product associated with a respective image; and ii) corresponding ground truth shape descriptors for features in each example image product; and train the model to predict new shape descriptors for new image products based on the training data.

[0015] According to a fifth aspect, there is provided a computer program product having a computer readable medium, the computer readable medium having computer readable code configured to, when executed by a suitable computer or processor, cause the computer or processor to perform an embodiment of either the first or second aspect.

[0016] For a better understanding and to show more clearly how the embodiments herein may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief explanation of the drawings]

[0017] [Figure 1] 1 illustrates a method according to some embodiments herein. [Figure 2] FIG. 1 illustrates an embodiment of a method 100. [Figure 3] FIG. 1 illustrates an example architecture of a model according to some embodiments. [Figure 4] 1 illustrates an example method according to some embodiments of the present disclosure. [Figure 5]FIG. 10 illustrates another method according to some embodiments herein. [Figure 6] FIG. 1 illustrates a system according to some embodiments herein. DETAILED DESCRIPTION OF THE INVENTION

[0018] As briefly mentioned above, machine learning (ML) segmentation, in which a machine learning model is trained to transform an image into its constituent shapes based on pixel / voxel values ​​and image gradients, can segment any image with some accuracy, but this method can be limited in that ML segmentation is generally not constrained by prior shape knowledge, which can also result in non-smooth or leaky segmentation results.

[0019] In contrast, model-based segmentation (MBS) can provide highly accurate segmentations due to the incorporation of prior shape knowledge, but MBS models can provide poor (or incorrect) fits when the objects they are fitting have shapes that are towards the limits of the model's allowed deformation modes.

[0020] It is an object of the embodiments herein to provide an improved method and system for segmenting images.

[0021] 1 illustrates a method 100 for segmenting features in an image according to some embodiments of the present disclosure. In a first block 102, the method 100 includes providing an image product associated with the image to a model trained using a machine learning process. In a second block 104, the method includes receiving, from the model based on the image product, an indication of a shape descriptor of a feature in the image. In a third block 106, the indicated shape descriptor is used in a model-based segmentation (MBS) to initialize the MBS and segment the feature.

[0022] Using a model (e.g., a machine learning model) to predict approximate shape descriptors to use in initializing the MBS model in this way allows the MBS model to generate a more accurate fit. For example, if the anatomical shape is "too extreme" and therefore outside the MBS capture range, MBS hierarchical adaptation will yield the best model fit within the capture range (which can be thought of as a local minimum fit solution).

[0023] Therefore, if the model can be better initialized (e.g., by using shape descriptors from an ML algorithm as described here), then "extreme" anatomical geometries that would otherwise be out of capture range can be captured (or, in other words, the global minimum of the fitting algorithm can be found).

[0024] More particularly, the image (e.g., the image to be segmented) may be any type of image. In some embodiments, the image may include a scientific image. In some embodiments, for example, the image may include a medical image.

[0025] A medical image can comprise an image acquired using any imaging modality. Examples of medical images include, but are not limited to, a computed tomography (CT) image (e.g., from a CT scan), such as a C-arm CT image, a spectral CT image, or a phase contrast CT image, an X-ray image (e.g., from an X-ray scan), a magnetic resonance (MR) image (e.g., from an MR scan), an ultrasound (US) image (e.g., from an ultrasound scan), a fluoroscopic image, a nuclear medicine image, or any other three-dimensional medical image.

[0026] More generally, the image may comprise an image captured using a charged coupled device CCD, such as found in a camera. Those skilled in the art will appreciate that the embodiments herein may be applied to other types of images and / or other data sets that may be segmented.

[0027] In general, an image can include a two-dimensional image or a three-dimensional image. An image can have multiple (or sets of) image components. For example, in embodiments where the image includes a two-dimensional image, the image components can include pixels. In embodiments where the image includes a three-dimensional image, the image components can include voxels.

[0028] A feature in an image may comprise any (e.g., real or simulated) object, shape of an object, or portion of an object or its shape that is visible (e.g., identifiable) in the image. In embodiments in which the image comprises a medical image, the feature may comprise an anatomical feature or portion of an anatomical feature, such as a portion of a lung, heart, brain, or any other anatomical feature.

[0029] Moving to blocks 102 and 104 of method 100 ("Providing an image product associated with the image to a model trained using a machine learning process" and "Receiving from the model, based on the image product, an indication of shape descriptors for one or more features in the image"), in embodiments herein, a machine learning model is used to predict one or more shape descriptors for features in the image that can be used by the MBS model to initialize the MBS model.

[0030] As described in more detail below, the machine learning model can predict shape descriptors, for example, based on the image itself or based on segment output from an initial segmentation performed by another type of segmentation process, such as a machine learning segmentation process. In this way, an initial, coarse segmentation can be used by the machine learning model (such as a neural network) to predict parameters that can be used to initialize the MBS model. In this way, a more accurate segmentation can be performed by the MBS model.

[0031] The model can comprise any type of model, such as, for example, a model trained using a machine learning process. Examples of models that can be used herein include, but are not limited to, neural network models, such as deep learning neural networks, and random forest models.

[0032] Those skilled in the art will be familiar with neural networks, but simply put, a neural network is a type of supervised machine learning model that can be trained to predict a desired output given input data. A neural network is trained by providing training data that includes example input data and the corresponding "correct" or ground truth results that are desired. A neural network includes multiple layers of neurons, each representing a mathematical operation that is applied to the input data. The output of each layer of the neural network is fed to the next layer, which generates an output. For each portion of the training data, the weights associated with the neurons are adjusted until optimal weights are found that produce predictions for the training examples that reflect the corresponding ground truth.

[0033] Although examples of neural networks are provided herein, those skilled in the art will understand that in general, the trained model may comprise any model that can be trained to take an image product as input and output (e.g., predict) a shape descriptor indication for one or more features in the image.

[0034] In some implementations, an image product associated with an image can include the image itself. For example, an image product associated with an image can include a portion of the image (e.g., all or a portion of the original image). In other words, in some embodiments, a model can be used to predict shape descriptors of features in an image based on the image itself (e.g., taking the image as input). In other examples, an image product can include a portion of the image (e.g., all or a portion of the original image) that has undergone a preprocessing process. Examples of preprocessing processes include, but are not limited to, smoothing or filtering the image to, for example, enhance features (or smooth background noise).

[0035] In some embodiments, the image products provided to the model may have segments generated by another pixel- or voxel-based segmentation (e.g., a segmentation process that results in segments having pixels or voxels (in 2D or 3D, respectively)). In other words, in some embodiments, the model may be used to predict shape descriptors for features in the image based on the segments generated in an initial segmentation of the image. The initial segmentation may be a coarse (e.g., approximate or quick) segmentation.

[0036] In some embodiments, the image products provided to the model may comprise segments generated by a machine learning (ML)-based segmentation process. In embodiments, the ML-based segmentation process may comprise an ML-based volume segmentation process. In general, the ML-based segmentation process may comprise any machine learning method for semantic image segmentation (2D or 3D segmentation). Those skilled in the art will be familiar with ML segmentation processes. An example of such a process is described in the aforementioned paper by Long et al., entitled "Fully Convolutional Networks for Semantic Segmentation."

[0037] The method may therefore further include performing an initial segmentation of the image using a machine learning (ML)-based segmentation process. Segments from the initial segmentation performed by the ML-based segmentation process may then be provided as input to a model, and based on the input, the model may output or predict an indication of the shape descriptor.

[0038] It is further noted that in some embodiments, both the original image (or a preprocessed version thereof) and the segments generated by the initial segmentation can be provided to a model trained using a machine learning process in order for the model to predict shape parameters. In other words, in some embodiments, the model can take as input the image and the segments generated by the ML segmentation algorithm.

[0039] This is illustrated in Figure 2, which shows an embodiment in which a deep learning (DL) neural network is used to estimate shape descriptors of organs from ML-based volume segmentation results of segmentation performed on medical images.

[0040] In this example, the feature to be segmented comprises a slice through a simulated 3D cylinder curved upward toward the top of the image. In this embodiment, the image slice is segmented by an ML segmentation process to generate a segment 202 of the cylinder, as shown in Figure 2a.

[0041] Figure 2b shows the outline of an MBS model 204 of a cylinder superimposed on a segment. In this embodiment, the cylinder 202 is deformed toward the limits of the allowable deformation of the model 204. When the MBS model is applied to this cylinder, a poor (or incorrect) fit may occur; for example, the corner of the model labeled 204a in Figure 2 may incorrectly fit to corner 202a of the cylinder instead of the correct corner 202b.

[0042] However, according to embodiments herein, the segments 202 generated by the ML segmentation process are provided as input to a deep learning neural network, which outputs or predicts shape descriptor indications for features in the image based on the segments 202.

[0043] The shape descriptor is provided as an input parameter to the MBS-based model mesh, which initializes the MBS mesh by (e.g., roughly) aligning the shape of the average model mesh with the shape of the ML result. This is shown in FIG. 2c, which shows the initialized MBS mesh 206 when initialized using the shape descriptor. Now, the MBS mesh is approximately aligned with the object, and a local MBS fine-segmentation can be applied (in the classical / normal way) to the original image to provide a more accurate segmentation 208 of the object, the result of which is shown in FIG. 2d.

[0044] In this way, models such as deep learning (DL) models can estimate approximate deformations for initializing an MBS mesh from segments generated by ML segmentation. This avoids the aforementioned problems associated with using MBS models when segmenting shapes toward the extremes of MBS model deformation. This combines the strengths of ML (which provides coarse segmentation regardless of shape) and the MBS segmentation process (which provides highly accurate segmentation when properly initialized) to provide an overall improved segmentation process.

[0045] Returning now to method 100, in some embodiments, the model can be trained to predict shape descriptor indications based on training data, where such training data includes: i) example image products; and ii) a corresponding ground truth shape descriptor for each example image product. For example, a training data set can be compiled to include example image products and a "correct" or desired output for each example input image. Further details regarding training are provided below with respect to FIG. 4.

[0046] In some embodiments, the shape descriptor may include one or more parameters that may be used (e.g., input) to the MBS model to initialize the MBS model, for example, by aligning the shape of a mean model mesh associated with the MBS to the shapes of one or more segments generated by the ML-based volumetric segmentation process according to the labeled shape descriptor.

[0047] In general, a shape descriptor may have one or more weights or eigenvalues ​​that may be used to initialize the shape of the model. In some embodiments, a shape descriptor may describe the offsets of individual vertices in the MBS model. In some embodiments, a shape descriptor may specify an affine transformation (or a set of affine transformations) that is applied to portions of a mesh with appropriate boundary conditions, thereby resulting in pathology-related mesh deformations (e.g., self-intersections). However, these are merely examples, and those skilled in the art will be familiar with other ways in which an MBS model may be initialized, and thus other forms that a shape descriptor may take, to implement embodiments of the present disclosure.

[0048] As described above, the shape descriptor indication can have one or more values ​​that can be used to initialize the MBS model. In some embodiments, the shape descriptor indication can have one or more values ​​or classifications from which the shape descriptor can be determined or calculated. In some examples, the values ​​may be continuous. In other examples, the values ​​may be discretized. Thus, in general, neural network-based estimation of shape descriptors can be treated as a regression or classification task.

[0049] In some embodiments, the model comprises a recurrent neural network, in which sense the shape descriptors output by the neural network may be in the form of continuous values.

[0050] In one embodiment, the ML-based segmentation is provided as a 3D volume containing integer class labels (e.g., each voxel in the volume can be labeled 1 or 0, thereby indicating that the voxel either forms part of the segmented volume or is not part of the segmented object). In this embodiment, the shape descriptor of the MBS model corresponds to an n-dimensional real-valued vector. As mentioned above, this parameterization ω can express the deformation of the original model shape, such that pathology-related mesh deformations (such as self-intersections) occur, either by describing the offsets of individual vertices (in this case the dimensionality of ω would be quite high), or by describing a set of affine transformations applied to parts of the mesh with appropriate boundary conditions; it should be noted that in the latter case (description of a set of affine transformations), the parameterization vector ω will have a smaller dimensionality, which may simplify the estimation task.

[0051] In this embodiment, a shape descriptor, To estimate TIFF0007755170000001.tif1020, a regression network N is used. In other words, I comprises the segments output from the ML segmentation process, as described above with respect to FIG. 2. As is typical in classical regression analysis, this embodiment uses a network architecture parameterized by a loss function L and a vector θ. Given a training set T containing pairs of segmentation volumes and associated shape descriptors, the optimal network parameterization θ * is achieved by minimizing the average loss over the entire training data, i.e. the optimization task: This can be obtained by solving TIFF0007755170000002.tif1594.

[0052] An example of a network architecture is shown in Figure 3. In this example, the network contains three different types of layer modules: convolution, batch normalization, rectified linear unit (CBR) block 302, pooling layer 304, and linear layer 306. The input layer 300 receives the input image, and the output layer 308 provides predictions of the shape descriptor(s). The CBR block consists of a 3x3x3 padded convolution (C), followed by batch normalization (B) and leaky rectified linear unit activation (R). For pooling, a 2x2x2 average pooling with stride 2 is chosen, which reduces the spatial tensor dimension by 0.5 at each step: i.e., (H m ,W m ,D m )=0.5*(H m-1 ,W m-1 ,D m-1 ). The network weights can be trained in mini-batches using a regularized L2 loss along with a prominent update rule such as SGD, AdaDelta, or Adam. The hyperparameters of this training setup, in particular the number of hidden layers d and the number of feature channels generated by the convolutions F, m may vary depending on the particular application. As a preprocessing step, the input segmentation volumes are resampled, padded, and cropped so that their spatial extension matches the shape of the network input tensor.

[0053] In this way, recurrent deep neural networks can be used to directly predict the values ​​of shape descriptors.

[0054] Note that in some embodiments, prior statistical knowledge of the distribution of shape descriptors across the training set may be incorporated into the loss function. For example, strong deviations from the mean shape descriptor vector may be penalized. This reduces the probability that the network will generate highly "unlikely" shape descriptors. This can be thought of as similar to Bayesian optimization, in the sense that the prior knowledge imposed on the shape descriptor distribution may introduce bias, but reduces variance.

[0055] Turning now to classification approaches, in some embodiments, the model includes a classification neural network. In such embodiments, the output indications of the shape descriptor are partitioned into discrete values. That is, the shape descriptor output by the neural network can be discretized. This can reduce the computational power required to implement the method.

[0056] For example, instead of directly relating the segmentation volume I to the shape descriptors through a learned regression model, a neural network can be used to predict the upper and lower bounds of each component of the associated shape descriptor.

[0057] As an example, the following approach can be used:

[0058] Given a ground truth dataset S containing a segmentation volume and a pair of shape descriptors (I,ω), we compute R based on the distribution of the shape descriptor vectors. n A partition of can be determined.

[0059] To illustrate, consider the following example: sample mean, Based on TIFF0007755170000003.tif1236, R n by iterating through all possible combinations of ">" and "<" relationships. TIFF0007755170000004.tif992 - 2 coprime ncan be decomposed into a set of

[0060] Decomposition R n =U i C i After selecting, each shape descriptor in the ground truth dataset can be assigned a class label based on the associated partitioning factor, i.e., a shape descriptor ω can be assigned a class label based on the partitioning factor C i If it is contained in the label l ω :=i can be assigned.

[0061] The obtained image labels (I,l ω ) serve as training data for the classification network. Solving classification tasks with neural networks has been extensively discussed in the literature, and therefore those skilled in the art will be familiar with possible network specifications and training regimes.

[0062] Following the steps outlined above, the resulting neural network consists of a segmentation element C with associated shape descriptors. i can be applied to the unseen segmented volume to predict C. i By randomly sampling from a distribution centered on, we obtain explicit shape descriptor estimates that can be used for MBS initialization.

[0063] More formally, in some embodiments, receiving an indication of the shape descriptor from the model can include receiving a partition element associated with the partitioned discrete value from the classification neural network. The partition element can represent an upper or lower bound of the shape descriptor for the feature in the image.

[0064] In embodiments in which partition elements (e.g., rather than values) of the shape descriptors are received from the model, the method may include converting the partition elements to values ​​or other forms that may be used to initialize the MBS model. In some embodiments, the method may further include determining a shape descriptor of a feature in the image from the partition elements by determining a centroid of the partition elements and using the centroid as a shape descriptor of the feature. Alternatively, the method may further include determining an estimate of the shape descriptor based on random sampling of a distribution centered on the partition element and using the estimate of the shape descriptor as a shape descriptor of the feature.

[0065] Returning to method 100, block 106 of initializing an MBS model and segmenting features using labeled shape descriptors in model-based segmentation may include initializing an MBS model using the shape descriptors and performing segmentation on the image using the initialized MBS model. For example, the shape descriptors may be provided to the MBS model as input parameters for performing segmentation on the image (e.g., original or preprocessed image).

[0066] This approach overcomes some of the downsides of MBS (out-of-bounds detection when the target organ shape in the image deviates strongly from the model's shape) and ML segmentation (lack of shape regularization) by using a model that predicts shape predictors from ML segments that can be used to initialize the MBS model. In this way, the two approaches can be used together to produce improved results. It can be applied to any voxel-based segmentation, especially those generated by ML algorithms, such as coarse volumetric segmentations of organs.

[0067] In some embodiments, as described above, the proposed algorithm estimates shape descriptors from ML results and maps them onto an MBS model, effectively deforming the model's average mesh shape to roughly match the ML results. As a result, MBS is only applied locally to fine-tune the segmentation results, thus simplifying the complex multi-level model hierarchies often currently employed. This method overcomes the common problem of MBS approaches, which require good initialization to ensure the model is well within the capture range of the organ boundary to be detected.

[0068] Another embodiment is shown in Figure 4, which shows a summary of how a machine learning segmentation process is used to generate segments that are provided as input to a (second) model trained using the machine learning process. The model uses the segments generated by the machine learning segmentation process to predict shape descriptors for use in initializing the MBS.

[0069] The method begins at block 402, where an image is provided to a machine learning segmentation model (labeled ML Algorithm* in FIG. 4), which segments the image to generate one or more segments of features within the image at block 404. The machine learning segmentation model operates on the original image data and derives a segmentation of the image at the voxel / pixel level.

[0070] The segments output by the machine learning segmentation model are provided as input to a model (labeled ML Algorithm** in FIG. 4) trained using a machine learning process. This model may include, for example, a neural network, and may use the segments to predict or estimate shape descriptors (406) for the segments.

[0071] The predicted / estimated shape descriptor is used to initialize the model-based segmentation process MBS 408. MBS 408 takes as input the image 402 and the shape descriptor 406. The shape descriptor is used to initialize MBS. The model-based segmentation produces a (mesh-based) segmentation output.

[0072] In another embodiment, a model trained using a machine learning process (e.g., the ML algorithm of FIG. 4) can also take the original image as input. For example, a neural network can be used to predict a shape predictor based on the segmentation from the ML-based segmentation and the original image.

[0073] 5, in some embodiments there is a method 500 of training a machine learning model for use in a segmentation process for segmenting features in an image. Briefly, the method comprises, in a first block 502, obtaining training data, the training data comprising: i) example image products, each example image product associated with a respective image; and ii) corresponding ground truth shape descriptors for features in each example image product. In a second block 504, the method comprises training the model to predict new shape descriptors for new image products based on the training data.

[0074] The images and features therein may include any type of image and any type of feature, including the types of images and features described above with respect to method 100. Furthermore, the model may be any type of model suitable for predicting shape descriptors from image products, as described above with respect to method 100. It will be understood that the details and embodiments of the models described above with respect to Figures 1-3 apply equally to embodiments of method 500.

[0075] The acquiring training data block 502 may include acquiring example image products, each example image product being associated with a respective image, and annotating the acquired example image products with corresponding ground truth ("correct") shape descriptors of features in each example image product.

[0076] As noted above, image products can include images or products derived from images. In some implementations, as detailed above, image products can include segments generated by, for example, an ML segmentation process.

[0077] The training data can include many training examples. In general, as will be appreciated by those skilled in the art, the accuracy of a model in predicting a shape descriptor can be further improved with a larger training set. Thus, in some embodiments, the training data can include hundreds or thousands of training examples.

[0078] As described in detail above with respect to method 100, in some embodiments, the model may comprise a classification neural network. In such embodiments, the ground truth values ​​may be discretized. As such, method 500 may further comprise partitioning the values ​​of the ground truth shape descriptors in the training data into discrete values. Training 504 the model to predict shape descriptors for new image products may include training the model to predict new partition values ​​of the shape descriptors for the new image products based on the training data.

[0079] In some embodiments, the segment value represents an upper or lower bound on the shape descriptor. As described above with respect to method 100, such upper or lower bounds can be used to determine values ​​of shape segments that can be used to initialize an MBS model, which can be performed, for example, by determining the centroids of the segment elements and using such centroids as shape descriptors of the features, or by computing estimates of the shape descriptors based on random sampling of a distribution centered on the segment elements and using the estimates of the shape descriptors as shape descriptors of the features.

[0080] As detailed above, one or more new shape descriptors are used in initializing a model-based segmentation MBS for segmentation of a new image.

[0081] 6, in some embodiments there is a system 600 for displaying a three-dimensional volume of images on a two-dimensional display. The system may form part of a specialized device such as a specialized medical device, alternatively the system may form part of a computer system such as a laptop, desktop PC or other device, or system 600 may form part of a cloud / distributed computing arrangement.

[0082] The system includes a memory 604 containing instruction data representing an instruction set, and a processor 602 in communication with the memory and configured to execute the instruction set. Generally, the instruction set, when executed by the processor, can cause the processor to perform any of the embodiments of method 100 or 500 described above. In some embodiments, the instruction set can include multiple software and / or hardware modules, each performing or configured to perform an individual step or steps of a method described herein.

[0083] More specifically, in some embodiments, the set of instructions, when executed by a processor, causes the processor to perform the steps of: providing an image product associated with an image to a model trained using a machine learning process; receiving from the model, based on the image product, an indication of a shape descriptor of a feature in the image; and initializing a model-based segmentation MBS using the indicated shape descriptor to initialize one or more features in a model-based segmentation MBS.

[0084] While providing image products associated with an image to a model trained using a machine learning process, receiving shape descriptor indications for features in the image from the model based on the image products, and initializing the MBS using the indicated shape descriptors in a model-based segmentation to segment one or more features have been described in detail with respect to method 100 and blocks 102, 104, and 106, it will be understood that those details apply equally to the configuration of system 600.

[0085] In another embodiment, the set of instructions, when executed by a processor, causes the processor to perform the steps of obtaining training data, the training data comprising: i) example image products, each example image product associated with a respective image; and ii) corresponding ground truth shape descriptors of features in each example image product; and training the model based on the training data to predict new shape descriptors for new image products.

[0086] Obtaining training data and training the model based on the training data to predict new shape descriptors for new image products is described in detail above with respect to method 500, but it will be understood that the details apply equally to system 600.

[0087] In any of the embodiments of system 600, processor 602 can include one or more processors, processing units, multi-core processors, or modules configured or programmed to control system 600 in the manner described herein. In an implementation, processor 602 can include multiple software and / or hardware modules, each configured or programmed to perform individual or multiple steps of the methods described herein. Processor 602 can include one or more processors, processing units, multi-core processors, and / or modules configured or programmed to control system 600 in the manner described herein. In some embodiments, for example, processor 602 can include multiple (e.g., interoperated) processors, processing units, multi-core processors, and / or modules configured for distributed processing. Those skilled in the art will understand that such processors, processing units, multi-core processors, and / or modules can each be located in different locations and can perform different steps of the methods described herein and / or different portions of a single step.

[0088] The memory 604 is configured to store program code executable by the processor 102 to perform the methods described herein. Alternatively or additionally, one or more memories 604 may be external to the system 600 (i.e., separate or remote). For example, one or more memories 604 may be part of another device. The memory 604 may be used to store images, image products, models, shape descriptors, segments, information, and / or data received, calculated, or determined by the processor 602 of the device 600 or from any interface, memory, or device external to the system 600. The processor 602 may be configured to control the memory 604 to store the images, image products, models, shape descriptors, segments, information, and / or received, calculated, or determined data.

[0089] In some embodiments, memory 604 may have multiple sub-memories, each capable of storing a portion of instruction data, such as at least one sub-memory capable of storing instruction data representing at least one instruction of an instruction set, and at least one other sub-memory capable of storing instruction data representing at least one other instruction of the instruction set.

[0090] In some embodiments, system 600 may further comprise a display for displaying / rendering the images, image products, MBS model, and / or any other information input to or output from system 600 (e.g., determined / calculated). The display may comprise, for example, a computer screen, a screen on a mobile phone or tablet, a screen forming part of a medical device or medical diagnostic tool, a screen forming part of a visor associated with goggles or a virtual reality display, or any other display.

[0091] The system may further include a user input such as a keyboard, mouse, or other input device that allows a user to interact with the system, for example, to provide initial input parameters used in the aspects described herein.

[0092] 6 shows only the components necessary to illustrate this aspect of the disclosure, and in actual implementations, system 600 may have additional components to those shown. For example, system 600 may include a battery or other power source for powering system 600, or a means for connecting system 600 to a mains power source.

[0093] In another embodiment, a computer program product includes a computer-readable medium having computer-readable code embodied therein that, when executed by a suitable computer or processor, causes the computer or processor to perform one or more of the methods described herein.

[0094] It will therefore be understood that the present disclosure also applies to computer programs, in particular computer programs on or in a carrier, arranged to implement the embodiments, which may be in the form of source code, object code, code intermediate sources and object code, for example in partially compiled form, or in any other form suitable for use in the implementation of the methods according to the embodiments described herein.

[0095] It will also be appreciated that such programs have many different structural designs. For example, program code implementing the functionality of a method or system may be subdivided into one or more subroutines. Many different ways of distributing functionality among these subroutines will be apparent to those skilled in the art. The subroutines may be stored together in an executable file to form a self-contained program. Such an executable file may include computer-executable instructions, such as processor instructions and / or interpreter instructions (e.g., Java interpreter instructions). Alternatively, one or more or all of the subroutines may be stored in at least one external library file and linked with the main program statically or dynamically, such as at run time. The main program includes at least one call to at least one subroutine. The subroutines may also include function calls to each other.

[0096] The carrier of a computer program may be any entity or apparatus capable of carrying the program. For example, the carrier may comprise a data storage device such as a ROM, for example a CD-ROM or a semiconductor ROM, or a magnetic recording medium, for example a hard disk. Furthermore, the carrier may be a transmissible carrier such as an electrical or optical signal, which may be transmitted via an electrical or optical cable or by radio or other means. When the program is embodied in such a signal, the carrier may be constituted by such a cable or other device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, this integrated circuit being configured to perform, or to be used in the performance of, the relevant method.

[0097] Variations to the disclosed embodiments can be understood and implemented by those skilled in the art, from a study of the drawings, the disclosure, and the claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. A computer program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be interpreted as limiting the scope of the claims.

Claims

1. 1. A method for segmenting features in an image, the method comprising: performing an initial segmentation of the image using a machine learning based segmentation process; providing image products having segments generated by the machine learning based segmentation process to a model trained using a machine learning process, the model being trained to predict shape descriptors for use in initializing model-based segmentation based on training data having: i) example image products; and ii) corresponding ground truth shape descriptors for each example image product, the model trained using the machine learning process including a classification neural network, and output indications of the shape descriptors being partitioned into discrete values; receiving, from the model based on the image product, a shape descriptor of the feature in the image, the step including receiving from the classification neural network a partition element associated with a partitioned discrete value, the partition element representing an upper or lower bound of the shape descriptor for the feature in the image; using the shape descriptor in a model-based segmentation of the image to initialize the model-based segmentation and segmenting the features using a mesh of a target structure that is iteratively adapted to the features in the image; and the method comprises: determining the centroid of the segmentation element and using the centroid as the shape descriptor for the feature; or determining an estimate of the shape descriptor based on random sampling of a distribution centered on the partition element, and using the estimate of the shape descriptor as a shape descriptor for the feature; determining the shape descriptor for the feature in the image from the segmentation elements by either The method further comprises:

2. 2. The method of claim 1 , wherein the step of using the shape descriptor in the model-based segmentation comprises initializing the model-based segmentation by using the shape descriptor to align a shape of a mean model mesh associated with the model-based segmentation to the shapes of the segments produced by the machine learning-based segmentation process.

3. The method of claim 1 , wherein the image product comprises one of a portion of the image or a portion of the image that has undergone a pre-processing process.

4. The shape descriptor is: one or more weights or eigenvalues ​​that may be used to initialize the shape of the model-based segmentation; vertex offsets in the model-based segmentation, and / or an affine transformation applied to the mesh 4. The method of claim 1, comprising:

5. 1. A method for training a machine learning model for use in a segmentation process for segmenting features in an image, the method comprising: acquiring training data, the training data comprising: i) example image products, each example image product having segments generated by a machine learning based segmentation process performed during an initial segmentation of a respective image; and ii) corresponding ground truth shape descriptors for use in initializing a model-based segmentation for features within each example image product; training the machine learning model to predict new shape descriptors for new image products based on the training data, where the new shape descriptors are used to initialize a model-based segmentation of each new image; and the machine learning model includes a classification neural network, wherein the shape descriptor indications are partitioned into discrete values, partition elements are associated with the partitioned discrete values, and the partition elements represent upper or lower bounds of the shape descriptor for the feature in the image; The training step includes: determining the centroid of said segmented element and using said centroid as a new shape descriptor; or determining an estimate of a new shape descriptor based on random sampling of a distribution centered on the partition element, and using the estimate of the new shape descriptor as the new shape descriptor; determining the new shape descriptor for a feature in the image from the segmentation elements by either The method further comprises:

6. the method further comprising partitioning values ​​of the ground truth shape descriptor in the training data into discrete values ​​representing upper or lower bounds of the ground truth shape descriptor; 6. The method of claim 5, wherein the step of training the machine learning model to predict new shape descriptors for the new image products comprises training the machine learning model to predict partition values ​​of shape descriptors for new image products based on the training data.

7. 1. A system for segmenting features in an image, comprising: a memory containing instruction data representing an instruction set; a processor in communication with the memory and configured to execute the set of instructions; The instruction set, when executed by the processor, causes the processor to: performing an initial segmentation of the image using a machine learning based segmentation process; providing image products having segments generated by the machine learning based segmentation process to a model trained using a machine learning process, the model being trained to predict shape descriptors for use in initializing model-based segmentation based on training data having: i) example image products; and ii) corresponding ground truth shape descriptors for each example image product, the model trained using the machine learning process including a classification neural network, and output indications of the shape descriptors being partitioned into discrete values; receiving, from the model based on the image product, the shape descriptor of the feature in the image, the step including receiving from the classification neural network a partition element associated with a partitioned discrete value, the partition element representing an upper or lower bound of the shape descriptor for the feature in the image; using the shape descriptor in a model-based segmentation of the image to initialize the model-based segmentation and segmenting the features using a mesh of a target structure that is iteratively adapted to features in the image; Execute The instruction set, when executed by the processor, causes the processor to: determining the centroid of the segmentation element and using the centroid as the shape descriptor for the feature; or determining an estimate of the shape descriptor based on random sampling of a distribution centered on the partition element, and using the estimate of the shape descriptor as a shape descriptor for the feature; determining the shape descriptor for the feature in the image from the segmentation elements by either A system that further executes the above.

8. 1. A system for training a machine learning model for use in a segmentation process for segmenting features in an image, comprising: a memory containing instruction data representing an instruction set; a processor in communication with the memory and configured to execute the set of instructions; wherein the instruction set, when executed by the processor, causes the processor to: acquiring training data, the training data comprising: i) example image products, each example image product having segments generated by a machine learning based segmentation process performed during an initial segmentation of a respective image; and ii) corresponding ground truth shape descriptors for use in initializing a model-based segmentation for features in each example image product; training the machine learning model to predict new shape descriptors for new image products based on the training data, where the new shape descriptors are used to initialize a model-based segmentation of each new image; Execute the machine learning model includes a classification neural network, wherein the shape descriptor indications are partitioned into discrete values, partition elements are associated with the partitioned discrete values, and the partition elements represent upper or lower bounds of the shape descriptor for the feature in the image; The training step includes: determining the centroid of said segmented element and using said centroid as a new shape descriptor; or determining an estimate of a new shape descriptor based on random sampling of a distribution centered on the partition element, and using the estimate of the new shape descriptor as the new shape descriptor; determining the new shape descriptor for a feature in the image from the segmentation elements by either The system further comprises:

9. 10. A computer readable medium having computer readable code embodied therein, said computer readable code, when executed by a suitable computer or processor, causing said computer or processor to perform the method of any one of claims 1 to 6.

Images