Shape-based generative adversarial networks for segmentation in medical imaging

Abstract

For segmentation in medical imaging, a shape generative adversarial network (shape GAN) is used in training. By including shape information in a lower-dimensional space than the pixels or voxels of the image space, the network can be trained using shape losses or optimizations. Adversarial losses and shape losses are used to train the network, so the resulting generator can segment complex shapes in 2D or 3D. Other optimizations can be used, such as using losses in image space.

Description

[0001] Related cases

[0002] This application claims priority from EP19190567.8 filed on August 7, 2019, the entire contents of which are incorporated herein by reference. Background Art

[0003] The present embodiment relates to automatic segmentation in medical imaging. Shape is widely used in medical image segmentation algorithms to constrain the segmented region to a class of learned shapes. Traditional segmentation is typically based on statistical shape models or level sets. Machine learning-based segmentation may not use shape. Deep learning methods have been used to train networks for segmentation. The loss function used in training is based on pixel differences (i.e., the training loss is defined in a pixel-by-pixel manner). This results in the loss of most of the available information. The lack of a shape prior typically results in incorrect topology. Some attempts have been made to include shape-based loss terms in the optimization of deep learning problems and to optimize directly in the shape space. However, these have been shown to be effective only for simple, relatively convex shapes in two dimensions (2D), and often fail to converge for more complex three-dimensional (3D) shapes. Summary of the Invention

[0004] By way of introduction, the preferred embodiments described below include methods, computer-readable media, and systems for segmentation. A shape generative adversarial network (Shape GAN) is used in training. By including shape information in a space with a lower dimensionality than the pixels or voxels of the image space, the network can be trained using a shape loss or optimization. By using adversarial training, incorrect shapes are penalized. Adversarial loss and shape loss are used to train the network so that the resulting generator can segment complex shapes in 2D or 3D. Other optimizations can be used, such as using a loss in image space.

[0005] In a first aspect, a method for segmentation in a medical imaging system is provided. A medical scanner scans a patient, providing first imaging data representing the patient. An image processor segments an object represented by the first imaging data. The segmentation is performed using a machine learning generative network applied to the first imaging data. The machine learning generative network is adversarially trained using optimization in shape space. An image based on the segmented object is output on a display.

[0006] In one embodiment, an ultrasound scanner performs the scanning. Other modalities of scanners may be used, such as a computed tomography system or a magnetic resonance system.

[0007] In various embodiments, the segmentation comprises applying the machine learning generative network as an image-to-image convolutional deep learning network; the machine learning generative network has been trained wherein the shape space comprises a lower dimensional space from a manifold or unsupervised learning, such as the shape space is a feature matrix from principal component analysis of a signed distance function; the machine learning generative network has been trained wherein the optimization comprises a first loss in image space and a second loss in shape space; the machine learning generative network is trained to output pixel-wise or voxel-wise signed distances from the object boundary; and / or the machine learning generative network has been trained with a discriminator using an adversarial loss on voxel-wise or pixel-wise distances.

[0008] In a further embodiment, shape parameters of the shape space are determined based on a distance function output by a machine learning generation network. The shape parameters are used to classify the object based on a comparison with a parameter distribution; and / or to indicate confidence in a segment based on a comparison with a parameter distribution.

[0009] In a second aspect, a system for object localization is provided. A medical scanner is configured to scan a patient, thereby generating image data representing the patient. An image processor is configured to utilize a machine learning image-to-image network to localize an object represented in the image data. The machine learning image-to-image network is trained using a shape prior. A display is configured to display an image as a function of the located object.

[0010] In one embodiment, the machine learning image-to-image network is configured to output pixel-wise or voxel-wise distances to object boundaries and is trained with an adversarial loss on the distances. In another embodiment, the shape prior is a feature value. The machine learning image-to-image network is trained on the feature values ​​using a smooth L1 loss.

[0011] In another embodiment, a machine learning image-to-image network is trained using a first loss in image space and a second loss in shape space.

[0012] In another embodiment, the image processor is configured to determine shape parameters from the image data and use the shape parameters to assist in diagnosis. For example, the shape parameters are used to detect characteristics of the object to identify similar situations and / or to determine the confidence level of the object's location.

[0013] In a third aspect, a method for machine training a model for segmentation is provided. A generator is machine trained to output segmentation based on a shape loss and an adversarial loss. The trained generator is stored.

[0014] In a further embodiment, the ground truth is generated as a per-voxel or per-pixel distance from the object boundary. The shape loss is a shape-based smooth L1 loss per distance, while the adversarial loss is a distance in image space.

[0015] In another embodiment, machine training includes training the generator machine as a fully convolutional neural network based on shape loss, adversarial loss, and image loss. The shape loss can be based on linearized feature values ​​from the segmented output by the generator.

[0016] The present invention is defined by the appended claims, and nothing in this section should be construed as limiting those claims. Features of one type of claim (e.g., method or system) may be used in another type of claim. Additional aspects and advantages of the present invention will be discussed below in conjunction with the preferred embodiments and may be claimed later, independently or in combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The components and drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the different views.

[0018] Figure 1 is a flowchart of one embodiment of a method for training for segmentation using a shape GAN;

[0019] Figure 2 Figure 1 shows an example shape GAN.

[0020] Figure 3 is a flow chart of one embodiment of a method for segmentation using an adversarially trained generator; and

[0021] Figure 4 is a block diagram of one embodiment of a system for segmentation. DETAILED DESCRIPTION

[0022] We present a shape-based generative adversarial model for medical image segmentation. We train a shape GAN where optimization is performed in shape space in addition to image space. The shape prior in the shape GAN allows for segmentation of complex shapes.

[0023] In image space, the model's generator can be trained to output pixel or voxel values ​​based on a difference function, such as a signed difference function. The difference in position from the object boundary is estimated. For adversarial training, using functions such as signed differences helps maintain the fidelity of anatomical shapes, allowing training for complex anatomical structures. Because the discriminator penalizes "fake" shapes, the discriminator in adversarial training influences the learned shape prior. The use of the discriminator is another incorporation of shape prior information in addition to the optimization of the generator's shape space.

[0024] Various loss functions can be used. For losses in shape space, smooth L1 loss is used as the loss function for shape parameters, such as eigenvalues, rather than L2 loss. Smooth L1 loss numerically stabilizes the network, enabling training with larger images and volumes. Smooth L1 loss can also be used in image space. Using image-based smooth L1 loss accelerates network convergence.

[0025] In applications, parameters in shape space can be used as diagnostic "markers." Shape space acts as a low-dimensional feature space that captures anatomical and functional properties. Shape space parameters for a given scan of a patient can be used to detect abnormalities, subsegments, pathologies, demographic groups, or other classifications. Classification can allow the use of specifically tuned algorithms for segmentation or other diagnostic aids. Shape parameters can be used as similarity metrics to enable comparison with similar cases, can be used to identify unusual objects to enhance training data, and / or can be used to identify objects that require the use of alternative algorithms or expert opinion.

[0026] Figure 1 This is a flow chart of one embodiment of a method for machine training a segmentation model. A shape GAN is trained to train a generator for segmentation. Using optimization in shape space, the generator of the shape GAN can be trained to segment complex shapes in 2D or 3D.

[0027] Method by Figure 4 system, workstation, server, computer or other image processor. Figure 1 for training, but the resulting machine learning generator can be implemented by Figure 3 The image processor of act 32 is used to perform segmentation for a given patient.

[0028] The actions can be performed in the order shown (from top to bottom or numerically) or in other orders. Additional, different, or fewer actions can also be provided. For example, additional inputs (e.g., patient characteristics, clinical measurements, patient attributes, and / or genomics) can be provided for training.

[0029] Obtain ground truth data for training. Obtain the ground truth data by loading from or accessing memory. Alternatively, receive the ground truth data over a computer network.

[0030] For training for segmentation, the ground truth data is based on the segmentation of the imaging data. The training process relates the input feature vector to the ground truth segmentation. In the case of deep learning, the input feature vector is the imaging data with or without additional information. For other types of machine learning, the input feature vector is information derived from the imaging data, such as Haar wavelets, steerable features, or values ​​calculated from the Hessian matrix.

[0031] The ground truth segmentation can be different from the imaging data annotated with the object. In act 10, ground truth is generated using a distance function. The ground truth positions of the segmented object boundaries are used to generate ground truth segments using a distance function. Ground truth values ​​are provided in image space, thus providing values ​​at different locations in 2D or 3D. For each location, a distance function is applied to determine the value. The distribution of values ​​provides the segmentation or ground truth.

[0032] Any distance function can be used. In one embodiment, a signed distance function (SDF) is used. Positions on one side of the object or within the boundary are assigned one sign (e.g., positive), while positions on the other side of the object or outside the boundary are assigned another sign (e.g., negative). Any mapping of distance to magnitude can be used, such as a linear, nonlinear, or exponential mapping, where lower values ​​are used for positions further from the boundary.

[0033] The distance function can be decomposed into two outputs: a distance function and a level set, where internal pixels are labeled as -1, external pixels are labeled as 1, and contours are labeled as 0. The generator is trained to predict two outputs (e.g., distance and level set), which together define the segmented object. The discriminator can be trained on the two outputs together or separately to learn shape information during training. Other outputs or combinations of outputs can be used.

[0034] For each ground truth sample, corresponding imaging data is obtained as training data or sample. Image data is data representing an object or patient that can be used to generate an image. The image data can be scan data before being formatted for display, such as scan data in a scan format that is different from the display format. The image data can be red, green, and blue (RGB) data that is formatted for display but not displayed or previously displayed. For each ground truth value, a corresponding image data set is provided. Tens, hundreds, or thousands of samples and corresponding ground truth values ​​are collected.

[0035] In action 12, a machine (e.g., a processor, a computer, a server, or an image processor) performs machine training on the generator to output a segmentation. Any type of machine training and corresponding classifier can be trained. A machine training classifier is any one or more classifiers. Single-class or binary classifiers, sets of different classifiers, cascade classifiers, hierarchical classifiers, multi-class classifiers, model-based classifiers, machine learning-based classifiers, or combinations thereof can be used. Multi-class classifiers include CART, K nearest neighbors, neural networks (e.g., multilayer perceptrons), hybrid models, or other. Probabilistic boosting trees can be used. Error correction output codes (ECOCs) can be used. Deep learning or deep reinforcement learning can be used for detection and segmentation.

[0036] In one embodiment, deep learning is used to train a neural network. A deep adversarial architecture can be used to improve segmentation performance. A machine (e.g., an image processor) learns to identify the location of an object based on input. In an iterative process, training determines the values ​​of the filter kernels, node values, weights, and / or other learnable parameters of the generator that relates the input vector to the desired output (i.e., the ground truth).

[0037] The model is trained to segment any object. For example, segmenting organs, such as identifying the position of the liver relative to other tissues. As another example, identifying invasive objects, such as positioning stents or catheters. Landmarks (e.g., blood vessels in the kidney) and / or parts (e.g., valves, heart chambers, or vascular bifurcations) can be segmented. The model is trained using samples from a database, such as patient scans, and the ground truth of the segments from those scans.

[0038] Any machine learning generator network can be used, such as a fully convolutional neural network. For example, the generator network is an image-to-image convolutional deep learning network. As another example, an encoder-decoder pair can be used. In yet another example, a UNet with or without skip connections is used. Other neural networks with or without convolution can be used. The generator receives an image as input and learns to generate a synthetic image as output.

[0039] The generator encodes the imaging data into several independent latent variables and generates synthetic data by sampling the latent variables. In deep learning, the latent variables are learned by machine training. The generative network returns the prior log-likelihood and is implemented as a piecewise differentiable function, such as used in deep learning. For example, the generative network is a deep learning model using a restricted Boltzmann machine, a deep belief network, a neural autoregressive density estimator, a variational autoencoder, its extensions, or other deep learning methods for generative modeling. In one embodiment, the trained deep generative network is a deep neural network with a set of j convolutional layers and k fully connected layers, each layer followed by a set of nonlinear activation functions, a set of pooling layers for feature reduction, and a set of scale-up layers for image generation. There are usually two parts in this architecture, one part (e.g., the encoder) uses a pooling layer to convolve the learned filter with an increased abstraction, and the latter part (e.g., the decoder) uses a scale-up layer to convolve the learned filter with a reduced abstraction. Any abstraction (feature reduction) amount and corresponding scale-up can be used. Other layer arrangements can be used. Any number of any type of layers in any arrangement with more or fewer parts may be used.

[0040] The input imaging data can be one of a 2D slice, a 3D slab, a 3D volume, or a 4D volume over time. Any number of output channels can be provided. Each output channel represents a composite image.

[0041] To train the generator, the log-likelihood of the output is maximized. The generator encodes features representing position or magnitude (i.e., kernels in convolutional layers) by their location in the image. Since generative training is unsupervised, training does not require matched pairs of good and bad images, which are difficult to obtain on a large scale in medical settings.

[0042] Deep learning on the generator uses a discriminator network. Generative adversarial training is used. The discriminator network is a neural network trained to determine whether the output distance or segmentation is the ground truth (i.e., the training data) or a neural network synthetically created by the generator. Using adversarial training can increase the accuracy of the distances or segmentations generated by the generator. Using a second network (i.e., the discriminator network) to decide between the generated distances and the ground truth distances allows for improved results on the generator being trained.

[0043] Figure 2 An exemplary training arrangement and corresponding network are shown. Figure 2 In the example of , the training data uses ultrasound volumes 20 as input samples and the signed differences of positions in 3D as ground truth 23. "pred" and "GT" refer to predictions and ground truth, respectively. The generator 21 is shown as a UNet or encoder-decoder with skip connections and bottleneck connections.

[0044] The output from the generator 21 (i.e., the estimated distance in image space) is used to input into the discriminator network 28 for adversarial training, and is used to determine shape parameters in shape space via the manifold 25 for shape priors for training the generator 21.

[0045] The discriminator network 28 includes any layer architecture. In one example, the discriminator network has a downsampling architecture portion (e.g., encoder) of convolutional and pooling layers similar to the generator 21, but reaches one or more fully connected layers to discriminate between the ground truth 23 and the generated distance 22. The fully connected layer can be a soft max layer that outputs a binary indication, such as a label as generated / synthesized or ground truth.

[0046] Additional, different, and / or fewer layers or types of layers may be used for the generator 21 and / or the discriminator network 28. Skip connections (i.e., feeding features forward to non-adjacent layers), feedback, and / or max pooling exponents may be used.

[0047] The results from training the discriminator network 28 are used to further train the generator 21. An iterative process is used in which the two networks pass results to each other for training. The generator 21 uses the output of the discriminator network 28 as an indication of adequacy (i.e., the ability of the discriminator network 28 to tell the difference from the ground truth will be minimized). The discriminator network 28 uses the output of the generator 21 as input to learn to discriminate.

[0048] Once trained, the generator 21 is applied without the discriminator network 28 and / or without the manifold 25. The generator 21 outputs distances indicative of segments in response to an input of imaging data.

[0049] To train, optimize the values ​​of weights, connections, convolution kernels, or other learnable parameters. Any optimization, such as Adam, can be used. Training can be iterated over any number of epochs.

[0050] The optimization operates using one or more losses. The generator 21 is machine trained to output segments (e.g., the difference from the boundary) based on different losses. Figure 2 In the example, three types of losses are used - reconstruction (image space segmentation) loss (L recon ), adversarial loss (L Disc ) and shape loss (L b ). Additional, different, or fewer losses may be used. For example, not using the reconstruction loss and / or adversarial loss.

[0051] In optimization, the losses can be combined to optimize the generator 21. Any combination can be used, such as a weighted sum. The weights are assigned based on the expected or empirical importance between the losses.

[0052] Optimization of the shape GAN is done in shape space as well as image space (e.g., using the generator 21 and discriminator network 28 in the shape space provided by the manifold 25). In one embodiment, the fully convolutional generator 21 predicts values ​​by the position of a signed distance function (SDF), where the value of each voxel represents its shortest signed distance from the ground truth contour (e.g., the object boundary). The loss in image space uses these pixel-by-pixel or voxel-by-voxel distance values. The loss in shape space uses lower dimensional information derived from the estimated distance (e.g., manifold space). Other shape-based losses can be used, such as a loss term that directly penalizes deviations from the mean shape using a contour distance metric.

[0053] Any loss function can be used. For example, L1 or L2 loss can be used. Different loss functions can be used for different losses. In one embodiment, a smooth L1 loss is used for the reconstruction loss in image space and the shape loss in shape space. Huber loss can be used. Any loss optimized for shape space or a generator reconstruction loss can be used. Mean squared error (MSE) can be used for one or both losses. Smoothing L1 can help convergence in the optimization.

[0054] For shape loss, a lower dimensional shape space is used. The lower dimensional space is lower than the estimated image space. For example, thousands of distance values ​​are processed (e.g., one distance value for each voxel in 3D) to reduce the number of values. This shape space is created from a manifold or unsupervised learning. Manifold 25 or other shape space processes can include learnable parameters that are adjusted during machine learning. Alternatively, during training, the manifold 25 or other shape space process is set or immutable.

[0055] In one embodiment, the shape space is formed as eigenvalues. Principal component analysis (PCA) generates a feature matrix b of the eigenvalues. pred As a shape space. The shape loss is determined by the eigenvalues ​​as shape parameters. Figure 2 In the example of , the estimated signed distance 22 output by the generator 21 is vectorized. The distance is linearized. Other formats can be used. The vectorized distance 24 is processed by the manifold 25. For example, the eigenvectors of the matrix of all linearized SDFs of the training data are calculated by principal component analysis (PCA), and each data sample is transformed using this eigenmatrix (W) to obtain the low-dimensional eigenvalues ​​(b) 26 of the sample. The error or shape loss L b is the eigenvalue b within these eigenvalues ​​26 relative to the distance 23 from the ground truth GT27. The shape loss enables optimization in a lower dimensional shape space rather than a higher dimensional image space.

[0056] To optimize using the adversarial loss, the discriminator network 28 is trained using different input pairs. To distinguish between generated and actual distances, pairs of ground truth distance 23 and generator estimated distance 22 are input, as well as pairs of ground truth distance 23 and original imaging data 20. Alternatively, only ground truth distance 23 and predicted distance 22 are used. The binary output of the discriminator network 28 is used to determine the adversarial loss L Disc This adversarial loss uses data in image space as input and is therefore related to image space losses. The adversarial loss is computed on the SDF22, 23 and is therefore used as a constraint to maintain feasible shapes of the anatomical structure, allowing the prediction of complex anatomical structures.

[0057] Reconstruction loss L recon Is the loss in image space. The difference between the estimated distance 22 and the ground truth distance 23 on a pixel-by-pixel or voxel-by-voxel basis provides the loss, such as using a smooth L1 loss. Including a reconstruction loss in the optimization can speed up convergence.

[0058] Once trained, Generator 21 has learnable parameter values ​​based on various losses through multiple iterations using the training data. Because different losses are included in the optimization, the resulting Generator 21 differs from when different losses were used. As a result, Generator 21 operates differently and can provide different results than a differently trained Generator 21.

[0059] exist Figure 1 In act 14, the learned or trained generator is stored. A generator is a matrix or architecture with learned values ​​for learnable parameters (e.g., convolution kernels) and set values ​​for other parameters. The machine learning network is stored for application to a given input feature vector of an unseen patient.

[0060] The learned network or generator is stored in memory or other storage with the training data. For example, copies of the learned network are distributed to or on different medical scanners for use in a hospital or medical practice. As another example, copies are stored in memory on one or more servers for segmentation as a service or remote segmentation.

[0061] Figure 3 A method for segmentation in a medical imaging system is described. The segmentation uses a machine learning generative network to locate one or more objects as segments. One or more positions of the one or more objects are determined.

[0062] Figure 1The method of can be implemented in the order shown (e.g., from top to bottom or numbers) or in a different order. For example, action 34 can be performed after action 36 and / or as a part of action 38.

[0063] Additional, different or fewer actions may be performed. For example, in the case where data is provided from a memory or by transmission, action 30 may be omitted. As another example, actions 36 and / or 28 are not performed.

[0064] The method is implemented by a medical imaging system, such as any system for performing image processing based on medical scans. The medical imaging system can be a medical diagnostic imaging system, an examination station, a workstation, a computer, a picture archiving and communication system (PACS) station, a server, a mobile device, a combination thereof, or another image processor. For example, Figure 4 The system shown or described in embodies the method, but other systems may be used. A hardware processor of any type of system interacting with a memory (e.g., a PACS database or cloud storage), a display, and / or a medical scanner may perform these actions.

[0065] The actions can be performed automatically. A user causes a patient to be scanned or obtains scan data of the patient from a previous scan. The user can activate the process. Once activated, the object is segmented and the segmented image is output to a display or medical record. User input of the location of any anatomical structures in the scan data can be avoided. Due to the generative network used for segmentation, user correction of the segmentation is unlikely to be required. Some user input may be provided, such as for changing modeling parameter values, correcting detected locations, and / or confirming accuracy.

[0066] In act 30, a medical scanner scans a patient. The medical scanner generates imaging data representing the patient. The image or imaging data may be acquired by or within the medical scanner. Alternatively, the acquisition may be from a storage device or memory, such as a previously created data set from a picture archiving and communication system (PACS). The processor may extract the data from a medical records database.

[0067] Imaging data can be in any format. Although the terms image and imaging are used, the image or imaging data can be in a format prior to the actual display of the image. For example, medical imaging data can be a plurality of scalar values ​​representing different positions in a Cartesian or polar coordinate format that is different from the display format. As another example, a medical image can be a plurality of red, green, and blue (e.g., RGB) values ​​output to a display for generating an image in a display format. The medical image can be an image currently or previously displayed in a display or in another format. Imaging data is a data set that can be used for imaging, such as scan data or a generated image representing a patient.

[0068] Any type of medical imaging data and corresponding medical scanner can be used. In one embodiment, the imaging data is a CT image acquired using a computed tomography (CT) system. For example, a chest CT data set can be acquired by scanning the lungs. For CT, the raw data from the detector is reconstructed into a three-dimensional representation. As another example, magnetic resonance (MR) data representing a patient is acquired using an MR system. The data is acquired using an imaging sequence for scanning the patient. K-space data representing a region within the patient's body is acquired. Fourier analysis is performed to reconstruct the data from k-space into a three-dimensional object or image space. In yet another example, the data can be ultrasound data. A beamformer and transducer array acoustically scan the patient. Polar coordinate data is detected and beamformed into ultrasound data representing the patient.

[0069] The medical imaging data represents tissue, fluid and / or bone structure of a patient.In other embodiments, the medical images represent both function (such as perfusion) as well as structure.

[0070] Medical imaging data represents a 2D or 3D region of a patient. For example, medical imaging data may represent a region or slice of a patient as pixel values. As another example, medical imaging data may represent a volume or three-dimensional distribution of voxels. The three-dimensional representation may be formatted as a stack or multiple two-dimensional planes or slices. A value is provided for each of multiple locations distributed in two or three dimensions. Medical imaging data is acquired as one or more data frames. A data frame represents a scanned region at a given time or period. A data set may represent an area or volume over time, such as providing a 4D representation of a patient.

[0071] In act 32, the image processor segments the objects represented in the imaging data. Segmentation provides an estimate of the location of an object, such as a boundary or any portion of an object. In one embodiment, the segmentation indicates a probability of an object boundary, such as an estimate of the distance to the boundary, where the shortest distance indicates the boundary. Any segmentation that identifies the location of a given object or objects can be used. Segmentation distinguishes one object represented in the imaging data from another object.

[0072] To perform segmentation, the image processor applies a machine learning generative network to the imaging data. The machine learning generative model localizes the object. The image processor can use the output estimated by the machine learning generative network for filtering, thresholding, skeletonization, or further image processing to identify the object. Alternatively, the output estimate identifies the object. The distance identifies the object.

[0073] Application of the generative network results in the creation of a composite image. The composite image may be a spatial distribution of distances representing proximity to a boundary or outline of an object. In other embodiments, the composite image may be the same imaging data, but also include labels for the boundary or object. The generative network generates the labels. Alternatively, the generative network generates imaging data that includes the labels. The composite imaging data may have labels added to the scanned data or image, such as values ​​that highlight the object in a particular color.

[0074] The generative model can be positioned in two or three dimensions. Objects can be positioned over time in 4D imaging data.

[0075] To perform segmentation, imaging data from a patient scan is fed into a machine learning generative network. The network, such as a trained image-to-image network learned as a convolutional deep network, generates segmentations or object locations in response to the input. For example, a distribution of signed distances to boundaries per pixel or voxel is generated from the imaging data.

[0076] The machine learning generative network is adversarially trained using a discriminator and a discriminative loss. The discriminator is not used in the application. The machine learning generative network is trained using optimization in shape space. The estimated distance is converted into shape space, which is a lower dimensional space, using manifold or unsupervised learning. For example, the shape space is the feature matrix from the principal component analysis of the signed distance function. The shape space may not be used in the application. The optimization for training the generative network uses a reconstruction loss in the image space, a shape loss in the shape space, and an adversarial loss. Once trained, the generative network does not calculate losses. Instead, the generative network operates as trained by optimization to generate synthetic images from the input of imaging data, such as signed distances.

[0077] In act 34, an image processor generates an image and outputs the image on a display, such as a screen. Based on the segmentation, the image is an image of the object. Only the object is displayed. Alternatively, the object is highlighted and displayed along with other tissues or objects represented in the imaging data. For example, an image is generated from the imaging data. The object is highlighted in the image, such as by being colored, displayed with a different brightness, or annotated (labeled with a symbol or alphanumeric text).

[0078] The image can be an image of a 2D plane or region within the patient's body. The image can be a cross-section through the patient and / or object. Alternatively, the image is a rendering distributed three-dimensionally onto a 2D display, a 3D holographic display, an augmented reality display, or a virtual reality display.

[0079] In action 36, the image processor determines shape parameters of the shape space based on the distance function output by the machine learning generation network. In an application, the shape space is used. The estimated distance or imaging data for a given patient is vectorized and transformed into a lower dimensional shape space. The shape parameters (e.g., b) for a specific patient are pred ) is used as a special marker to give valuable information.

[0080] In action 38, the values ​​of the patient's shape parameters are used to aid in diagnosis. The aid is output to a medical record and / or display. For example, annotations are added to the segmented image. The annotations indicate the class or other information about the object determined from the values ​​of the shape parameters. Recommendations may be output, such as for further investigation, confidence levels in the segmentation, and / or other post-processing to be applied.

[0081] The value of the shape parameter can be compared to a distribution or threshold to aid diagnosis. Characteristics of the shape parameter can be calculated and used to aid diagnosis.

[0082] In one embodiment, b pred A "normal" distribution is defined by the training set. The mean distribution of the shape parameter values ​​is determined from the training data. The distribution of values ​​for different classes can be determined. The patient's distribution can be compared to one or more normal or reference distributions to determine class membership or similarity.

[0083] Objects in a patient can be classified based on a comparison of the patient's shape parameters with a reference distribution of the parameters. For example, b pred can be considered as an anomaly. Therefore, the value of the shape parameter can be used to perform dynamic anomaly detection. As another example, a sub-equal segment can be detected by the outlier value of its shape parameter.

[0084] In another embodiment, the comparison is used to indicate confidence in the segmentation. pred The function of distance from b pred The mean of the distribution derives a measure of confidence in the quality of the segmentation. Lower confidences associated with distances exceeding one or more threshold shape parameter values ​​can be used to automatically invoke expert review or highlight lower confidences to a radiologist.

[0085] The values ​​of the shape parameters can be used to perform other classifications. For example, a database of reference hearts and corresponding shape parameter distributions for adult hearts, pediatric hearts, pathological conditions, or another category can be used to create reference values ​​for the shape parameters for different categories. The feature values ​​for a particular patient are used to classify the patient into these various categories. The classification can then be used to select a machine learning generated network or other post-processing algorithm based on the category (i.e., different networks for different categories).

[0086] Abnormal patient values ​​b pred Can be used to enhance training. Improve the training set by finding abnormal patients and including their imaging data as samples for refinement or retraining. These values ​​can be used to find similar cases (used as functional and anatomical similarity measures).

[0087] This comparison can be achieved using a distance metric, such as the average or weighted average difference between shape parameter values. Alternatively, a classifier, such as a naive Bayes classifier, a support vector machine, or a decision tree, is trained on the shape parameters to explicitly identify specific abnormalities, pathologies, demographic groups, or categories. The trained model outputs class membership or other information in response to input of the shape parameter values.

[0088] Figure 4 A system 40 for object positioning is shown. The system 40 implements Figure 1 、 Figure 3 In one embodiment, the system 40 is configured to apply a machine learning generative network. Given input imaging data, the system 40 uses the generative network to generate a spatial distribution of locations for objects represented in the imaging data. Although the system 40 is described below in the context of applying a previously learned generative network, the system 40 can also be used to machine train the generative network using a shape loss and / or an adversarial loss with or without reconstruction.

[0089] System 40 includes an image processor 44, a memory 46, a display 48, and a medical scanner 42. Image processor 44, memory 46, and display 48 are shown separate from medical scanner 42, such as as part of a workstation, computer, or server. In alternative embodiments, image processor 44, memory 46, and / or display 48 are part of medical scanner 42. In still other embodiments, system 40 does not include medical scanner 42. Additional, different, or fewer components may be used.

[0090] Medical scanner 42 is a CT, MR, ultrasound, camera, or other scanner for scanning a patient's lungs. In one embodiment, medical scanner 42 is a CT system having an x-ray source and detector mounted on a movable gantry for three-dimensional scanning of the patient's upper torso. An image processor 44 or other processor in medical scanner 42 performs computed tomography to determine scalar values ​​of tissue response to x-rays in three dimensions. In another embodiment, medical scanner 42 is an ultrasound scanner. Acoustic energy is generated by a transducer based on electrical signals generated by a transmit beamformer. A receive beamformer generates samples in response to echoes from different locations within the patient's body.

[0091] Scanning provides scan data representing the interior of the patient's body. The medical scanner 42 is configured, via user settings or presets, to scan the patient, thereby generating scan data representing all or at least a portion of an object within the patient's body. The imaging data may be in any format, such as scalar values ​​distributed in a regularly or evenly dispersed 3D grid (i.e., uniform voxels).

[0092] Memory 46 is a buffer, cache, RAM, removable media, hard drive, magnetic, optical, database, or other memory now known or later developed. Memory 46 is a single device or a group of two or more devices. Memory 46 is shown as being associated with or part of image processor 44, but may also be external to or remote from other components of system 40. For example, memory 46 is a PACS database that stores scan data from medical scanner 42.

[0093] The memory 46 stores scan or image data, machine learning generated or image to image network 45, and / or information used to segment objects in image processing. For training, training data (ie, input feature vectors and ground truth) is stored in the memory 46.

[0094] Memory 46 is additionally or alternatively a non-transitory computer-readable storage medium having processing instructions. Memory 46 stores data representing instructions executable by the programmable image processor 44. Instructions for implementing the processes, methods, and / or techniques discussed herein are provided on a computer-readable storage medium or memory, such as a cache, buffer, RAM, removable media, hard drive, or other computer-readable storage medium. The machine-learning generated or image-to-image network 45 may be stored as part of the instructions for segmentation. Computer-readable storage media include various types of volatile and non-volatile storage media. The functions, actions, or tasks illustrated in the figures or described herein are performed in response to one or more sets of instructions stored in or on a computer-readable storage medium. The functions, actions, or tasks are independent of a particular type of instruction set, storage medium, processor, or processing strategy and may be performed by software, hardware, integrated circuits, firmware, microcode, etc., operating alone or in combination. Similarly, processing strategies may also include multiprocessing, multitasking, parallel processing, etc.

[0095] In one embodiment, the instructions are stored on a removable media device for reading by a local or remote system. In other embodiments, the instructions are stored at a remote location for transmission over a computer network or over a telephone line. In yet other embodiments, the instructions are stored within a given computer, CPU, GPU, or system.

[0096] The image processor 44 is a general-purpose processor, a digital signal processor, a three-dimensional data processor, a graphics processing unit, an application-specific integrated circuit, a field-programmable gate array, a digital circuit, an analog circuit, an artificial intelligence processor, a combination thereof, or other now known or later developed device for applying machine learning image-to-image network 45 and / or segmentation. The image processor 44 is a single device, a plurality of devices, or a network. For more than one device, parallel or sequential partitioning of processing can be used. The different devices comprising the image processor 44 can perform different functions, such as one processor performing segmentation by applying image-to-image network 85 and another processor performing classification based on the value of a shape parameter. In one embodiment, the image processor 44 is a control processor or other processor of a medical diagnostic imaging system (e.g., a medical scanner 42). The image processor 44 is a hardware device that is configured by or operates according to stored instructions, design (e.g., an application-specific integrated circuit), firmware, or hardware to perform the various actions described herein.

[0097] The image processor 44 is configured to utilize a machine learning image-to-image network to locate objects represented in image data. Image data or features derived therefrom are input to the image network, which in response generates segmentations. Object locations, such as object boundaries, are generated by the image-to-image network based on prior training of the network.

[0098] The machine-learned image-to-image network 45 is trained using shape priors. For example, features are formed based on the distances generated by the image-to-image network during training. The machine-learned image-to-image network 45 trains the features using a smooth L1 loss. Other losses may also be used in training optimization, such as a reconstruction loss in image space and / or an adversarial loss.

[0099] Based on past training, the machine learning image-to-image network 45 is configured to output pixel-wise or voxel-wise distances to object boundaries. The loss in training is distance-based.

[0100] The image processor 44 can be configured to determine shape space parameters from the image data. For example, the generated image, such as an estimated distance or another segmentation, is linearized (e.g., vectorized) and transformed, such as using principal component analysis, other manifolds, or unsupervised models. The transformation determines the value of the shape parameters.

[0101] The image processor 44 can be configured to use the values ​​of the shape space parameters for a given patient to assist in diagnosis, prognosis, and / or treatment. For example, the image processor 44 detects characteristics of the object from the shape space parameters, identifies similar cases from the shape space parameters, and / or identifies a confidence level for the location of the object from the shape space parameters.

[0102] The image processor 44 can generate an image. The image-to-image network output (e.g., segmentation) is used to generate the image. The image is an image of the patient with a highlight or relative indication of different parts (such as an object). Any segmented image or any image representing the object can be generated. In addition to an image showing only the detected tissue (i.e., a highlight or graphic for the segmentation), the image also provides additional visualization of the object.

[0103] The display 48 is a CRT, LCD, plasma, projector, printer, or other output device for displaying images or other outputs from the image processor 44 and / or the medical scanner 42. The display 48 is configured to display an image of the subject by storing an image plane memory for the created image. Any image that is a function of the positioned subject can be displayed.

[0104] Although the present invention has been described above with reference to various embodiments, it will be appreciated that many changes and modifications may be made without departing from the scope of the invention. It is therefore intended that the foregoing detailed description be considered illustrative rather than restrictive, and that it is the following claims, including all equivalents, that are intended to define the spirit and scope of the invention.

Claims

1. A method for segmentation in a medical imaging system, the method comprising: scanning a patient by a medical scanner, said scanning providing first imaging data representative of the patient; segmenting, by an image processor, an object represented in the first imaging data, the segmenting using a machine learning generative network applied to the first imaging data, the machine learning generative network having been adversarially trained using optimization in shape space; as well as outputting an image of the object on a display based on the segmentation; Wherein the segmentation comprises segmenting with a machine learning generative network that has been trained to output pixel-wise or voxel-wise distances to the boundaries of the object, and wherein the shape information is in a lower dimensional space.

2. The method according to claim 1, wherein Scanning includes scanning using the medical scanner as an ultrasound system.

3. The method according to claim 1, wherein Segmentation involves applying a machine learning generative network as an image-to-image convolutional deep learning network.

4. The method according to claim 1, wherein Segmenting includes segmenting using a machine learning generative network that has been trained, wherein the shape space includes a lower dimensional space from a manifold or unsupervised learning.

5. The method according to claim 4, wherein Segmenting includes segmenting using a machine learning generative network that has been trained, wherein the shape space includes a feature matrix from a principal component analysis of a signed distance function, wherein the signed distance function outputs a signed distance from a boundary of the object in pixels or voxels.

6. The method according to claim 1, wherein Segmenting includes segmenting using a trained machine learning generative network, wherein the optimization includes a first loss in image space and a second loss in shape space.

7. The method according to claim 1, wherein Segmentation involves segmenting with a machine learning generative network trained to output pixel-wise or voxel-wise signed distances to the boundaries of an object.

8. The method according to claim 1, wherein Segmentation includes segmenting using a machine learning generative network that has been trained using an adversarial loss with a discriminator using input from a voxel-wise or pixel-wise distance from the boundary of the object.

9. The method according to claim 1, further comprising: determining shape parameters in shape space based on a distance function output by the machine learning generation network, wherein the distance function outputs a distance in pixels or voxels from a boundary of the object; as well as The object is classified based on a comparison of the shape parameter to the parameter distribution.

10. The method according to claim 1, further comprising: determining shape parameters in shape space based on a distance function output by a machine learning generative network, wherein the distance function outputs a distance in pixels or voxels from a boundary of the object; as well as The confidence in the segmentation is indicated based on a comparison of the shape parameter to the parameter distribution.

11. A system for object positioning, the system comprising: a medical scanner configured to scan a patient, said scanning resulting in image data representative of the patient; an image processor configured to localize objects represented in the image data using a machine learning image-to-image network that has been trained using a shape prior; as well as a display configured to display an image based on the located object; wherein the machine learning image-to-image network is configured to output pixel-wise or voxel-wise distances to the boundary of an object, and wherein the shape information is in a lower dimensional space.

12. The system according to claim 11, wherein The machine learning image-to-image network is trained using an adversarial loss on distance.

13. The system according to claim 11, wherein: The shape prior comprises eigenvalues, and wherein the machine learning image-to-image network is trained using a smooth L1 loss on the eigenvalues.

14. The system according to claim 11, wherein: The machine learning image-to-image network is trained using a first loss in image space and a second loss in shape space.

15. The system according to claim 11, wherein The image processor is configured to determine shape parameters from the image data and to detect characteristics of the object from the shape parameters.

16. The system according to claim 11, wherein The image processor is configured to determine shape parameters from the image data and to identify similar cases from the shape parameters.

17. The system according to claim 11, wherein: The image processor is configured to determine shape parameters from the image data and to identify a confidence level for the position of the object from the shape parameters.

18. A method for machine training a model for segmentation, the method comprising: machine training the generative network to output segmentations based on shape loss and adversarial loss to provide a machine learning generative network; as well as Storing the machine learning generated network; The method includes training a machine learning generative network to output pixel-wise or voxel-wise distances to the boundary of an object, and wherein the shape information is in a lower dimensional space.

19. The method of claim 18, further comprising generating a ground truth value as a per-voxel or pixel distance to an object boundary, wherein the shape loss is a smooth L1 loss based on shape in distance and the adversarial loss is for distance in image space.

20. The method according to claim 18, wherein Machine training involves machine training the generative network as a fully convolutional neural network based on a shape loss, an adversarial loss, and an image loss, wherein the shape loss is based on linearized feature values ​​from the segments output by the generative network.

Images