Data Augmentation for Machine Learning Methods

Abstract

To train a machine learning model using anatomical knowledge, a mechanism is provided for generating / synthesizing realistic training data. An anatomical model can be obtained. Information from annotated training data input (i.e., "ground truth" information) can be used to model anatomical variations from the obtained model in a training data population. This anatomical model can then be modified to incorporate several random factors to generate one or more augmented models of realistic anatomical structures. The augmented anatomical structures are then transferred from the model domain to a data input domain to generate new data inputs or data inputs for training the machine learning model. This latter process can be accomplished in various ways, for example, using GANs such as CycleGAN and labeled images or deformation vector fields.

Description

[Technical field]

[0001] The present invention relates to the field of generating augmented data to assist in the training of machine learning methods. [Background technology]

[0002] There is a growing trend to analyze medical data using machine learning methods to predict or identify features of the medical data. Such methods aid clinicians in evaluating patients, for example to improve their diagnosis or treatment. Exemplary machine learning methods can perform tasks such as image segmentation or landmark detection. Summary of the Invention [Problem to be solved by the invention]

[0003] However, machine learning methods require a large amount of consistently annotated training data. Indeed, the robustness / performance of these machine learning methods depends heavily on the availability of a sufficiently large and accurate training dataset. However, obtaining such a dataset is difficult and / or expensive, especially in the medical domain. This is mainly because medical experts, who may have limited time available, must be paid to annotate the training data. For some applications, it may also be necessary to involve multiple experts to reach a consensus in order to ensure that the training data is sufficiently accurate.

[0004] There is also a continuing problem of specificity of training data, since the statistics of the data used to train a machine learning method should match the statistics of the population that will later be analyzed by the machine learning method, which adds to the difficulty of obtaining a suitable data set for training a machine learning method for a particular application.

[0005] Data augmentation has become increasingly interesting in order to automatically generate new entries of training data. A typical method of data augmentation involves modifying existing training data entries to synthesize new training entries. When the training data input includes images, data augmentation may involve rotating, skewing and / or flipping (parts of) the images, or introducing artifacts into the images (e.g., adding noise or color anomalies).

[0006] Particularly in the medical field, there is a desire to improve data augmentation for machine learning methods, which would allow for more accurate machine learning and generate information to aid clinicians in their work, diagnosis or treatment. [Means for solving the problem]

[0007] The invention is defined by the claims.

[0008] According to an example consistent with an aspect of the present invention, a computer-implemented method for generating annotated augmented data elements for training a machine learning algorithm that processes anatomical data elements to generate one or more annotations is provided.

[0009] The computer-implemented method comprises the steps of obtaining one or more annotated data elements, each annotated data element providing information about an anatomical structure of the same type, obtaining an anatomical model of said type of anatomical structure, transforming the anatomical model using one or more transformation techniques, and generating one or more annotated augmented data elements based on the transformed anatomical model, each annotated augmented data element providing predictive information of the anatomical structure, wherein obtaining the anatomical model and / or transforming the anatomical model are based on at least one or more annotated data elements. The present invention enables the generation of additive / synthetic / augmented annotated data elements of anatomical structures for training machine learning algorithms.

[0010] A machine learning algorithm is any algorithm that processes suitable data elements, e.g., classifies anatomical structures associated with the data elements, predicts properties of the anatomical structures, segments the data elements, identifies properties of the data elements, etc., to generate annotations.

[0011] An annotated data element is an example of a data element that has been suitably annotated (e.g., labeled or otherwise associated with the data) with information that corresponds to the output of a machine learning algorithm. For example, if the machine learning algorithm classifies a data element, an annotated data element would include the example data element and its classification (as an annotation).

[0012] The invention proposes to generate and / or modify an anatomical model of a certain type of anatomical structure (e.g. heart, liver, lungs, etc.) based on one or more annotated data elements. In particular, the anatomical model is deformed or modified using deformation techniques. The deformed anatomical model is then used to generate annotated augmented data elements, which are artificial instances of one or more annotated data elements.

[0013] Each annotated data element provides information about the same type of annotated structure (e.g., liver, spleen, heart, etc.), but may also provide information about different instances of the same type of annotated structure. For example, a first annotated data sample may provide information about the heart of a first individual / subject, and a second annotated data sample may provide information about the heart of a second individual / subject.

[0014] It will be apparent to one skilled in the art that an anatomical model is a digital or virtual anatomical model of the anatomical structure of interest.

[0015] In some embodiments, each annotated data element comprises a set of one or more annotated medical images, and each annotated augmented data element comprises a set of one or more annotated augmented medical images.

[0016] Optionally, the anatomical model is a three-dimensional anatomical model of the type of anatomical structure.

[0017] Obtaining the anatomical model may include generating the anatomical model using the one or more annotated data elements.

[0018] In some embodiments, the obtained anatomical model comprises first data representing a general shape of the anatomical structure informed by the one or more annotated data elements and second data representing variations from the general shape of the anatomical structure, and the step of deforming the anatomical model comprises perturbing a portion of the second data and applying at least the perturbed portion of the second data to the first data to generate a deformed anatomical model.

[0019] The general shape may for example be an average shape of the anatomical structure derived from one or more annotated data elements, i.e. one or more annotated data elements may be processed to derive an average shape of the anatomical structure (represented by the annotated data elements).

[0020] The first data may, for example, include a matrix of vertices (i.e., a mesh) that together form or serve as an average or mean shape of the anatomical structure for which the annotated data samples provide information about the anatomical structure, i.e., the "general shape" of the anatomical structure. The second data may, for example, provide information about how each particular anatomical structure (for which each particular annotated data sample provides information) differs or deviates from the general shape.

[0021] In at least one embodiment, the first data represents an average / mean shape of an anatomical structure when one or more annotated data elements provide information of the anatomical structure, represents a maximum shape of the anatomical structure when one or more annotated data elements provide information of the anatomical structure, or represents a minimum shape of the anatomical structure when one or more annotated data elements provide information of the anatomical structure.

[0022] The first data may include a matrix of vertices and / or polygon information (such as triangle information) that defines the general shape of an anatomical structure of that type.

[0023] In some embodiments, the anatomical model is deformable according to a plurality of deformation techniques, and the step of deforming the anatomical model includes deforming the anatomical model using a subset of the plurality of deformation techniques.

[0024] The step of generating one or more annotated augmented data elements may include the steps of generating one or more augmented data elements using the deformed anatomical model, and annotating each augmented data element with information derived from the deformed anatomical model to generate the annotated augmented data elements.

[0025] Generating the one or more annotated augmented data elements can include processing the deformed anatomical model using a generative adversarial network to generate the one or more annotated augmented data elements.

[0026] As an alternative, the step of generating one or more annotated augmented data elements may include the steps of calculating a vector deformation field using the deformed anatomical model and applying the vector deformation field to at least one of the annotated data elements to generate the annotated augmented data element.

[0027] The step of generating one or more annotated augmented data elements may include the steps of processing the deformed anatomical structure to calculate a vector deformation field, and processing one or more of the annotated data elements using the vector deformation field to generate one or more annotated augmented data elements.

[0028] The method may further include receiving user input and controlling the number of one or more annotated data elements generated in response to the user input and / or controlling the number or type of deformation techniques used in deforming the anatomical model to generate a deformed anatomical model in response to the user input.

[0029] Further, a computer-implemented method for training a machine learning algorithm using annotated data elements is proposed, comprising the steps of obtaining one or more annotated data elements, generating annotated augmented data elements by performing the above-described method one or more times, and training the machine learning algorithm using the one or more annotated data elements and the annotated augmented data elements.

[0030] Any suitable machine learning algorithm for obtaining useful clinical data can be used. For example, the machine learning algorithm may be for labeling or segmenting medical images, identifying the occurrence of cysts or nodules, predicting the presence of a particular condition, disease or diagnosis, etc. In general, the machine learning algorithm may be any algorithm that processes data elements to generate clinically significant and / or useful annotations.

[0031] According to an aspect of the present invention there is provided a computer program product comprising computer program code means which, when executed on a computing device having a processing system, causes the processing system to perform all of the steps of the method according to any of the claims. The computer program may be stored on a non-transitory computer readable medium.

[0032] According to an example according to an aspect of the present invention, a processing system is provided for generating annotated augmented data elements for training a machine learning algorithm that processes anatomical data elements to generate one or more annotations.

[0033] The processing system is adapted to perform the steps of obtaining one or more annotated data elements, each annotated data element providing information about an anatomical structure of the same type; obtaining an anatomical model of the type of anatomical structure; deforming the anatomical model using one or more deformation techniques; and generating one or more annotated augmented data elements based on the deformed anatomical model, each annotated augmented data element providing predicted information of an anatomical structure, wherein the steps of obtaining the anatomical model and / or the steps of deforming the anatomical model are based on at least one or more annotated data elements.

[0034] The processing system and / or computer program may be suitably adapted to carry out any of the methods described herein.

[0035] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0036] For a better understanding of the present invention and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief description of the drawings]

[0037] [Figure 1] 4 is a flow chart illustrating a method according to an embodiment. [Diagram 2] FIG. 13 shows a schematic of a technique for generating augmented data elements from a deformed anatomical model. [Diagram 3] FIG. 1 illustrates a system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] The present invention will now be described with reference to the drawings.

[0039] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems and methods, are for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the devices, systems and methods of the present invention will become better understood from the following description, appended claims and accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to denote the same or similar parts.

[0040] The present invention provides a mechanism to create / synthesize realistic training data for training machine learning models with anatomical knowledge. An anatomical model can be obtained. Information from annotated training data inputs (i.e., "ground truth" information) can be used to model anatomical variations from the obtained model in a training data population. This anatomical model can then be modified to incorporate some random coefficients to generate one or more augmented models of realistic anatomical structures. The augmented anatomical structures are then transferred from the model domain to the data input domain, thereby generating new data inputs or data inputs for training the machine learning model. This latter process can be accomplished in various ways, for example, using a GAN such as CycleGAN and label images, or deformation vector fields.

[0041] The embodiments are based on the realization that realistic data entries for training machine learning models used in the medical field can be created or synthesized by modifying anatomical models.

[0042] Reference is made herein to an "annotated data element." An annotated data element is any data element (e.g., an image or a set of images) that also includes annotations or additional information. The annotations (or additional information) are the target or desired output of a machine learning method for which the annotated data element is intended to be used as training data.

[0043] As an example, if one wishes to provide a machine learning method for automatically segmenting ultrasound images, each annotated data element comprises an example ultrasound image and (as "annotations") segmentation information for that example ultrasound image.

[0044] As another example, if one wishes to provide a machine learning method for automatically detecting the presence or absence of pleural effusion from x-ray images of a patient's lungs, each annotated data element would include an example x-ray image (the patient's lungs (present)) and (as an "annotation") an indicator of the presence or absence of pleural effusion.

[0045] The exact format and content of the annotated data elements are not necessary to appreciate the concepts underlying the invention or to realize the advantages of the invention, and indeed, those skilled in the art will readily appreciate and understand different examples of data elements (e.g., different types of images, physiological data measurements, body data measurements, etc.) and appropriate annotations (e.g., segmentation information, labeling information, identified conditions, identification of features (e.g., cysts or nodules), etc.).

[0046] FIG. 1 illustrates a method 100 according to an embodiment of the invention.

[0047] The method 100 comprises a step 101 of obtaining one or more annotated data elements, each annotated data element providing information about the same type of anatomical structure (e.g., heart, lungs, chest, liver, kidneys, brain, etc.) In other words, step 101 comprises obtaining training data.

[0048] The one or more annotated data elements may be obtained from a database, storage device, or other memory module. In other examples, the one or more annotated data elements are obtained directly from one or more measurement tools, such as an ultrasound machine, an MRI machine, a patient monitor, etc.

[0049] The one or more annotated data elements may comprise, for example, an annotated medical image, although other medical data is contemplated (e.g., physiological measurements, physical measurements, etc.) Medical images include any 2D or 3D medical image, such as a CT, MR or 3D ultrasound image.

[0050] The annotated data elements may provide information about different instances or examples of the same type of anatomical structure, such as the hearts of different subjects, the same heart at different times (e.g., the heart dilated or systolic), the same heart at different angles, or any combination of the above.

[0051] In some examples, each annotated data element includes a set of one or more medical images that provide information about the same instance of an anatomical structure.

[0052] The method 100 further comprises the step 102 of obtaining an (anatomical) model of the anatomical structure.

[0053] In some embodiments, step 102 may include obtaining a generic model of the anatomical structure from a database, memory or other storage module. Thus, an existing or pre-defined model may be used, for example a model representing a global population.

[0054] In other embodiments, a model of the anatomical structure is constructed from one or more annotated data elements. As one example, the annotated data elements can be processed to determine the average or mean shape of the (different instances of) the anatomical structure represented by the annotated data elements. In another example, the annotated data elements can be processed to determine the maximum or minimum shape of the (different instances of) the anatomical structure represented by the annotated data elements.

[0055] A suitable method for constructing anatomical models from medical images can be found in O. Ecabert et al., "Automatic Model-Based Segmentation of the Heart in CT Images," IEEE Trans. Med. Imaging, vol. 27, no. 9, pp. 1189-1101, Sep. 2008, and is suitable for use in the present invention. However, those skilled in the art will recognize that other methods for generating models of anatomical structures from medical data can be used.

[0056] In either embodiment, the model of the anatomical structure may comprise, for example, first data representing the general shape of the anatomical structure whose information is provided by one or more annotated data elements, and (optionally) second data representing variations from the first data. The general shape may be derived from the one or more annotated data elements, or may be a pre-determined model.

[0057] A suitable example of the first data is a matrix including vertices (and / or polygon information) of the average shape of an anatomical structure, where the training data provides information about the anatomical structure. TIFF0007670057000001.tif76. In another example, the matrix TIFF0007670057000002.tif76 is replaced by a matrix where one or more annotated data elements provide information of the anatomical structure, providing the max / max or min / min shape of the anatomical structure.

[0058] matrix TIFF0007670057000003.tif76 may, in some embodiments, contain information defining a mesh that represents the shape of the structure.

[0059] This can be computed by generating, for each instance of an anatomical structure (e.g., each annotated data element), a model of the anatomical structure instance, e.g., a matrix that provides information about the vertices of a mesh that represents the shape of the anatomical structure instance. For the jth instance of an anatomical structure, this is the matrix x j The generated matrices can then be averaged (or max / min of the generated matrices) to generate a generic model of the anatomical structure (of that type) that is informed by the one or more annotated data elements.

[0060] Preferably, the mean / max / min shapes are the mean / max / min shapes of any instance (or combination of instances) of the anatomical structure for which the annotated data elements provide information of the anatomical structure.

[0061] matrix TIFF0007670057000004.tif76 may meet the following criteria: TIFF0007670057000005.tif923

[0062] The anatomical model may further include information regarding the annotations provided by the annotated data elements. For example, the annotations may provide identification of different features of the anatomical model, which may be retained in the anatomical model. This information may be used later when generating the annotated extended data elements.

[0063] The method further comprises the step 103 of deforming the anatomical model using one or more deformation techniques.

[0064] At least one of steps 102 and 103 is performed using the one or more annotated data elements, such that the anatomical model is generated and / or deformed using the one or more annotated data elements.

[0065] Step 103 can be implemented in many ways, a first scenario of which is described below.

[0066] In this first scenario, the anatomical model has the first data mentioned above, and second data that represents variations from the general shape of the anatomical structure.

[0067] In some examples, the second data represents a global / population variation, for example, a variation from a general shape (eg, average / max / min) according to a known or global variation.

[0068] However, in a preferred embodiment, the second data is generated using one or more annotated data elements to represent variations from a general shape of different ones of the one or more annotated data elements, and the second data is derived from the one or more annotated data elements. The second data may, for example, indicate how the shape of each instance of an anatomical structure for which the annotated data elements provide information of the anatomical structure varies from the general shape.

[0069] A combination of one or more annotations and global / collective variations can be used to define the second data, for example the second data can have variations from a general shape derived from one or more annotated data elements, such variations being modified based on the global variations.

[0070] In some examples, when the second data is derived from one or more annotated data elements, the second data may be, for each instance of an anatomical structure described in the annotated data element, calculated as (matrix It is possible to describe the shape transformations required to convert a generic shape (represented by TIFF0007670057000006.tif76) into the shape of an instance of an anatomical structure.

[0071] For example, following the concept of principle component analysis, the shape of the j-th instance of an anatomical structure (representing one of the annotated data elements) is the matrix x j The required shape deformation can be expressed as the following term Pb j Is expressed by: TIFF0007670057000007.tif13132

[0072] P is Matrix given by TIFF0007670057000008.tif27152 tif76 (representing the general shape). In equation (2), N represents the number of different instances of an anatomical structure for which the annotated data elements provide information of the anatomical structure.

[0073] b j describes the shape transformation with respect to a particular principal axis, such that for the j-th instance of an anatomical structure: TIFF0007670057000010.tif17134

[0074] The variable b can represent the second data, so b j represents the second data element. In particular, b j is the matrix x j The matrix (average shape) is given by The parameters used to transform TIFF0007670057000011.tif76 can be expressed as the matrix x j represents an instance of an anatomical model depicted in one or more annotated data elements.

[0075] In another example, instead, this section b j Let x represent a global or preset variation from a generic shape, which reflects, for example, known variations of anatomical models according to a global population. Thus, b can be derived from global information.

[0076] The step of determining the variation from the general shape can be performed in either step 102 or step 103, each of which can be performed using annotated data elements of global information (e.g., from the literature) according to different embodiments.

[0077] Step 103 may include perturbing or modifying a portion of the second data and generating a deformed anatomical model by applying at least the perturbed portion of the second data to the first data.

[0078] This is the case, for example, for synthetic shape transformations. Shape transformation b to give TIFF0007670057000012.tif106 j This can be implemented by slightly modifying the second data, i.e., by transforming the second data. j One or more deformation techniques can be used to modify the augmented or deformed anatomical model (matrix The resulting image (represented as TIFF0007670057000013.tif85) can be generated using the following formula: TIFF0007670057000014.tif14124

[0079] Shape transformation b j The deformation of can be performed using a preset deformation, for example according to known global variations in the shape or structure of the anatomical model, which can be derived from global or population data. j The deformation of, for example, b j and b j+1 Interpolation between b j and b j+2 Interpolation between b and j This can be done by interpolating between two instances of

[0080] A matrix (e.g., representing the jth distinct instances of the second data) TIFF0007670057000015.tif85 2 or more compound shape transformations TIFF0007670057000016.tif106 can be applied to generate reasonable variations in the shape of the anatomical model.

[0081] In this manner, a deformed anatomical model can be generated, where the deformed anatomical model represents an artificial or synthetic instance of the anatomical structure (i.e., not derived directly from actual measurements).

[0082] The method 100 further comprises generating 104 one or more annotated augmented data elements based on the deformed anatomical model, each annotated augmented data element providing predicted information of an anatomical structure.

[0083] Many approaches are possible for converting the deformed anatomical model into annotated augmented data elements. The following examples are given in the context of each data element containing a medical image of an instance of an anatomical structure, i.e., an example of conversion from the model domain to the image domain. However, a person skilled in the art will understand that the proposed approach can be adapted to data elements of any form.

[0084] In a first example, annotated augmented data elements can be generated using Cycle-GAN (or other suitable generative adversarial networks, GANs). Cycle-GANs can be trained to transfer style, for example, between labels or gray plateau images (which may be generated from a deformed / augmented anatomical model) and regular medical images, such as ultrasound images.

[0085] This process is illustrated diagrammatically in Figure 2, which shows a deformed anatomical model 210. In this scenario, the annotated data elements include segmented and labeled ultrasound images.

[0086] In step 201, the deformed anatomical model is used to generate a labeled image 220. The labeled image includes labels or annotations derived from the deformed anatomical model (i.e., the anatomical model identifies elements of the synthetic instance of the anatomical structure).

[0087] Then, in step 202, the labeled image 220 is processed using Cycle-GAN to generate an augmented image 230 (forming augmented data elements). Cycle-GAN effectively imparts the style of a typical medical image, such as an ultrasound image, to the labeled image 220. The augmented image can be annotated with the labels of the labeled image to generate annotated augmented data elements.

[0088] A more complete description of the process of using Cycle-GAN to transfer style from one image to another (used here to transfer the style of ultrasound images to generated label images) can be found in J.-Y. Zhu, T. Park, P. Isola, and AA Efros, "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks," ArXiv170310593 Cs, Mar. 2017.

[0089] One advantage of Cycle-GANs is that their training does not require matched image pairs, and therefore it is much easier to obtain large amounts of training data (for Cycle-GANs) compared to generating annotated data elements (to train machine learning methods).

[0090] In the second example, the extended anatomical model (matrix Using the vector deformation field technique, a realistic image could be generated using the vector deformation field F, which is represented by the TIFF0007670057000017.tif85. In particular, the vector deformation field F can be calculated using the following formula: TIFF0007670057000018.tif12132 where γ is the expansion coefficient and x j is a matrix that represents a model of the jth instance of the anatomical structure. The vector deformation field F can then be applied to the data elements that correspond to the jth instance of the anatomical structure. This can be performed using an interpolation scheme such as linear or B-spline interpolation.

[0091] An extension data element may share the annotations of the corresponding annotated data element from which the annotated data element is generated.

[0092] In particular, the deformation of the anatomical model is performed with reference to a particular instance of an anatomical structure for which a particular annotated data element provides information, and thus the deformed model can be associated with annotations of this particular annotated data element, which annotations can be propagated to the annotated augmented data elements generated from the deformed model.

[0093] 1, steps 103 and 104 may be repeated one or more times to generate one or more respective annotated augmented data elements. In this manner, multiple augmented data elements may be generated for training a machine learning method.

[0094] This process may be performed by repeating step 103 multiple times, e.g., applying different deformations to the shape deformation b before performing step 104 for each deformed anatomical model. j and / or apply different / same transformations to the same instance of shape transformation b j This can be implemented by applying the method of the present invention to different instances of the anatomical model to generate a number of different deformed anatomical models.

[0095] In another embodiment, steps 103 and 104 may be implemented by applying different deformations to the shape deformation b j and / or apply different / same transformations to the same instances of shape transformation b j The method may be performed iteratively in succession by applying the method to different instances of

[0096] In this way, multiple annotated augmented data elements can be generated using an anatomical model and known variations (real-world examples) of the anatomical model by modifying the known variations and synthesizing new, yet plausible, variations of the anatomical model.

[0097] As mentioned before, the present invention is intended to be used as a pre-stage before training machine learning methods such as (convolutional) neural networks to generate large training data. In particular, by using the present invention, an additional set of pseudo ground truth data (i.e., annotated augmented / synthetic data elements) is generated. Using the original dataset and the augmented data for subsequent network training makes the final machine learning model (trained using such data) more robust or appropriate (e.g., when the number of datasets available initially is limited).

[0098] The machine learning method may perform, for example, image segmentation or landmark detection on 3D medical images such as 3D cardiac ultrasound. Those skilled in the art will understand appropriate annotations and formats for such training data elements to train the machine learning method.

[0099] In some embodiments, a user or individual may have options to control the amount of additional data that is generated, for example, the number of times steps 103 and 104 of method 100 are repeated.

[0100] This can be achieved by receiving user input and controlling the number of one or more annotated data elements that are generated in response to the user input, which can be by controlling how many variances are applied to the annotated data elements, such as by iterating over Equation (3) or a similar approach.

[0101] In some examples, the user input may alternatively or additionally control the number or type of deformation techniques used in deforming the anatomical model to generate the deformed anatomical model. Thus, there may be additional parameters that can be used to control the number of deformation techniques used to modify the anatomical model.

[0102] The annotated augmented data elements may be stored in a memory or database for later use in training machine learning methods, which may be performed in optional step 105.

[0103] Furthermore, a corresponding method for training a machine learning algorithm with the annotated data elements is proposed, comprising the steps of obtaining one or more annotated data elements, generating annotated augmented data elements by performing the above-mentioned method one or more times, and training a machine learning algorithm using the one or more annotated data elements and the annotated augmented data elements.

[0104] Those skilled in the art will appreciate that a machine learning algorithm is any self-training algorithm that processes input data to generate or predict output data, where the input data includes data elements (including augmented data elements) and the output data includes information corresponding to annotations (e.g., labels of medical images, predicted medical diagnoses, measurements of physical features, etc.).

[0105] Suitable machine learning algorithms for use in the present invention will be apparent to those skilled in the art. Examples of suitable machine learning algorithms include decision tree algorithms and artificial neural networks. Other machine learning algorithms such as logistic regression, support vector machines or naive Bayes models are suitable alternatives.

[0106] The structure of an artificial neural network (or simply a neural network) is inspired by the human brain. A neural network is composed of layers, each layer containing multiple neurons. Each neuron contains a mathematical operation. In particular, each neuron can contain different weighted combinations of a single type of transformation (e.g., the same type of transformation, sigmoid, etc., but with different weightings). In the process of processing the input data, each neuron performs a mathematical operation on the input data, resulting in a numerical output, and the output of each layer in the neural network is fed successively to the next layer. The last layer provides the output.

[0107] Methods for training machine learning algorithms are well known. Typically, such methods include obtaining training data, including training input data entries and corresponding training output data entries. An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and the corresponding training output data entry is used to modify the machine learning algorithm. This process can be repeated until the error converges, and the predicted output data entry is sufficiently similar (e.g., ±1%) to the training output data entry. This is commonly known as a supervised learning technique.

[0108] For example, if a machine learning algorithm is formed from a neural network, the mathematical operations of each neuron can be modified until the error converges. Known methods for modifying neural networks include gradient descent, backpropagation algorithms, etc.

[0109] The training input data entries correspond to the data elements and the extended data elements, and the training output data entries correspond to the corresponding annotations of these data elements.

[0110] A person skilled in the art could easily develop a processing system to perform any of the methods described herein, and each step of the flow chart thus represents a different operation performed by a processing system and may be performed by a respective module of said processing system.

[0111] Thus, embodiments may utilize a processing system. The processing system may be implemented in a variety of ways using software and / or hardware to perform the various functions required. The processor is one example of a processing system that uses one or more microprocessors that may be programmed using software (e.g., microcode) to perform the required functions. However, the processing system may be implemented with or without a processor, and may also be implemented as a combination of dedicated hardware to perform some functions and a processor (one or more programmed microprocessors and associated circuitry) to perform other functions.

[0112] Examples of processing system components that may be used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0113] In various implementations, the processor or processing system may be associated with one or more storage media, e.g., volatile and non-volatile computer memory, e.g., RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that perform the required functions when executed on the one or more processors and / or processing systems. The various storage media may be incorporated within the processor or processing system, or may be transportable such that one or more programs stored on the storage media are loaded into the processor or processing system.

[0114] A suitable example of a processing system 300 is shown in Figure 3. The processing system forms one aspect of an augmented data generation system 30 according to one embodiment of the present invention.

[0115] The processing system 300 is adapted to obtain one or more annotated data elements, each annotated data element providing information about the same type of anatomical structure. The annotated data elements can be obtained from a database, the augmented data generation system 30, or from another memory module.

[0116] Furthermore, the processing system 300 is adapted to obtain an anatomical model of said type of anatomical structure, which can be performed using any of the methods described above.

[0117] The processing system 300 is adapted to deform the anatomical model using one or more deformation techniques, which can be performed using any of the methods described above.

[0118] The acquisition and / or deformation of the anatomical model is performed using one or more annotated data elements.

[0119] The processing system 300 is further adapted to generate one or more annotated augmented data elements based on the deformed anatomical model, each annotated augmented data element providing predicted information of an anatomical structure.

[0120] The processing system 300 may be configured to perform any of the steps of any of the methods described above, for example, for training a machine learning algorithm.

[0121] It is understood that the disclosed methods are preferably computer-implemented methods. As such, the concept of a computer program having code means for performing any of the described methods when the program is executed on a processing system, e.g., a computer, is also proposed. Thus, different parts, lines or blocks of code of a computer program according to an embodiment can be executed by a processing system or a computer to perform any of the methods described herein. In alternative implementations, the functions depicted in the block diagrams or flow charts can be performed out of the order depicted in the figures. For example, two blocks shown in succession can in fact be executed substantially simultaneously, or the blocks can be executed in the reverse order depending on the functionality involved.

[0122] Variations to the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. Where a computer program is described above, the computer program can be stored or distributed on a suitable medium, for example an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, for example via the Internet or other wired or wireless telecommunication systems. When the phrase "adapted to" is used in the claims or the description, the phrase "adapted to" is intended to be similar to the phrase "configured to". Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. A computer-implemented method for generating annotated augmented data elements for training a machine learning algorithm that processes anatomical data elements to generate one or more annotations, comprising: obtaining one or more annotated data elements, each annotated data element providing information about the same type of anatomical structure, each annotated data element comprising a set of one or more annotated medical images; obtaining an anatomical model of said type of anatomical structure, said anatomical model being a three-dimensional anatomical model of said type of anatomical structure; deforming the anatomical model using one or more deformation techniques; generating one or more annotated augmented data elements based on the deformed anatomical model, each annotated augmented data element providing predicted information of the anatomical structure, each annotated augmented data element comprising a set of one or more annotated augmented medical images; having the steps of obtaining the anatomical model and transforming the anatomical model are based on at least the one or more annotated data elements; and wherein the step of generating the one or more annotated augmented data elements comprises processing the deformed anatomical model using a generative adversarial network to generate the one or more annotated augmented data elements. A computer-implemented method.

2. The computer-implemented method of claim 1 , wherein the step of obtaining the anatomical model comprises generating the anatomical model using at least the one or more annotated data elements.

3. the obtained anatomical model comprises first data representing a general shape of an anatomical structure for which information of the anatomical structure is provided by the one or more annotated data elements, and second data representing a variation of the anatomical structure from the general shape; 3. The computer-implemented method of claim 2, wherein the step of deforming the anatomical model comprises generating a deformed anatomical model by perturbing a portion of the second data and applying at least the perturbed portion of the second data to the first data.

4. The first data is the mean / average shape of an anatomical structure for which the one or more annotated data elements provide information of the anatomical structure; a maximum shape of an anatomical structure for which the one or more annotated data elements provide information of the anatomical structure; or a minimum shape of an anatomical structure for which the one or more annotated data elements provide information of the anatomical structure; The computer-implemented method of claim 3 , wherein:

5. 5. A computer-implemented method according to claim 3 or 4, wherein the first data comprises a matrix of vertex and / or polygon information defining the general shape of an anatomical structure of said type.

6. 6. The computer-implemented method of claim 1, wherein the anatomical model is deformable according to a plurality of deformation techniques, and wherein the step of deforming the anatomical model comprises deforming the anatomical model using a subset of the plurality of deformation techniques.

7. said step of generating said one or more annotated extended data elements further comprising: generating one or more augmented data elements using the deformed anatomical model; annotating each augmented data element with information derived from the deformed anatomical model to generate annotated augmented data elements; 7. A computer implemented method according to claim 1, comprising:

8. said step of generating said one or more annotated extended data elements further comprising: processing the deformed anatomical structure to calculate a vector deformation field; processing one or more of the annotated data elements with the vector deformation field to generate the one or more annotated augmented data elements; 8. A computer implemented method according to claim 1, comprising:

9. receiving a user input; controlling the number of one or more annotated data elements generated in response to the user input, and / or controlling the number or type of deformation techniques used in deforming the anatomical model to generate the deformed anatomical model in response to the user input; The computer-implemented method of claim 1 , further comprising:

10. A computer-implemented method for training a machine learning algorithm using annotated data elements. obtaining one or more annotated data elements; - generating annotated extended data elements by performing one or more times a method according to any one of claims 1 to 9; training the machine learning algorithm using the one or more annotated data elements and the annotated augmented data elements; 16. A computer-implemented method comprising:

11. A computer program comprising computer program code which, when executed on a computing device having a processing system, causes the processing system to carry out each of the steps of the method according to any one of claims 1 to 10.

12. 1. A processing system for generating annotated augmented data elements for training a machine learning algorithm that processes anatomical data elements to generate one or more annotations, the processing system comprising: obtaining one or more annotated data elements, each annotated data element providing information about a same type of anatomical structure, each annotated data element comprising a set of one or more annotated medical images; obtaining an anatomical model of said type of anatomical structure, said anatomical model being a three-dimensional anatomical model of said type of anatomical structure; deforming the anatomical model using one or more deformation techniques; generating one or more annotated augmented data elements based on the deformed anatomical model, each annotated augmented data element providing predicted information of the anatomical structure, each annotated augmented data element comprising a set of one or more annotated augmented medical images; configured to run the steps of obtaining the anatomical model and transforming the anatomical model are based on at least the one or more annotated data elements; and wherein the step of generating the one or more annotated augmented data elements comprises processing the deformed anatomical model using a generative adversarial network to generate the one or more annotated augmented data elements. Processing system.

Images