Data-aware augmentation for deep learning

Abstract

Described herein is a computer-implemented method for generating an augmented image. The method comprises receiving an input image, which may be a clinical image, the input image comprising one or mor

Description

This disclosure relates to data augmentation, and in particular to the use of data-dependent augmentation in generating augmented clinical data. Background The potential of utilizing deep learning techniques for clinical data can be limited by patient privacy requirements and imbalances in available data, as a substantial quantity and variability of clinical data is required for deep learning techniques to be utilised successfully. Limiting the amount of data available for a deep learning model can easily cause overfitting, where the resulting model performs well with training data but not with unseen data. Poor prediction performance can also be caused by biased data collection methods. For example, clinical data regarding brain tumours is typically gathered at a later stage, once a tumour has become visible on a scan (MRI, CT, etc), or once a patient has become symptomatic and is diagnosed accordingly. In order to build a deep learning model which can predict the evolution of an early-stage tumour, clinical data is needed at an earlier stage to capture the early stage tumour, as well as at a later stage, to capture the evolution. However, there may be far less clinical data available at the earlier stages of a tumour, and more available at the later stages, leading to an imbalance in the data available. It is desirable to collect more data at earlier stages in the tumour cycle, but this is not always possible due to, for example, a prevalence of late diagnoses where tumours go unnoticed until a later stage in the tumour cycle. Data augmentation can be used to increase the span of training data for better generalization, by synthesizing synthetic data from training data, and using the additional synthetic data to train the deep learning model. Common data augmentation techniques can include affine transformation such as scaling, rotation, flipping and elastic deformation, and changes in voxel intensities. Typically, an augmentation with the same parameters is applied to each data sample in a set of training data, without regard for any differences between the data samples. However, this can limit the distribution of the added synthetic training data, and thus the generalisation and robustness of the model trained using the data. The present disclosure seeks to address these and other disadvantages encountered in the prior art by providing an improved method of augmenting data. Summary According to an aspect, there is provided a computer-implemented method for generating an augmented image. The method comprises receiving an input image, the input image comprising one or more regions of interest, and transforming a portion of the input image using a data function. The data function receives at least one region of interest of the image as an input, and generates a transformed image portion as an output based on the received region of interest. An augmented image is generated using the transformed image portion.. The data function may further comprise extracting an image portion corresponding to an identified region of interest, and transforming the extracted portion. The method can further comprise adding the transformed extracted image portion to the input image. Adding the transformed extracted image portion to the input image can comprise overlaying the transformed extracted image portion onto the input image. Transforming a portion of the input image may comprise at least one of a geometric transformation, a colour transformation, and / or a kernel filter. Where transforming a portion of the input image comprises a geometric transformation, the geometric transformation can comprise one or more of: rotation, scaling, flipping and / or elastic transformation. The method may further comprise segmenting the input image to identify a region of interest. The at least one region of interest received by the data function may comprise a tumour. The tumour may be a late-stage cancer tumour. The transformed image portion may comprise a transformed tumour, optionally wherein the transformed tumour is representative of an early stage cancer tumour. The method may comprise segmenting the input image to identify a late-stage cancer tumour, wherein the data function further comprises: extracting an image portion corresponding to the segmented late-stage tumour; transforming the extracted image portion; and adding the transformed extracted portion to the input image. Transforming the extracted image portion may comprise reducing the size of the extracted image portion. The input image can comprise one or more MR images or one or more CT images. The input image can comprise one or more MR images, and the augmented image can comprise one or more CT images. According to a second aspect, there is provided a system for generating an augmented image, the system comprising a memory storing machine-executable instructions and a data function as disclosed herein, a processor for controlling the system, wherein execution of the machineexecutable instructions causes the processor to perform the methods disclosed herein. According to a third aspect, a computer-readable medium is provided, the computer-readable medium containing instructions which, when executed by the processor, cause the processor to perform the methods disclosed herein. Figures Specific embodiments are now described, by way of example only, with reference to the drawings, in which: Figures la-lf depict the application of common augmentation techniques to a data set; Figure 2a depicts a flowchart of a standard augmentation operator applied to training data; Figure 2b depicts a flowchart of a data-dependent augmentation operator applied to training data; Figure 3 depicts an example using radiomics to adjust a data augmentation hyperparameter differently for each data sample; Figure 4 depicts a method of generating an augmented image; Figure 5 depicts an exemplary method of generating an augmented magnetic resonance image; Figure 6 illustrates a block diagram of one implementation of a computing device 600 within which a set of instructions, for causing the computing device to perform any one or more of the methodologies discussed herein, may be executed. Detailed Description In overview, and without limitation, the application discloses a method for generating augmented data using a data-dependent augmentation scheme. As the augmentation scheme is data-dependent, each data sample augmented using the scheme may undergo a different augmentation. The augmented data is then added to the training data, and used to train a model, increasing the span of the distribution of the training data and hence enabling improved performance of models trained on the augmented data. Advantageously, the presently disclosed method allows for a reduction in data imbalances caused by biased collection methods, particularly in regard to medical data, allowing for improved model training. As noted above, clinical data regarding, for example, brain tumours is typically gathered at a later stage, once a tumour has become visible on a scan (MRI, CT, etc), or once a patient has become symptomatic and is diagnosed accordingly. There is thus typically a data imbalance in the clinical data, as more clinical data relating to the later stages of tumour growth and development is available compared to the earlier stages of tumour growth and development. The claimed method reduces the data imbalance by allowing for the creation of augmented data which is representative of these earlier stages of tumour growth. The augmented data can then be used in addition to the available clinical data to train a predictive model to predict, for example, the evolution of a tumour. Figures la-Id depict the application of common augmentation techniques to a data set. Common data augmentation techniques include affine transformations such as scaling, rotation, flipping and elastic transformations. In Figure la, an "original" MR image of a brain, shown in the axial plane can be seen. An "original" MR image refers to an MR image taken of a patient which has not been augmented or edited in any way, i.e. an MR image in the form in which it was acquired. A tumour (110) can be seen in the upper left of the brain shown in Figure la, along with a number of other visible features (120). Figure lb shows a first augmented version of the image shown in Figure la. In particular, Figure lb shows the image in Figure la after it has been flipped. As can be seen in the figure, the tumour (112) can now be seen in the upper right of the brain, and the other visible features (122) are now on the left side of the brain. Figure lc shows a second augmented version of the image shown in Figure la. In particular, Figure lc shows the image in Figure la after it has undergone a rotation. It can be seen that the tumour (114) has rotated with the image, along with the other visible features (124). Figure Id shows a third augmented version of the image shown in Figure la. In particular, Figure Id shows a version of Figure la wherein the image has been scaled. In other words, Figure la has undergone a scaling operation, and is now resized as shown on a on Figure Id. The scaling operation has been applied to the image as a whole, and thus all features of the image (116, 126) remain in the same proportion to each other as shown in Figure la. Augmentation techniques such as those shown in Figures la-Id can be used to create augmented training data but may not provide training data of the required level of generalisation to make a deep learning model suitably robust. Figure le shows an "original" MR image of a brain shown in the axial plane, wherein a tumour (118) can be seen in the top left of the brain. Figure If shows an "original" MR image of a different brain, again shown in the axial plane, wherein a tumour (128) of different size and composition can be seen in the top right of the brain. If, for example, Figure le was not included in training data for a voxelwise classification model trained to identify the precise location of tumours in brain scans of this nature, and Figure IF was included in the training data for the model, then it may be difficult for the model to accurately anticipate and identify the tumour present in the image of Figure le. Figure 2a depicts a flowchart of a standard augmentation operator applied to training data. As can be seen in the figure, the augmentation operator acts on the training data to create augmented training data prior to the training of a model. Such a standard augmentation operator applies an augmentation with constant parameters to the training data. In other words, the same augmentation operation is applied to each data sample in the training data. For example, to generate an example augmented output % , a set of augmentation functions {Ti, T2, .... Tn} with constant parameters {0i, 0 2,.... 0 n} are applied to an input x as follows: = 0k^for 1 <k <n, where x^ = x and x = x^. Constant parameters {¢1, 0 2,.... 0 n} are typically treated as hyperparameters and require fine-tuning for better performance. These constant parameters are applied to each sample of data in the data set, thus each sample of data is augmented via the same mechanism using this augmentation operator. For example, where an augmentation operator such as that defined above includes only a single scaling transformation (Ts), the augmentation operator can be represented as follows: x = Ts(x; 0S), where x is the input data, and x is the augmented data. In an example, the same scaling transformation (Ts) is applied to each data sample, and as such each data sample undergoes the same augmentation. For example, if the scaling transformation (Ts) comprised doubling the size of a data sample to which it was applied, then each data sample to which the transformation (Ts) was applied would have its size doubled. In other words, the same augmentation operation is applied in to each data sample, regardless of any characteristics of, or information contained within, the data samples themselves. As a result, a model trained using data which has been augmented in this manner may not generalize well to unseen data, or at the very least may not generalize to unseen data in an optimal manner. In other examples, the scaling transformation (Ts) can be a scaling transformation function which samples scaling factors based on a distribution. Where the scaling transformation is such a function, not all samples will undergo the same scaling transformation, and instead they will be scaled by the same probably distributed factors. This approach is not ideal when all data samples are scaled between, for example. 0.5 and 2 (i.e. between half and double), as larger tumours can be shrunk more aggressively, while smaller tumours should not be shrunk further or scaled up as much. In contrast, Figure 2b depicts a flowchart of a data-dependent augmentation scheme applied to training data. As can be seen, the augmentation scheme is data-dependent, and thus the augmentation operator applied to the data may be different for each data sample in the training data. The augmented data is then added to the training data, and used to train a model, increasing the span of training data and allowing for improved model performance. A data-dependent augmentation scheme can be defined as follows: x^ = Tk(x(k U0k)k ^for 1 <k <n, where x^ = x, x = x^ , and where U0 is a data function. The inclusion of a data function, such as a parameterised data function, in the augmentation operator allows for data information from the input data sample to be considered when applying the augmentation functions {Ti, T2,.... Tn}. In other words, the introduction of a data function to the augmentation scheme allows for data-specific augmentations to be applied to training data, instead of the same sets of parameters for all data samples. This allows for training data to be created which increases the span of the training data, and thus allows for deep learning models trained using the training data to better generalise and perform better on unseen data. There are a number of ways in which the data function U0 can be defined in order to indirectly estimate and alter the data distribution for training. For example, the data function U0 can progress with each iteration as in curriculum learning, or the data function U0 can be learned, for example using a machine learning model. A set of predefined features can determine the augmentation parameters of data function U0. For example, the set of predefined features can include a set of well-defined features or representations that describe data variability. In some implementations, engineered features may be used to augment the data. In such implementations, features of interest (i.e. predefined features) are extracted from the data, and augmented using the data function. A feature extraction function can be used to extract a feature of interest (R), and the data function U0 can be defined as follows: U0 = U0 (R(x,yJ) for a feature of interest R(x,y). In other words, a feature extraction function can be used to extract one or more features of interest from a data sample, and those one or more features can then be augmented using the one or more augmentation functions. Thus, an augmented data sample is generated which differs from the initial data sample only in that the one or more features of interest have been augmented. As different data samples can have different features of interest, utilising such a feature extraction function allows for data-dependent augmentations to be applied to different data samples, and as such allows for an increased span of training data to be generated. In yet other implementations, one or more features of interest can serve as a guide for data augmentation. That is, an augmentation applied to a particular feature in a data sample may be guided by one or more other features of interest in that data sample. For example, for tumour segmentation, a volume of a tumour in a data sample can guide the scaling range for an augmentation comprising a scaling transformation for that data sample. Alternatively, the distribution of tumour volumes within the data set can guide the scaling range for a set of data samples. That is to say, an augmentation applied to a particular data sample may be guided by one or more features of interest in other data samples. For example, a scaling factor applied to a data set comprising a tumour may be guided by the sizes of other tumours present in other data samples in the data set. Another example is that of synthetic CT, wherein it is desired to calculate a dose distribution based on an MR image. Synthesizing CT from MR allows for the density of body tissues to be identified given HU values. Images acquired from people with more fat tissue may require an augmentation which comprises less elastic deformation when compared to images from people with less fat tissue, as fat tissue is more deformable than muscle tissue. Therefore, the original dataset of images may have less variation for people with less fat tissue to train the synthetic model. To address this, body fat ratio can be used to guide the data augmentation operator for elastic deformation, allowing MR and CT images of people with lower body fat ratios to undergo more aggressive elastic deformation. In some implementations, the performance of a model, such as a deep learning model, can be used to guide data augmentation. In other words, information regarding the performance of a model trained using a given data set can also be used to adjust augmentation hyperparameters used to generate augmented data samples. For example, data augmentation may be guided based on the accuracy of a model (i.e. the number of correct predictions made by the model divided by the total number of predictions made by the model). Where a model is performing well, robust data augmentation can be employed in order to test the limits of the model. In such implementations, the model being trained may be performing well at identifying one or more features of interest in data samples in a data set. As such, the data samples may be augmented to generate augmented samples which are less readily identifiable for the model. For example, where a model can readily classify and / or distinguish an abnormal structure in a data sample, such as a tumour, due to the abnormal structure being marked by a distinct boundary structure, that data sample can be augmented to generate an augmented data sample with a tumour with a less-defined boundary structure. This augmentation can be achieved by applying an augmentation comprising a blurring effect to at least part of the data sample. By augmenting a plurality of data samples in the training set in this manner and adding these augmented data samples into the training set, a more robust model can be trained and / or the limits of model performance can be tested. Where a model is not performing well, a gentler data augmentation can be employed to aid learning and facilitate convergence of the model. In such implementations, the model being trained may not be performing well at identifying one or more features of interest in data samples in a data set. As such, the data samples may be augmented to generate augmented samples which are more readily identifiable for the model. For example, where a model is unable to classify and / or distinguish an abnormal structure in a data sample, such as a tumour, due to a lack of clarity associated with a boundary of the abnormal structure, that data sample can be augmented to generate a data sample comprising an abnormal structure with a more defined boundary. By augmenting a plurality of data samples in the training set in this manner and adding these augmented data samples into the training set, a better-performing model can be trained. In implementations where the performance of a model guides augmentation, a performancefeedback loop can be utilised, wherein augmentation hyperparameters are altered based on model performance during training of the model. This performance-feedback loop can be utilised in combination with the feature-dependent augmentation approach of other implementations disclosed herein. For example, data samples comprising one or more features of interest may at first undergo the same or a similar augmentation, but as the performance of the model being trained improves, a subset of the features of interest may undergo a different augmentation based at least in part on one or more characteristics of the one or more features of interest. For example, a more aggressive augmentation may be applied to larger tumours when the performance of a model improves. In this example, tumours of varying volume may initially share the same scaling range, but as the model's performance improves, the scaling range for larger tumours may be expanded. In yet other implementations, mixed-sample augmentation can be utilised, wherein two data samples are combined to generate an augmented data sample. For example, two input images may be blended together, in order to form an augmented image. In a mixed sample augmentation approach, a region of interest is identified and a first data sample from the training data set is selected, the first data sample being within the region of interest. A second data sample from the training set is then selected, and the first and second data samples are combined to form an augmented data sample. The data samples can be combined, for example, by taking the pixel-wise average of two images. Where mixed sample augmentation has taken place using a first image comprising a tumour and a second image which does not comprise a tumour, the deep learning model should predict an equal probability to the classes for the augmented image. In a preferred embodiment for mixed-sample augmentation, a first data sample from the training set is an image comprising a small tumour which is combined with a second data sample from the training set to generate an augmented image. An augmented image comprising a small tumour is thus generated. By performing a data-dependent augmentation on only the data samples which comprise one or more features of interest, the data samples which do not comprise any features of interest are treated differently in the training data. For example, a predefined feature, or feature of interest, in a set of MR images can be the presence of a tumour. For each image in the set of MR images, a feature extraction function can be utilised to extract a portion of the MR image which comprises a tumour, if such a portion exists. The augmentation functions can then be applied to the portion of the image comprising the tumour. As it is improbable that any two images in a set of MR images will comprise tumours which have identical characteristics, the feature extraction function will extract a tumour with different characteristics for each image comprising a tumour (for example, tumours of different locations, sizes, densities, etc). Thus, each data sample comprising a feature of interest will undergo a different augmentation due to the differences in the feature extracted from each respective data sample, providing data-dependent augmentation of the respective MR images. In another example, a predefined feature, or feature of interest, in a set of MR images can be the presence of a late-stage cancer tumour. A "late-stage" tumour is defined here as a tumour which is at a later stage in its growth and development. In other words, a "late-stage" tumour may be a large tumour and may be a tumour related to a Stage 2, 3 or 4 cancer. An "early stage" tumour is defined as a tumour which is at an earlier stage in its growth and development. In other words, an "early-stage tumour" may be a small tumour and may be a tumour related to a Stage 1 cancer. For each image in the set of MR images, a feature extraction function can be utilised to extract a portion of the MR image which comprises a late-stage tumour, if such a portion exists. The augmentation functions can then be applied to the portion of the image comprising the late-stage tumour. As above, each data sample comprising a feature of interest will undergo a different augmentation, providing data-dependent augmentation of the respective MR images. Performing the data-dependent augmentation on only MR images in the set which comprise a late-stage tumour allows, for example, for MR images which already comprise an early-stage tumour to be preserved. A more specific example is shown in figure 3, where radiomics (feature extraction from medical images) is utilised to extract features of interest from training data comprising medical images, in order to perform data-dependent augmentation on the medical images to create synthetic training data. For each medical image in the training set, an identified feature of interest can be extracted from the image and associated data information can be used to adjust the hyperparameters of one or more augmentation operators or functions. Where the one or more augmentation operators include a data function, the hyperparameters of the data function may be adjusted according to the data information gleaned from the extracted feature of interest. The medical image can then be augmented to generate an augmented medical image, which is then added to the training data set. Information regarding the performance of a model trained using the data can also be used to adjust the augmentation hyperparameters, in some implementations, as discussed further above. Figure 4 depicts a method 400 of generating an augmented image. Such a method includes performing a data-dependent augmentation in order to generate augmented data. Such augmented data can be added to a training data set used to train a deep learning model, enabling the deep learning model to better generalise to unseen data, as explained further above. The method is suitable for being performed, for example, by system 600 described further below. At step 410, an input image is received, the input image comprising one or more regions of interest. The input image may be received by the system 600. In further implementations, the input image may be an input MR image and may be received by MR image processing circuitry 602. Where the input image is an input MR image, the input MR image may be received directly from an MRI acquisition device. In implementations where the input image comprises an input MR image, the input MR image is based on MRI data acquired during an MR imaging session. Such an MR image may be a 3D or 2D image, and may be oriented in the axial plane, the sagittal plane, or the coronal plane. The input M R image comprises imaging data associated with an anatomical region of interest, organ, or a volume of interest of a patient. In some implementations, the input MR image may be of a patient, and in particular, may be of at least part of the brain of a patient. In such implementations, the input MR image may be an image of at least part of a brain which comprises one or more tumours. For example, the input MR image may be an image of the brain of a patient which comprises a cancerous tumour and may be a late-stage cancerous tumour. For example, the input MR image may be similar to the image in Figure la. Alternatively, the input image may comprise, for example, CT data, or data from another imaging modality. In implementations wherein the input image comprises CT data, the input image may be based on CT data acquired during a CT imaging session. Alternatively, the input image may be based on synthetic CT data. The input image may comprise CT imaging data associated with an anatomical region of interest, organ, or a volume of interest of a patient. At step 420, the input image is optionally segmented to identify a region of interest. The image may be segmented using standard segmentation techniques such as threshold segmentation, edge detection segmentation, clustering-based image segmentation, Mask R-CNN, etc. In some implementations, the input image may be segmented, at least in part, using ground truth label data in order to identify a region of interest. In other words, the input image may comprise associated ground truth label data, which may be used to identify a region of interest of the image. The segmented region of interest may comprise an anatomical region of interest, an organ, or a volume of interest of a patient. In implementations where the input MR image is of a patient, and in particular at least part of the brain of a patient, a region of interest may comprise at least part of a tumour. In further implementations, the region of interest may comprise at least part of a cancerous tumour, optionally wherein the tumour is a late-stage cancer tumour. In other words, the region of interest comprises a pixel or group of pixels representing the tumour. In some implementations, such as those described in the paragraph above, the input MR image is segmented to identify a late-stage cancer tumour in the MR image. Alternatively, representation learning techniques such as principal component analysis or autoencoders may be used to reduce the dimensionality of the input image, to extract one or more features of interest and / or to identify a region of interest. In such implementations, a region of interest is a region in the feature space. For example, the Gaussian distribution is commonly used to model data distribution, including distributions of tumour size. Typically, there is ample data for tumours of a "normal" size, but very limited data for tumours which are at extreme ends of the size distribution (i.e. very small or very large tumours). Plotting this distribution can help to identify a region of interest, which represents quantiles with minimal data points within the feature space. In practice, this region can be obtained by choosing a subset of eigenvectors (linked to top eigenvalues) from the covariance matrix of the feature data, and then sampling data points from the region of interest. Dimensionality reduction techniques can also be used to identify data samples that are less abundant in the training data. By identifying the features of interest that contribute to the rarity of certain data samples, additional data can be synthesized that closely resembles those samples. At step 430, a portion of the input image is transformed using a data function. Such a data function may form part of a data-dependent augmentation scheme, as is described with respect to figure 2b. In other words, a portion of the input image undergoes an augmentation operation, the augmentation operation comprising the data function. Implementations of such a data function are described with reference to Figure 2b above. As noted above with respect to Figure 2b, implementing a data-dependent augmentation scheme, such as an augmentation operator including a data function, allows for data information from the input sample to be considered when augmenting the input sample. In particular, transforming a portion of the input image using a data function allows for data-specific augmentations to be applied to training data, instead of, for example, an augmentation with the same set of parameters being applied to all images in a training data set. The data function receives at least one region of interest of the image as an input. The at least one region of interest of the image may comprise the region of interest identified via segmentation or representation learning in step 420 of the present method. The data function then generates a transformed image portion as an output, based on the received region of interest. The data function may extract the at least one region of interest of the image. In other words, the data function may extract an image portion corresponding to the at least one region of interest of the image. For example, the data function may comprise a feature extraction function, such as that outlined with respect to figure 2b. The feature extraction function may be utilised to extract one or more predefined features, the one or more predefined features forming the at least one region of interest of the image. For example, in implementations where the input image comprises a latestage cancer tumour, the feature extraction function may be utilised to extract a region of interest comprising the tumour from the image. The extracted image portion is then transformed using the data function. Transforming a portion of the input image may comprise one or more of a geometric transformation, a colour transformation, and / or a kernel filter. In some implementations wherein the transformation comprises a geometric transformation, the transformation may comprise one or more of a rotation, scaling, flipping and / or an elastic transformation. The transformation is applied to at least part of the input image using the data function. Transforming a portion of the input image may comprise more than one transformation operation. For example, an extracted image portion may undergo a combination of one or more geometric transformations, and one or more colour transformations to generate the transformed image portion. In particular, the transformation is applied to the region of interest extracted from the input image. Such a region of interest may comprise an anatomical region of interest, an organ or a volume of interest of a patient, as outlined further above. Thus, transforming a portion of the input image may comprise applying a transformation to the region of interest identified in and extracted from the input image. That is to say, transforming the portion of the input image may comprise transforming a region of interest, such as a tumour, by applying one or more of a geometric transformation, a colour transformation, and / or a kernel transformation. Where that transformation is a geometric transformation, the region of interest may undergo a rotation, scaling, flipping, and / or elastic transformation. In some implementations, the transformation is applied to a patch of the input image, the patch comprising, for example, a tumour and an area of surrounding tissue. In other words, the region of interest extracted from the input image may comprise a tumour and an area of tissue surrounding the tumour. Where the transformation is applied to a patch of the input image, the patch may be transformed such that the area or volume covered by the patch remains the same, but the scaling of any features within the patch are modified. For example, where a patch comprises a tumour and surrounding healthy tissue, the patch may be transformed such that the tumour has undergone a reduction in size, and the area of healthy tissue increased, but the volume or area of the patch has remained the same. In some implementations, the transformation is uniformly applied to the patch. For example, for a scaling operation the scaled patch may be smaller than the original patch but large enough to replace or cover up the original, unsealed region of interest. In such implementations, the patch may be configured to be of a size and / or configuration so that, after the transformation, for example scaling, the transformed patch covers / replaces the original region of interest (for example a tumour) and, in some implementations, the area surrounding the transformed region of interest in the transformed patch blends in with the area surrounding the original region of interest. In some implementations, the blending in may be facilitated by a blending operation, for example a blurring or averaging operation of the surrounding area of the transformed patch and the original surrounding area. In some implementations, transforming the extracted image portion comprises reducing the size of the extracted image portion. In other words, transforming the extracted image portion may comprise applying a scaling transformation to the extracted image portion, wherein the scaling transformation reduces the size of the extracted image portion, where the size of the extracted image portion refers to one or more dimensions of the extracted image portion in at least one direction. For example, the size of the extracted image portion may be the dimensions of the extracted image portion in the x-direction and y-direction, where coordinate points on the image are given in relation to an X-Y axis. In implementations where the region of interest comprises a tumour, the tumour may be rotated, flipped, scaled, or elastically transformed at step 430 of the claimed method. In implementations where the tumour is a late-stage cancerous tumour, the tumour may undergo a transformation such that the transformed tumour is representative of an early-stage cancerous tumour. That is to say, the late-stage tumour may be transformed such that it has the characteristics or qualities (for example, size, density, shape) of an early-stage cancerous tumour. The transformed tumour may be representative of an earlier stage of growth or development of the late-stage tumour present in the input image. For example, a scaling operation may be applied to the late-stage cancerous tumour, and the tumour may be resized according to the scaling operation such that it acquires the size of an early-stage cancerous tumour. This advantageously allows for synthetic clinical data relating to early-stage cancerous tumours to be generated to correct or compensate for imbalances in available clinical data. At step 440, an augmented image is generated using the transformed image portion. Generating the augmented image can comprise adding the transformed image portion to the input image to form an augmented image. Adding the transformed extracted image portion to the input image can comprise overlaying the transformed image portion onto the input image. The transformed extracted image portion may be overlaid onto the input image in approximately the same location or position as the region of interest. In other words, adding the transformed image portion to the input image may include overlaying the transformed image portion on to a portion of the input image comprising the region of interest identified in step 420 of the present method. In implementations where the volume or area of the transformed image portion is equal to that of the extracted image portion, or is sufficiently large to cover the original region of interest as described above, the transformed image portion can be overlaid onto the input image without a need for further augmentation. In implementations where the transformed image portion is smaller than the extracted image portion, any image data, e.g. surrounding tissue, needed to cover the remaining area of the extracted image portion can be filled in using known image infill techniques. In other implementations, the transformed extracted image portion may be overlaid onto the input image in a location, orientation or position which does not comprise the identified region of interest. In other words, the transformed extracted image portion may be overlaid onto a part of the input image other than the part of the input image comprising the identified region of interest. In some implementations where the transformed extracted image portion is representative of an early-stage cancerous tumour, and the identified region of interest on the input image comprises a late-stage cancerous tumour, the overlaying of the transformed extracted image portion onto the input image comprises overlaying the transformed image portion representative of the early-stage tumour over the portion of the input image which comprises the late-stage tumour. In this way, an augmented image can be generated which is representative of an early-stage cancer tumour, and / or which is representative of an earlier stage of the extracted tumour's evolution. In other implementations where the transformed extracted image portion is representative of an early-stage cancerous tumour, and the identified region of interest on the input image comprises a late-stage cancerous tumour, the overlaying of the transformed extracted image portion onto the input image comprises overlaying the transformed image portion representative of the early-stage tumour over a portion of the input image which does not include the late-stage tumour. In this way, an increased span of training data is generated. Figure 5 illustrates an exemplary method 500 of generating an augmented MR image comprising an early-stage cancer tumour using an MR input image comprising a late-stage cancer tumour. The steps of method 500 may be the same as or similar to those outlined in relation to method 400. At step 510, an input MR image is received, where the input MR image comprises a late-stage cancer tumour. The input MR image may be received by the system 600. In further implementations, the input MR image may be received by MR image processing circuitry 602. The input MR image may be received directly from an MRI acquisition device, where the MRI acquisition device acquired the MR image during an MRI scan of a patient. For example, the input MR image may be an image of the brain of a patient which comprises a cancerous tumour, which may be a late-stage cancerous tumour. At step 520, the input MR image is segmented to identify the late-stage cancer tumour. Any of the segmentation techniques outlined in relation to method step 420 may be utilised to segment the MR image and identify the tumour. This comprises identifying the portion of the MR image in which the tumour is located. That is to say, this step may comprise identifying the group of pixels representing the tumour in the MR image. At step 530, the image portion comprising the tumour is transformed using a data function. The data function may be the same or similar to any data function outlined in relation to step 430 and / or figure 2b. The data function receives the portion of the MR image comprising the identified tumour as an input, and generates a transformed tumour as an output, where the transformed tumour represents an early-stage cancer tumour. In implementations where the data function comprises a feature extraction function, the data function extracts an image portion corresponding to the segmented late-stage tumour from the input M R image. The extracted image portion is then transformed by the data function such that the transformed image portion resembles the early-stage cancer tumour. This may be achieved using any of the transformations outlined in step 430 of method 400. For example, the transformed image portion comprising the late-stage tumour may undergo a geometric transformation wherein the size of the transformed image portion is reduced (i.e. a scaling operation), such that it acquires the size of an early-stage cancerous tumour. The transformed image portion is thus representative of an early-stage cancer tumour, or at least comprises some of the qualities or characteristics of an early-stage cancer tumour. The transformed tumour may be representative of an earlier stage of growth or development of the late-stage tumour present in the MR input image. At step 540, an augmented MR image is generated using the transformed image portion. This may comprise adding the transformed image portion to the input MR image and may further comprise overlaying the transformed image portion onto the input MR image, in approximately the same location, orientation or position as the late-stage tumour was located, orientated or positioned in the input MR image. In this way, an augmented MR image is generated which is representative of an earlier stage of the tumour's evolution, or which is more generally representative of an early stage tumour. Figure 6 illustrates a block diagram of one implementation of a computing device 600 within which a set of instructions, for causing the computing device to perform any one or more of the methodologies discussed herein, may be executed. In alternative implementations, the computing device may be connected (e.g., networked) to other machines in a Local Area Network (LAN), an intranet, an extranet, or the Internet. The computing device may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The computing device may be a personal computer (PC), a tablet computer, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single computing device is illustrated, the term "computing device" shall also be taken to include any collection of machines (e.g., computers) that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. The example computing device 600 includes a processing device 602, a main memory 604 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a static memory 606 (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device 618), which communicate with each other via a bus 630. Processing device 602 represents one or more general-purpose processors such as a microprocessor, central processing unit, or the like. More particularly, the processing device 602 may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processing device 602 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. Processing device 602 is configured to execute the processing logic (instructions 622) for performing the operations and steps discussed herein. The computing device 600 may further include a network interface device 608. The computing device 600 also may include a video display unit 610 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 612 (e.g., a keyboard or touchscreen), a cursor control device 614 (e.g., a mouse or touchscreen), and an audio device 616 (e.g., a speaker). The data storage device 618 may include one or more machine-readable storage media (or more specifically one or more non-transitory computer-readable storage media) 628 on which is stored one or more sets of instructions 622 embodying any one or more of the methodologies or functions described herein. The instructions 622 may also reside, completely or at least partially, within the main memory 604 and / or within the processing device 602 during execution thereof by the computer system 600, the main memory 604 and the processing device 602 also constituting computer-readable storage media. The various methods described above may be implemented by a computer program. The computer program may include computer code arranged to instruct a computer to perform the functions of one or more of the various methods described above. The computer program and / or the code for performing such methods may be provided to an apparatus, such as a computer, on one or more computer readable media or, more generally, a computer program product. The computer readable media may be transitory or non-transitory. The one or more computer readable media could be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or a propagation medium for data transmission, for example for downloading the code over the Internet. Alternatively, the one or more computer readable media could take the form of one or more physical computer readable media such as semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disc, and an optical disk, such as a CD-ROM, CD-R / W or DVD. In an implementation, the modules, components and other features described herein can be implemented as discrete components or integrated in the functionality of hardware components such as ASICS, FPGAs, DSPs or similar devices. A "hardware component" is a tangible (e.g., non-transitory) physical component (e.g., a set of one or more processors) capable of performing certain operations and may be configured or arranged in a certain physical manner. A hardware component may include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component may be or include a special-purpose processor, such as a field programmable gate array (FPGA) or an ASIC. A hardware component may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. Accordingly, the phrase "hardware component" should be understood to encompass a tangible entity that may be physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. In addition, the modules and components can be implemented as firmware or functional circuitry within hardware devices. Further, the modules and components can be implemented in any combination of hardware devices and software components, or only in software (e.g., code stored or otherwise embodied in a machine-readable medium or in a transmission medium). Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as "receiving", "determining", "comparing ", "enabling", "maintaining," "identifying," or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other implementations will be apparent to those of skill in the art upon reading and understanding the above description. Although the present disclosure has been described with reference to specific example implementations, it will be recognized that the disclosure is not limited to the implementations described, but can be practiced with modification and alteration within the spirit and scope of the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative sense rather than a restrictive sense. The scope of the disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It will be understood that the above description of specific embodiments is by way of example only and is not intended to limit the scope of the present disclosure. Many modifications of the described embodiments, some of which are now described, are envisaged and intended to be within the scope of the present disclosure. The above implementations have been described by way of example only, and the described implementations and arrangements are to be considered in all respects only as illustrative and not restrictive. It will be appreciated that variations of the described implementations and arrangements may be made without departing from the scope of the disclosure. 14 04 25

Claims

1. A computer-implemented method for generating an augmented clinical image using a data-dependent augmentation scheme, comprising:receiving an input image, the input image comprising one or more regions of interest;transforming a portion of the input image using a data function configured to apply a data-dependent augmentation, wherein the data function:receives at least one region of interest of the image as an input, wherein the at least one region of interest received by the data function comprises a late-stage tumour,andgenerates a transformed image portion as an output based on the received region of interest, wherein the transformed image portion comprises a transformed tumour representative of an early stage cancer tumour; andgenerating an augmented image using the transformed image portion.

2. The method of claim 1, wherein the data function further comprises:extracting an image portion corresponding to the identified region of interest; and transforming the extracted image portion.

3. The method of claim 2, further comprising:adding the transformed extracted image portion to the input image.

4. The method of claim 3, wherein adding the transformed extracted image portion to the input image comprises overlaying the transformed extracted image portion onto the input image.

5. The method of any preceding claim, wherein transforming a portion of the input image comprises at least one of a geometric transformation, a colour transformation, and / or a kernel filter.

6. The method of claim 5, wherein transforming a portion of the input image comprises a geometric transformation, the geometric transformation comprising one or more of: rotation, scaling, flipping and / or elastic transformation.

7. The method of any preceding claim, further comprising segmenting the input image to identify a region of interest.14 04 258. The method of claim 1, further comprising:segmenting the input image to identify the late-stage cancer tumour;wherein the data function further comprises:extracting an image portion corresponding to the segmented late-stage tumour;transforming the extracted image portion;adding the transformed extracted image portion to the input image.

9. The method of claim 8, wherein transforming the extracted image portion comprises reducing the size of the extracted image portion.

10. The method of any preceding claim, wherein the input image comprises one or more MR images or one or more CT images.

11. The method of any preceding claim, wherein the input image comprises one or more MR images and the augmented image comprises one or more CT images.

12. A system for generating an augmented image, the system comprising:a memory storing machine-executable instructions and the data function of any preceding claim;a processor for controlling the system, wherein execution of the machine executable instructions causes the processor to control the system to perform the method of any preceding claim.

13. A computer-readable medium containing instructions which, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 11.

Images