Creating synthetic medical images

Abstract

An apparatus for creating a composite medical image includes a memory having instruction data representing a set of instructions, and a processor in communication with the memory and configured to execute the set of instructions, which, when executed by the processor, cause the processor to perform the steps of acquiring first and second medical images, determining elastic deformations that can be used to register the first medical image to the second medical image, and creating a composite medical image by weighting the determined elastic deformations and applying the weighted elastic deformations to portions of the first medical image.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION Embodiments of the present invention relate to medical imaging. In particular, but not exclusively, embodiments herein relate to creating synthetic medical images. [Background technology]

[0002] The present disclosure is in the area of ​​medical imaging. Medical images can be scarce, expensive, and difficult to obtain. Examples of medical images include computed tomography (CT) images, such as C-arm CT images, spectral CT images, or phase contrast CT images (e.g., from a CT scan), X-ray images (e.g., from an X-ray scan), magnetic resonance (MR) images (e.g., from an MR scan), ultrasound (US) images (e.g., from an ultrasound scan), fluoroscopy images, and nuclear medicine images. Summary of the Invention [Problem to be solved by the invention]

[0003] U.S. Patent Application Publication No. 2020 / 00900349A1 discloses a medical image diagnostic device that performs image registration between medical image data. This publication discloses pre-registration between medical data at a display scale of a first specified value, and then performing formal image registration at two or more display scales of a second specified value. The second specified value is smaller than the first specified value. This publication discloses partial image registration through a matching method that uses relative fine structure. This takes into account relatively macroscopic image information in the image and performs finer registration at a display scale of a second specified value smaller than the first specified value, thereby achieving fine and global image registration.

[0004] U.S. Patent Application Publication No. 2005 / 0146536A1 discloses a statistically-based image blending method for stitching multiple digital sub-images together into a single final past image, which discloses blending pixel intensities of at least two digital sub-images together to create the single stitched image.

[0005] The scientific paper "Generative Adversarial Networks in Medical Imaging: A Review" by Xin Yi, Ekta Walia, and Paul Baby describes the use of Generative Adversarial Networks (GANs) to create training data.

[0006] It is an object of embodiments of the present invention to improve the availability of medical images.

[0007] The aforementioned medical images may be scarce, difficult to obtain, and / or expensive. In this field, a large corpus of medical images is needed to develop (train and test) model classifiers, such as machine learning models, for improved image interpretation output. One way to overcome this shortage of medical image data is to use synthetic image data. However, classical methods for generating synthetic data may be based on phantoms (e.g., physical or synthetic models specifically designed to evaluate, analyze, and / or adjust the performance of an imaging device) that do not necessarily represent actual clinical data.

[0008] It is an object of embodiments of the present invention to improve this situation by providing an apparatus and method for creating synthetic medical images. [Means for solving the problem]

[0009] Thus, according to a first aspect of the present disclosure, there is provided an apparatus for creating a composite medical image, the apparatus comprising: a memory having instruction data representing a set of instructions; and a processor in communication with the memory and configured to execute the set of instructions, which, when executed by the processor, cause the processor to perform the steps of acquiring first and second medical images, determining elastic deformations that can be used to register the first medical image to the second medical image, and creating a composite medical image by weighting the determined elastic deformations and applying the weighted elastic deformations to portions of the first medical image.

[0010] Thus, in this manner, the device can create synthetic medical images from other (e.g., real) medical images by extrapolating between such medical images. The resulting synthetic image may be more likely to represent the real medical image compared to, for example, an image generated from a model. This is because the distortion or extrapolation is based on variability actually seen in the human population (e.g., the type of deformation is based on actually observed differences rather than purely hypothetical differences). Thus, the shapes of anatomical features in the resulting synthetic image are more likely to fall within the range of shapes seen in the real population.

[0011] According to a second aspect, there is a method of creating a composite medical image, the method comprising the steps of acquiring first and second medical images, determining elastic deformations that can be used to register the first medical image to the second medical image, and creating the composite medical image by weighting the determined elastic deformations and applying the weighted elastic deformations to portions of the first medical image.

[0012] According to a third aspect, there is a computer program product including a computer readable medium having computer readable code embodied therein, the computer readable code being configured, upon execution of instructions by a suitable computer or processor, to cause the computer or processor to perform the method of the second aspect.

[0013] For a better understanding, and to show more clearly how embodiments herein may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which: [Brief explanation of the drawings]

[0014] [Figure 1] 1 illustrates an apparatus according to some embodiments herein. [Figure 2] 1 illustrates a method according to some embodiments herein. [Figure 3] FIG. 10 illustrates another method according to some embodiments herein. DETAILED DESCRIPTION OF THE INVENTION

[0015] 1 illustrates an apparatus for creating a composite image according to some embodiments of the present disclosure. The apparatus may form part of (e.g., be included within) a computer system such as, for example, a laptop, desktop computer, or other device. In some embodiments, the apparatus 100 may form part of a cloud / distributed computing configuration.

[0016] The apparatus includes a memory 104 comprising instruction data representing a set of instructions, and a processor 102 in communication with the memory and configured to execute the set of instructions. Generally, the set of instructions, when executed by the processor, causes the processor to perform any of the embodiments of method 200 or 300 described below.

[0017] More specifically, the set of instructions, when executed by a processor, cause the processor to perform the steps of: i) acquiring first and second medical images; ii) determining (calculating) elastic deformations that can be used to register the first medical image to the second medical image; and iii) creating a composite medical image by weighting the determined elastic deformations and applying the weighted elastic deformations to a portion of the first medical image.

[0018] Processor 102 may have one or more processors, processing units, multi-core processors, or modules configured or programmed to control device 100 in the manners described herein. In particular implementations, processor 102 may have multiple software and / or hardware modules each configured to or for performing an individual step or steps of the methods described herein. Processor 102 may have one or more processors, processing units, multi-core processors, and / or modules configured or programmed to control device 100 in the manners described herein. In some implementations, for example, processor 102 may have multiple (e.g., interoperating) processors, processing units, multi-core processors, and / or modules configured for distributed processing. It will be understood by those skilled in the art that such processors, processing units, multi-core processors, and / or modules may be located in different locations and may perform different steps and / or different portions of a single step of the methods described herein.

[0019] The memory 104 is configured to store program code executable by the processor 102 to perform the methods described herein. Alternatively or additionally, the one or more memories 104 may be external to the device 100 (i.e., separate from or remote from the device). For example, the one or more memories 104 may be part of another device. The memory 104 may be used to store, for example, the first and second medical images, the created composite image, user input received from a user, and / or any other information and / or data received, calculated, or determined by the processor 102 of the device 100 or from any interface, memory, or device external to the device 100. The processor 102 may be configured to control the memory 104 to store the first and second medical images, the created composite image, and / or any other information described herein.

[0020] In some embodiments, memory 104 may have multiple sub-memories, each capable of storing a portion of instruction data, such as at least one sub-memory capable of storing instruction data representing at least one instruction of a set of instructions, and at least one other sub-memory capable of storing instruction data representing at least one other instruction of the set of instructions.

[0021] The apparatus may further include a user input device 106, such as a keyboard, mouse, or other input device, that allows a user to interact with the apparatus, for example, to provide first and / or second user inputs described below. In other embodiments, such user input device 106 may be remote from apparatus 100, and apparatus 100 may receive any user input from remote user input device 106 using electronic signals, for example, via a wired or wireless internet connection.

[0022] 1 shows only the components needed to explain this aspect of the invention, and it should be understood that in practical implementations, device 100 may have additional components to those shown. For example, device 100 may include a battery or other power source for powering device 100, or a means for connecting device 100 to a mains power source.

[0023] More particularly, the first medical image may include a medical image of a human or animal subject, e.g., a "real-life" human or animal subject. The first medical image may be acquired, for example, during a medical exam or scan. The first medical image may include a stock image from a database of medical images.

[0024] Alternatively, the first medical image may comprise a composite medical image (in such an example, the methods described herein may then be used to create other composite images from the initial first and second composite images).

[0025] The second medical image may include an image acquired using the same or a different method than the first medical image. For example, the second medical image may be acquired during a medical exam or scan. The second medical image may include a stock image from a database of medical images. Alternatively, the second medical image may include a synthetic medical image.

[0026] The second medical image may include a medical image of the same type of subject (e.g., a human or an animal) as the first medical image. The second medical image may include a different subject than the first medical image, e.g., the first medical image may include an image of a first person (e.g., a first patient or subject) and the second medical image may include an image of a second person (e.g., a second patient or subject).

[0027] The first and second medical images generally include common or overlapping features (such as common anatomical features). Generally, the first and second medical images can include one or more common landmarks that allow the first medical image to be registered to the second medical image (step ii). For example, both the first and second medical images can include images of the same anatomical feature. Examples of anatomical features include, but are not limited to, the lungs, the heart, the ventricles of the heart, the brain, the fetus, etc.

[0028] In general, for example, a first medical image may include a particular anatomical feature of a first person, and a second medical image may include the same anatomical feature of a second person.

[0029] The first and second medical images may comprise medical images of any imaging modality, including, but not limited to, computed tomography (CT) images (e.g., from a CT scan), such as a C-arm CT image, a spectral CT image, or a phase contrast CT image, an X-ray image (e.g., from an X-ray scan), a magnetic resonance (MR) image (e.g., from an MR scan), an ultrasound (US) image (e.g., from an ultrasound scan), a fluoroscopy image, a nuclear medicine image, or any other type of medical image.

[0030] In some embodiments, the first medical image may include an image of the same modality as the second medical image, for example, the first and second medical images may both be X-ray images, or both be MRI images, etc.

[0031] The first and second medical images may comprise two-dimensional images. In other embodiments, the first and second medical images may comprise three-dimensional images. In some embodiments, the first and second medical images may comprise two-dimensional slices across respective three-dimensional image volumes. The first and second medical images generally comprise a plurality (or set) of image components. For example, in embodiments where the first and second medical images comprise two-dimensional images, the image components may comprise pixels. For example, in embodiments where the first and second medical images comprise three-dimensional images, the image components may comprise voxels.

[0032] Returning to the apparatus 100, in some embodiments, in block i), the first and / or second medical images may be obtained, for example, from a database of images. Those skilled in the art will appreciate that the first and second medical images may be obtained by other means, for example, in real time as the scan is performed or via the internet.

[0033] In some embodiments, device 100 can be configured to receive user input (referred to herein as "third user input") that includes an indication of one or more demographic criteria (e.g., criteria related to the type of composite image required). The third user input can be obtained, for example, from the user input device of Figure 1. Examples of demographic criteria include, but are not limited to, age, gender, or ethnicity.

[0034] In some embodiments, the device can be configured to select the first and / or second medical images from the database of images based on demographic information and demographic criteria of each human subject in the images in the database of images.

[0035] For example, a user may provide user input indicating that they wish to generate a composite image of a subject having a particular age and / or gender. Accordingly, in block i), the processor may be configured to select first and second medical images from a database of medical images of subjects of matching age and gender. In this manner, users may generate composite medical images of different demographic groups.

[0036] In block ii), the processor is configured to determine elastic (e.g., non-rigid) deformations that can be used to register the first medical image to the second medical image. Those skilled in the art will be familiar with methods for performing image registration and determining elastic deformations that can be used to register the first medical image to the second medical image. Elastic deformations can include translational, rotational, and / or nonlinear components (e.g., elastic deformations can generally distort the first medical image). Thus, elastic deformations can be described as "deformation fields." Exemplary methods for image registration are described, for example, in T. Netsch et al. entitled "Towards Real-Time Multi-Modality 3-D Medical Image Registration." International Conference on Computer Vision, 2001. However, those skilled in the art will recognize that these are exemplary and that any suitable method for determining elastic deformations for registering the first medical image to the second medical image can be used.

[0037] In general, the elastic deformation can be determined based on landmarks throughout the first medical image or based on a portion of the first medical image, in other words, the elastic deformation field can be determined for a portion of the first medical image, and not necessarily throughout the first medical image.

[0038] Next, in block (iii), the processor is configured to create a synthetic medical image by weighting the determined elastic deformation (e.g., applying a weighting) and applying the weighted elastic deformation to a portion of the first medical image. In other words, a portion of the first medical image is selected and a portion of the determined elastic deformation is applied to this portion to create a synthetic image. Thus, the first medical image is deformed or modified to create an image having geometric features between the first medical image and the second medical image.

[0039] The weighting can include any value w in the range 0 < w < 1, where w is, for example, any percentage or ratio of the total elastic deformation.

[0040] The amount of deformation may be configurable. For example, in some embodiments, the weighting may be user-configurable. For example, the apparatus 100 can be configured to receive a second user input having a value for the weighting (e.g., from a user input device such as the user input device 106). For example, the user can, for example, determine how the synthetic image is distorted with respect to the first medical image.

[0041] In some embodiments, the set of instructions, when executed by the processor, causes the processor to repeat block (iii) for a plurality of different weightings to create a plurality of different synthetic images. For example, the second user input can include a plurality of different weightings. In this way, a plurality of different synthetic images can be efficiently created.

[0042] As described above, the composite image is created by weighting the determined elastic deformations and applying the weighted elastic deformations to a portion of the first medical image. Thus, the weighted elastic deformations can be applied to the entire first medical image or to a portion of the first medical image. Thus, in some embodiments, causing a processor to create the composite image includes causing a processor to apply the weighted elastic deformations to the entire first medical image such that the created composite image is the same size as the first medical image. In other embodiments, the composite image can be created by applying the weighted elastic deformations to a subset of image elements of the first medical image, where the resulting composite image includes only the subset of image elements, e.g., the composite image can include a selected portion of the complete first medical image.

[0043] In some embodiments, the apparatus can be configured to receive a first user input (e.g., from a first user input device 106) indicating a region of interest and a set of instructions that, when executed by the processor, cause the processor to perform blocks i), ii), and iii) in response to receiving the first user input.

[0044] The first user input can include a set of coordinates that define a region of interest. In some embodiments, the first medical image can be displayed to a user, and the user can select a region of interest using a user input device, such as a mouse or touch screen.

[0045] In embodiments where the first and second medical images include three-dimensional images, the region of interest may include a volume (referred to herein as a "volume of interest"). In this manner, a user may indicate a portion or area of ​​the complete first medical image for which a composite image should be generated, for example, allowing a user to accurately and efficiently generate a composite image of the region of interest from a larger image.

[0046] In some embodiments, this can be further augmented by further use of image segmentation. For example, in some embodiments, the set of instructions, when executed by a processor, causes the processor to perform the step of segmenting the first medical image to generate the segmentation. In such embodiments, causing the processor to perform the step of creating a composite image in block iii) includes causing the processor to perform the step of applying a weighted elastic deformation to a portion of the first medical image that includes the region of interest based on the segmentation.

[0047] For example, the processor can determine a location of the region of interest from the segmentation. For example, a user can indicate (in the first user input) a region of the body, e.g., a "chest region." In block iii), a synthetic medical image can be created by weighting the determined elastic deformations and applying the weighted elastic deformations to portions of the first medical image labeled as the chest region in the segmentation.

[0048] In some embodiments, the region of interest includes an anatomical feature and the first user input includes an indication of the anatomical feature. In block iii), a synthetic medical image can be created by weighting the determined elastic deformations and applying the weighted elastic deformations to portions of the first medical image that include the anatomical feature (as indicated in the first user input), as determined by the segmentation.

[0049] In this way, a user can specify a particular anatomical feature and create a composite image of that anatomical feature in an efficient manner.

[0050] Those skilled in the art are familiar with image segmentation, but simply put, image segmentation involves extracting shape / form information about objects or shapes captured in an image. This can be accomplished by converting the image into constituent blocks or "segments," with pixels or voxels within each segment having common attributes. In some methods, image segmentation can involve fitting a model to one or more features in the image.

[0051] One method of image segmentation is model-based segmentation (MBS), in which a triangular mesh of a target structure (e.g., heart, brain, lungs, etc.) is adapted to features in the image in an iterative manner. Segmentation models typically encode population-based appearance features and shape information. Such information describes allowable shape variations based on the actual shape of the target structure in members of the population. Shape variations can be encoded, for example, in the form of eigenmodes that describe how changes to one part of the model are constrained, or depending on the shape of other parts of the model. Model-based segmentation has been used in various applications to segment one or more target organs from medical images; see, for example, Ecabert, O., et al. 2008 entitled "Automatic Model-Based Segmentation of the Heart in CT Images"; IEEE Trans. Med. Imaging 27 (9), 1189-1201.

[0052] Another segmentation method uses machine learning (ML) models to convert an image into multiple constituent shapes (e.g., block shapes or block volumes) based on similar pixel / voxel values ​​and image gradients.

[0053] However, those skilled in the art will understand that these are merely examples and that any segmentation method can be used on the first and / or second medical images. As discussed above, segmentation can be used to identify regions of interest or anatomical features to select appropriate regions or portions of the complete first medical image for creating a composite image.

[0054] Thus, an apparatus is provided for creating synthetic medical images in an efficient manner that allows a user to create a synthetic image of a particular region or anatomical feature of interest in relation to an individual's specific demographic information.

[0055] 2, in some embodiments, there is a method 200 for creating a synthetic medical image. Method 200 can be performed by a device or system, such as device 100. Method 200 can also be performed by one or more blocks of a computer program.

[0056] Briefly, in a first step 202, the method includes acquiring first and second medical images. In a second step 204, the method includes determining elastic deformations that can be used to register the first medical image to the second medical image. In step 206, the method includes creating a composite medical image by weighting the determined elastic deformations and applying the weighted elastic deformations to a portion of the first medical image.

[0057] Obtaining the first and second medical images is discussed in detail with respect to block i) performed by the apparatus 100 of Figure 1, and it will be understood that the details apply equally to step 202 of the method 200. Determining elastic deformations that can be used to register the first medical image to the second medical image is discussed above with respect to block ii) performed by the apparatus 100 of Figure 1, and it will be understood that the details apply equally to step 204 of the method 200. Creating a composite medical image by weighting the determined elastic deformations and applying the weighted elastic deformations to portions of the first medical image is discussed above with respect to block iii) performed by the apparatus 100 of Figure 1, and it will be understood that the details apply equally to step 206 of the method 200.

[0058] 3, a method 300 implemented by an apparatus such as apparatus 100 is shown, according to some embodiments. In this embodiment, first and second medical images 302, 304 are acquired. The first and second medical images are selected to match the gender, age, and region (or zone) of the body as indicated by a user in user input from user input device 106. The first and second medical images are provided as input to a first module (e.g., a first computer module) 306, which determines elastic deformations that can be used to register the first medical image to the second medical image.

[0059] The second module 308 segments the first medical image to generate a segmentation. The segmentation can be performed using any segmentation method, as described above. The segmentation is used to determine a region or volume of interest in the first medical image (depending on whether the first and second medical images are two-dimensional or three-dimensional, respectively). The region or volume of interest can be determined based on the segmentation and user input indicating the desired region or volume of interest (e.g., a region or volume specified by the user in the user input can be located in the image using the segmentation). The third module 312 creates a composite medical image 314 by weighting the determined elastic deformation and applying the weighted elastic deformation to the volume of interest 310 in the first medical image. The volume of interest can be selected based on needs from the composite data and can include the entire image. In this way, a user can input parameters such as age, gender, and anatomical region or feature, and a composite image matching the input parameters can be quickly and reliably created.

[0060] Turning now to other embodiments, in another embodiment, a computer program product is provided that includes a computer-readable medium having computer-readable code embodied therein, the computer-readable code being configured, upon execution of instructions by a suitable computer or processor, to cause the computer or processor to perform a method or methods described herein, e.g., method 200 or method 300.

[0061] It will therefore be understood that the present disclosure also applies to computer programs adapted to carry out the embodiments, in particular computer programs on or in a carrier. The program may be in the form of source code, object code, code intermediates, and object code, such as in partially compiled form, or in any other form suitable for use in the implementation of the methods described herein.

[0062] It will further be understood that such programs may have many different architectural designs. For example, program code implementing the functionality of a method or apparatus may be subdivided into one or more subroutines. Many different ways of distributing functionality among these subroutines will be apparent to those skilled in the art. The subroutines may be stored together in an executable file to form a self-contained program. For example, the executable file may include computer-executable instructions, such as processor instructions and / or interpreter instructions (e.g., Java interpreter instructions). For example, one or more or all of the subroutines may be stored in, for example, an external library file and linked with the main program, for example, statically or dynamically, at run time. The main program includes at least one call to at least one of the subroutines. The subroutines may also have function calls to each other.

[0063] The carrier of a computer program may be any entity or device capable of carrying the program. For example, the carrier may comprise a data storage device such as a ROM, for example a CD-ROM or a semiconductor ROM, or a magnetic recording medium, for example a hard disk drive. Furthermore, the carrier may be a transmissible carrier such as an electric or optical signal, which may be conveyed via an electric or optical cable or by radio or by other means. When the program is embodied in such a signal, the carrier may be constituted by such a cable or other device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, the integrated circuit being adapted for, or used for, performing the relevant method.

[0064] Variations to the disclosed embodiments can be understood and effected by those skilled in the art, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. A computer program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. An apparatus for creating a composite medical image, comprising: a memory containing instruction data representing a set of instructions; a processor in communication with the memory and configured to execute the set of instructions; the set of instructions, when executed by the processor, cause the processor to: i) acquiring first and second medical images, the first and second medical images having common or overlapping features; ii) determining elastic deformations that can be used to register the first medical image to the second medical image; iii) creating the composite medical image by weighting the determined elastic deformations and applying the weighted elastic deformations to portions of the first medical image to create an image having features that are geometrically intermediate between features in the first medical image and features in the second medical image; A device that performs the following.

2. The set of instructions, when executed by the processor, further cause the processor to perform the steps of segmenting the first medical image to generate a segmentation; The apparatus of claim 1 , wherein the step of creating the synthetic medical image comprises applying the weighted elastic deformation to a portion of the first medical image that includes a region of interest based on the segmentation.

3. 3. The apparatus of claim 2, wherein the apparatus is configured to receive a first user input indicating the region of interest, and the set of instructions, when executed by the processor, cause the processor to perform steps i), ii), and iii) in response to receiving the first user input.

4. The apparatus of claim 3 , wherein the region of interest includes an anatomical feature, and the first user input includes an indication of the anatomical feature.

5. The apparatus of claim 3 , wherein the first and second medical images comprise three-dimensional images, and the first user input includes an indication of a volume of interest in the first medical image.

6. 6. Apparatus according to any one of claims 1 to 5, wherein the weighting is user configurable.

7. The apparatus of claim 6 , wherein the apparatus is configured to receive a second user input comprising a value for the weighting.

8. The apparatus of claim 1 , wherein the first medical image and the second medical image comprise medical images of a human subject.

9. 9. The apparatus of claim 8, wherein the apparatus is configured to receive a third user input indicating one or more demographic criteria, and wherein the step of acquiring the first and second medical images includes selecting the first and / or second medical images from a database of images based on demographic information of each human subject in the database of images and the demographic criteria.

10. The apparatus of claim 9 , wherein the demographic criteria include age or gender.

11. 11. The apparatus of claim 1, wherein the set of instructions, when executed by the processor, causes the processor to perform the step of repeating step iii) for a plurality of different weightings to create a plurality of different composite images.

12. The apparatus of claim 1 , wherein the first and second medical images are of the same imaging modality.

13. 13. The apparatus of claim 1, wherein the elastic deformation comprises a strain field and / or the weighting, the weighting having a value w in the range 0<w<1.

14. 1. A method for creating a composite medical image, comprising: i) acquiring first and second medical images, wherein the first and second medical images include common or overlapping features; ii) determining elastic deformations that can be used to register the first medical image to the second medical image; iii) creating the composite medical image by weighting the determined elastic deformations and applying the weighted elastic deformations to portions of the first medical image to create an image having features that are geometrically intermediate between features in the first medical image and features in the second medical image; A method having the following.

15. 15. A computer readable medium having computer readable code thereon, said computer readable code, when executed by a suitable computer or processor, causing said computer or processor to perform the method of claim 14.

Images