Apparatus and method for visualizing imaging data

Abstract

[0003] Embodiments relate to a method for mapping additional information related to a 2D medical image of a region of interest onto a 3D medical image volume of the region of interest, as well as an apparatus and a computer program configured to perform such a method. The method according to one embodiment includes the steps of receiving a 2D composite medical image of the region of interest and the 3D medical image volume of the region of interest, decomposing the 2D composite medical image into an additional image including at least a base image and the additional information, identifying a representative 2D slice of the 3D medical image volume corresponding to the base image based on an evaluation of the base image and the 3D medical image volume, evaluating a transformation of the base image that maps the base image to the representative 2D slice, and combining the image and the representative 2D slice formed by applying the evaluated transformation to the additional image to generate a new composite 2D medical image of the region of interest.

Description

[Technical Field]

[0001] Various example embodiments relate to imaging data. In particular, aspects and embodiments relate to apparatus and methods for mapping or visualizing additional information about a 2D medical image of a region of interest onto a 3D medical image volume of the region of interest. [Background technology]

[0002] A patient experiencing a chronic condition or disease may undergo one or more treatments and associated follow-on treatments or aftercare. The initial investigation, treatment, and resulting follow-up process may be understood to represent a series of substantially independent interactions with the subject, each of which may involve the acquisition or creation of one or more images or sets of imaging data related to the subject and the condition or disease of interest.

[0003] By way of example, cancer patients may undergo initial investigations and may also undergo follow-up imaging studies, typically in a radiology department, after undergoing cancer treatment. Such studies may be performed for the purposes of tumor response assessment and / or evaluation of tissue changes after undergoing cancer treatment.

[0004] When imaging a patient, the imaging method often supports the acquisition of 3D imaging data.

[0005] In the case of cancer treatment, for example, imaging data relating to previous tumor evaluations or examinations, and imaging data capturing details of previous investigations or treatments (e.g., radiation therapy doses), can be useful to various medical professionals interacting with the patient. For example, correlation of equivalent 3D imaging data from previous evaluations or treatments with the imaging data being acquired can support, for example, measurement or assessment of tumor changes. Such direct correlation between 3D imaging datasets may also enable, for example, radiologists to more easily distinguish between tumor progression and radiation therapy side effects.

[0006] Technically, the task of "correlation" or image labeling by dose level can be solved by performing a non-rigid registration between the follow-up 3D image (CT, MRI, etc.) and the previously acquired 3D planning image, and by warping the known delivered dose information (or dose levels) known for the 3D planning image to the geometry of the follow-up 3D image. Similarly, 3D image registration can be used to correlate findings in a current 3D medical image with findings in a previously acquired 3D medical image.

[0007] While technically feasible, such registration between simultaneous tracking 3D imaging datasets acquired in a radiology practice to planning 3D imaging data previously acquired in a radiotherapy department can be made difficult by various factors, including, for example, data access limitations between departments and / or different institutions that may not share the same image data archive. Furthermore, even within the same hospital, searching and retrieving previous 3D CT and radiotherapy dose distribution data from PACS archives can be slow.

[0008] Savjani et al., "A Framework for Sharing Radiation Dose Distribution Maps in the Electronic Medical Record for Improving Multidisciplinary Patient Management" [Radiol Imaging Cancer 2021 Mar 12;3(2):e200075. doi: 10.1148 / rycan.2021200075. eCollection 2021 Mar], notes that specialized software and hardware in radiation oncology practices can result in radiation treatment histories that are inaccessible to other medical subspecialties. This reference cites the difficulty that arises when 3D radiation treatment data specific to radiation oncology departments is pushed to hospital picture archiving and communication systems (PACS), noting that raw radiation treatment data is not readable by the PACS provided by many hospitals.

[0009] Some adaptations to the processes and methods for sharing imaging data may support the creation of information of ongoing use to medical professionals by providing a way to combine previously acquired information with new imaging datasets. Summary of the Invention [Problem to be solved by the invention]

[0010] While technically feasible, such registration between simultaneous tracking 3D imaging datasets acquired in a radiology practice to planning 3D imaging data previously acquired in a radiotherapy department can be made difficult by various factors, including, for example, data access limitations between departments and / or different institutions that may not share the same image data archive. Furthermore, even within the same hospital, searching and retrieving previous 3D CT and radiotherapy dose distribution data from PACS archives can be slow. [Means for solving the problem]

[0011] The scope of protection sought for various exemplary embodiments of the invention is indicated by the independent claims. The exemplary embodiments and features described herein that do not fall within the scope of the independent claims, if any, should be interpreted as examples useful for understanding various embodiments of the invention.

[0012] According to various, but not necessarily all, exemplary embodiments, an apparatus is provided that is configured to map additional information associated with a 2D image of a region of interest onto a 3D image volume of the region of interest, the apparatus having at least one processor and at least one memory that stores instructions that, when executed by the at least one processor, cause the apparatus to at least receive a 2D composite image of the region of interest, receive a 3D image volume of the region of interest, decompose the 2D composite image into at least a base image and an additional image having the additional information, identify a representative 2D slice of the 3D image volume that corresponds to the base image based on an evaluation of the base image and the 3D image volume, evaluate a transformation of the base image that maps the base image to the representative 2D slice, and combine the representative 2D slice with an image formed by applying the evaluated transformation to the additional image to generate a new composite 2D image of the region of interest.

[0013] In one embodiment, the 2D image of the region of interest comprises a color image.

[0014] In one embodiment, the composite 2D image comprises a 2D summary image of a 3D image volume representing the region of interest.

[0015] In one embodiment, the composite 2D image includes additional information regarding the treatment or procedure performed on the region of interest.

[0016] In one embodiment, the additional information includes the geometric distribution of the therapeutic dose applied to the region of interest.

[0017] In one embodiment, the treatment comprises radiation therapy.

[0018] In one embodiment, the geometric distribution comprises isodose lines or color map dose information.

[0019] In one embodiment, the additional information includes one or more annotations including an outline of the feature of interest, and / or a marker of the feature of interest, and / or a measurement of the feature of interest.

[0020] In one embodiment, decomposing the 2D composite image includes decomposing the 2D composite image into at least a grayscale base image and an additional image that includes the additional information.

[0021] In one embodiment, decomposing the 2D composite image includes analyzing pixels of the 2D composite image to determine a grayscale base image and an additional image containing the additional information.

[0022] In one embodiment, identifying a representative 2D slice of the 3D image volume that corresponds to the base image includes evaluating one or more planar images that form the 3D image volume against the base image, and selecting a planar image from the evaluated one or more planar images that best matches the base image.

[0023] In one embodiment, identifying a representative 2D slice of the 3D image volume and estimating a transformation of the base image that maps the base image to the representative 2D slice are determined as a combined calculation.

[0024] In one embodiment, identifying a representative 2D slice of the 3D image volume and evaluating a transformation of the base image that maps the base image to the representative 2D slice includes using an image registration technique such as slice-to-volume or 2D-to-2D registration.

[0025] In one embodiment, evaluating a transformation of the base image that maps the base image to the representative 2D slice includes evaluating one or more differences between the images resulting from one or more of rotation, translation, deformation, or scaling.

[0026] According to various, but not necessarily all, example embodiments, a computer-implemented method is provided for mapping additional information associated with a 2D image of a region of interest to a 3D image volume of the region of interest, the method including: receiving a 2D composite image of the region of interest and the 3D image volume of the region of interest; decomposing the 2D composite image into at least a base image and an additional image including the additional information; identifying a representative 2D slice of the 3D image volume corresponding to the base image based on an evaluation of the base image and the 3D image volume; evaluating a transformation of the base image that maps the base image to the representative 2D slice; and combining the image and the representative 2D slice formed by applying the evaluated transformation to the additional image to generate a new composite 2D image of the region of interest.

[0027] In one embodiment, the 2D image of the region of interest comprises a color image.

[0028] In one embodiment, the composite 2D image comprises a 2D summary image of a 3D image volume representing the region of interest.

[0029] In one embodiment, the composite 2D image includes additional information regarding the treatment or procedure performed on the region of interest.

[0030] In one embodiment, the additional information includes the geometric distribution of the therapeutic dose applied to the region of interest.

[0031] In one embodiment, the treatment comprises radiation therapy.

[0032] In one embodiment, the geometric distribution comprises isodose lines or color map dose information.

[0033] In one embodiment, the additional information includes one or more annotations including an outline of the feature of interest, and / or a marker of the feature of interest, and / or a measurement of the feature of interest.

[0034] In one embodiment, decomposing the 2D composite image includes decomposing the 2D composite image into at least a grayscale base image and an additional image that includes the additional information.

[0035] In one embodiment, decomposing the 2D composite image includes analyzing pixels of the 2D composite image to determine a grayscale base image and an additional image containing the additional information.

[0036] In one embodiment, identifying a representative 2D slice of the 3D image volume that corresponds to the base image includes evaluating one or more planar images that form the 3D image volume against the base image, and selecting a planar image from the evaluated one or more planar images that best matches the base image.

[0037] In one embodiment, identifying a representative 2D slice of the 3D image volume and estimating a transformation of the base image that maps the base image to the representative 2D slice are determined as a combined calculation.

[0038] In one embodiment, identifying a representative 2D slice of the 3D image volume and evaluating a transformation of the base image that maps the base image to the representative 2D slice includes using an image registration technique such as slice-to-volume or 2D-to-2D registration.

[0039] In one embodiment, evaluating a transformation of the base image that maps the base image to the representative 2D slice includes evaluating one or more differences between the images resulting from one or more of rotation, translation, deformation, or scaling.

[0040] According to various, but not necessarily all, exemplary embodiments, a computer program product is provided that, when executed on a computer, is operable to perform on the computer a method for mapping additional information associated with a 2D image of a region of interest to a 3D image volume of the region of interest, the method comprising: receiving a 2D composite image of the region of interest and a 3D image volume of the region of interest; decomposing the 2D composite image into at least a base image and an additional image including the additional information; identifying a representative 2D slice of the 3D image volume that corresponds to the base image based on an evaluation of the base image and the 3D image volume; evaluating a transformation of the base image that maps the base image to the representative 2D slice; and combining the representative 2D slice with an image formed by applying the evaluated transformation to the additional image to generate a new composite 2D image of the region of interest.

[0041] According to various, but not necessarily all, exemplary embodiments, a non-transitory computer-readable medium is provided that stores computer program code including instructions that, when executed by a processor, cause the computer to perform a method for mapping additional information associated with a 2D image of a region of interest to a 3D image volume of the region of interest, the method including: receiving a 2D composite image of the region of interest and the 3D image volume of the region of interest; decomposing the 2D composite image into at least a base image and an additional image including the additional information; identifying a representative 2D slice of the 3D image volume that corresponds to the base image based on an evaluation of the base image and the 3D image volume; evaluating a transformation of the base image that maps the base image to the representative 2D slice; and combining the image formed by applying the evaluated transformation to the additional image with the representative 2D slice to generate a new composite 2D image of the region of interest.

[0042] Some embodiments recognize that it is possible to utilize information provided in a 2D image and map that information to new imaging information that forms part of a 3D image volume. Providing a visualization of such mapping in the form of a new composite 2D image can assist imaging professionals in more efficiently evaluating acquired image data representing a region of interest.

[0043] In particular, embodiments recognize that mapping additional information onto a portion of a 3D image volume of a region of interest may assist an imaging professional in performing the technical task of assessing the clinical relevance of acquired 3D image data, which may be performed for the purpose of determining an appropriate course of treatment or for assessing or diagnosing a disease or chronic condition based on the acquired image data.

[0044] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments and configurations described hereinafter.

[0045] Further particular and preferred aspects are set out in the accompanying independent and dependent claims. Features from the dependent claims may be combined with features of the independent claims as appropriate and in combinations other than those explicitly set out in the claims.

[0046] Where a device feature is described as operable to provide a functionality, this will be understood to include a device feature that provides that functionality or is adapted or configured to provide that functionality.

[0047] Several exemplary embodiments will now be described with reference to the accompanying drawings. [Brief explanation of the drawings]

[0048] [Figure 1A] 2 shows two exemplary 2D images of a region of interest of a subject. [Figure 1B] 2 shows two exemplary 2D images of a region of interest of a subject. [Figure 2]1 shows a schematic representation of the inputs and outputs of the method according to one configuration. [Figure 3] 10 shows a schematic representation of the inputs and outputs of a method according to a further arrangement; [Figure 4] 1 shows a schematic diagram of a system for use in one configuration. [Figure 5] 1 shows a schematic of the main steps of the method according to one configuration. DETAILED DESCRIPTION OF THE INVENTION

[0049] Before describing exemplary embodiments in further detail, a context in which aspects and embodiments should be understood is first provided.

[0050] A patient experiencing a chronic condition or disease may undergo one or more interactions with various medical professionals. The initial investigation, treatment, and resulting follow-up process may be understood to represent a series of substantially independent interactions with the patient, each of which may involve the acquisition or creation of one or more images or sets of imaging data related to the patient and the condition or disease of interest.

[0051] As an example, a cancer patient may undergo an initial investigation including imaging of the area of ​​interest, treatment of the area of ​​interest which may be planned or may occur using the imaging process, and may also undergo follow-up imaging studies, typically in a radiology department, for example, after receiving cancer treatment.

[0052] Ongoing imaging studies of a patient can be performed for a variety of purposes, including monitoring disease progression, tumor response assessment, and / or evaluation of tissue changes after receiving cancer treatment.

[0053] When imaging a patient, imaging methods often used support the acquisition of 3D imaging data in the form of a 3D image volume. Examples of 3D imaging methods include computed tomography scans (CT scans) and magnetic resonance imaging scans (MRI scans). Such imaging techniques allow for the acquisition of a 3D data set or image volume related to a region of the body.

[0054] In the case of cancer treatment, for example, CT and / or MRI scans may be performed to obtain information regarding tumor evaluation or examination. The CT or MRI imaging data may contain details regarding previous investigations or treatments (e.g., radiation therapy doses) that are useful to various medical professionals who continue to interact with the patient.

[0055] For example, correlation of new 3D imaging data being acquired with comparable 3D imaging data from a previous evaluation or treatment can support, for example, measurement or assessment of tumor changes. Such direct correlation between 3D imaging datasets may, for example, allow a radiologist to more easily distinguish between tumor progression and side effects of radiation therapy.

[0056] Technically, the task of "correlation" or image labeling by dose level can be solved by performing a non-rigid registration between a newly acquired 3D image (CT, MRI, etc.) and a previously acquired 3D planning image, and by warping the known delivered dose information (or dose levels) known for the 3D planning image to the geometry of the follow-up 3D image. Similarly, 3D image registration can be used to correlate findings in a current 3D medical image with findings in a previously acquired 3D medical image.

[0057] While technically feasible, such registration between new 3D imaging datasets and previously acquired 3D imaging data in a radiation therapy department (e.g., registering the new imaging dataset to a dataset used to plan radiation therapy doses) can be made difficult by various factors, including, for example, data access limitations between departments and / or different institutions that may not share the same image data archive. Furthermore, even within the same hospital, searching and retrieving previous 3D CT and radiation therapy dose distribution data from PACS archives can be slow.

[0058] Previously collected complete 3D data may not be available or may be subject to practical limitations. However, some simpler information summarizing details of previously acquired imaging data along with details of the applied treatment may be available, for example, when performing a subsequent study. As an example, some 2D information, e.g., images or similar information summarizing details of previously acquired imaging data along with the delivered radiation treatment plan, is typically made available and associated with records kept for the patient.

[0059] Information about a patient can be made available to various parties or medical professionals as the patient progresses and is treated or monitored. For example, information summarizing details of treatment administered to a patient may include softcopy or hardcopy documents.

[0060] The information summary maintained for a patient may include one or more summary 2D images of the region of interest. The summary images may include, for example, RGB images of representative axial, coronal, and sagittal slices of 3D planning imaging data. Such imaging data may include, for example, 3D computed tomography (CT) data acquired for the patient. The summary 2D images of the region of interest provided in the information summary may be fused with additional information regarding the treatment or procedure to be performed for the patient. An example of such treatment-related information includes the geometric distribution of the planned radiation therapy dose. Such dose information may include, for example, color-coded isodose lines or a color map representing the radiation dose. The color map allows a user to visualize the dose based on the color-coded map. The color-coded map assigns colors to different dose ranges. The color map may include a color gradient that correlates with the dose. A summary document may be created by the radiation oncologist at the end of the treatment. The summary document is created for the purpose of communicating the details of the treatment or procedure to other disciplines and departments.

[0061] Similarly, representative 2D images from a previous 3D imaging event, e.g., a tumor assessment exam, can be summarized in one or more documents, screenshots, or stored in an electronic medical record (EMR) as one or more 2D images in a standard format (e.g., jpeg). The representative 2D images from the previous imaging event may include various additional information, such as tumor region-of-interest (ROI) contours, tumor size measurements (e.g., lines, arrows, text, or the like indicating tumor diameter), and annotated images. 3D medical images in a picture archiving and communication system (PACS) archive typically do not include such annotations. For a medical professional, e.g., a radiologist, to perform a subsequent study to acquire additional imaging data, it can be time-consuming to load any 3D images from the previous exam or study from the PACS archive, reconstruct previous measurements that may have been provided in 2D format, and correlate any current findings with such previous measurements.

[0062] While in principle it may seem possible for an imaging or other medical professional to combine information from two sources (a newly acquired 3D data set and a 2D summary of a previous study, treatment or intervention) simply by looking at these sources of information, it will be appreciated that the complexity of the information and imaging data acquired, together with changes in the anatomy, e.g., rotation, translation, and / or deformation, different scaling, etc., means that it may be difficult for a user to directly correlate treatment dose or other information from the summary representative 2D image, or information provided in connection with a previously acquired imaging study, with the 3D patient anatomy in a follow-up CT or MRI scan.

[0063] Having now described the context in which aspects and embodiments should be understood, a summary of methods according to some possible implementations is provided.

[0064] The arrangement may provide a method for mapping additional information about a 2D image of a region of interest onto a 3D image volume of the region of interest. The arrangement may provide an apparatus configured to perform such a method. The apparatus may have a computer, for example in the form of at least one processor, and at least one memory storing instructions that, when executed by the at least one processor, cause the computer to perform the method.

[0065] A method according to one configuration includes receiving a 2D composite image of a region of interest and a 3D image volume of the region of interest. The method may include decomposing the 2D composite image into at least a base image and an additional image containing additional information. The method may include identifying a representative 2D slice of the 3D image volume corresponding to the base image based on an evaluation of the base image and the 3D image volume. The method may include evaluating a transformation of the base image that maps the base image to the representative 2D slice. The method may include combining the representative 2D slice with an image formed by applying the evaluated transformation to the additional image to generate a new composite 2D image of the region of interest.

[0066] In general, implementations recognize that it is possible to utilize information provided in a 2D image and map that information to new imaging information that forms part of a 3D image volume. Providing visualization of such mapping in the form of a new composite 2D image can assist imaging professionals in more efficiently evaluating acquired image data representing a region of interest. In particular, mapping additional information to a portion of a 3D image volume of a region of interest may assist imaging professionals in performing the technical task of assessing the clinical relevance of the acquired 3D image data. Such assessment may be performed for purposes of determining an appropriate course of treatment or to evaluate or diagnose a disease or chronic condition based on the acquired image data.

[0067] As previously mentioned, aspects and embodiments recognize that an information summary maintained associated with a patient may include one or more summary 2D images of the region of interest. The summary images may include, for example, RGB images of representative axial, coronal, and sagittal slices of 3D planning imaging data. Such imaging data may include, for example, 3D computed tomography (CT) data acquired for the patient. The summary 2D images of the region of interest provided in the information summary may be fused with additional information related to image evaluation and / or the treatment or procedure performed for the patient. An example of such treatment-related information includes the geometric distribution of the planned radiation therapy dose. Such dose information may include, for example, color-coded isodose lines or color-gradient radiation doses. A summary document may be created by the radiation oncologist at the end of treatment. The summary document is created for the purpose of communicating treatment and procedure details to other disciplines and departments.

[0068] Aspects and embodiments recognize that such summary information, including 2D information, can be utilized, for example, in conjunction with a newly acquired 3D imaging dataset to provide a user, e.g., a medical professional, with useful information to enhance and enable evaluation of the newly acquired 3D imaging dataset. Useful information may include augmented imaging data that provides user assistance for interpreting the newly acquired 3D imaging data. For example, the augmented imaging data may assist a user in identifying and distinguishing tissue changes within the radiation treatment region compared to tissue changes outside the radiation treatment region.

[0069] Similarly, aspects and embodiments recognize that representative 2D images of a previous 3D imaging event, e.g., a tumor assessment exam, may be summarized in one or more documents, screenshots, or stored in an electronic medical record (EMR) as one or more 2D images in a standard format (e.g., jpeg). The representative 2D images of the previous imaging occasion may include various forms of additional information that are not part of the raw, acquired image data. For example, additional information included in the 2D image of the region of interest may include one or more annotations indicating the outline or appearance of the tumor region of interest (ROI), tumor size measurements (e.g., lines, arrows, text, or the like indicating tumor diameter), etc.

[0070] Aspects and embodiments recognize that summary information about a previous imaging event, including 2D information, can be utilized in conjunction with a newly acquired 3D imaging dataset to provide useful information to a user. The useful information may include augmented imaging data that provides user assistance in interpreting the newly acquired 3D imaging data. For example, the augmented imaging data may assist a user regarding tumor progression or the like, even if no treatment is being administered.

[0071] Aspects and embodiments provide apparatus and methods that support visualization of information derived from 2D images summarizing prior findings and / or treatments on a portion of a subsequently acquired 3D imaging dataset. Such information may include, for example, one or more geometric features related to the prior findings and / or treatments. The representative 2D image containing the additional information may comprise part of a treatment summary report or may be summary information related to a prior imaging occasion. The representative 2D image may include one or more images provided in a record associated with the patient or imaging subject. The representative image may include images stored in a standard image format within an electronic medical record (EMR) system.

[0072] Thus, according to some configurations, 2D imaging information, 2D imaging datasets, and / or information mapping to 2D images can be received or otherwise acquired, so that information regarding such 2D images can be used in relation to 3D imaging volumes acquired for similar regions of interest.

[0073] In order to take advantage of the additional information provided in or associated with the 2D image, it may be necessary to decompose the 2D image into at least two parts, which may include base image data and additional information that does not form part of the base image data.

[0074] According to one exemplary configuration, a representative 2D RGB summary image including an image of the region of interest including additional information and / or annotation color dose distribution is Grayscale slice of the planning image (S) and Additional features, e.g., color geometric dose features (GDFs) can be decomposed into

[0075] According to one configuration, color (e.g., isodose or tumor extent) lines present in the representative 2D summary image can be extracted, for example, using a simple RGB decomposition of the picture and detecting pixels with unequal R, G, and B values. In some examples, transparent color patches of alpha-blended images can be extracted using mathematical optimization techniques. Thus, the summary image can be suitably divided into at least two images, one image related to the base image of the region of interest and a further image containing additional information related to the base image.

[0076] According to some configurations, once the summary image is decomposed into at least two images, a transformation (T) is calculated that maps a grayscale slice (S) of a base image, e.g., a planning image, to a substantially equivalent plane or slice of a 3D dataset acquired, e.g., in the form of a 3D volume or follow-up image. The 3D imaging dataset V may include, e.g., new CT or MRI information acquired related to the region of interest.

[0077] According to some configurations, a transformation is selected or evaluated to find a plane P or slice of the 3D dataset V that is found to substantially correspond to the 2D base image data. The evaluation may include, for example, evaluating each plane P in the volume V that may correspond to a representative slice S. The evaluation may aim to determine the index of the slice (or plane) in V that best matches the base image.

[0078] The transformation T can be determined using, for example, slice-to-volume registration techniques or simple 2D-to-2D image registration techniques, which may adapt the transformation of the base image to align with the plane of the image volume that best matches the base image to account for one or more differences between the images resulting from one or more of rotation, translation, deformation, or scaling.

[0079] According to some configurations, a mapping or transformation correlating the base image of the 2D image to a plane or slice of the 3D image dataset is determined, and the mapping can be applied to the additional information image. The additional information image may include, for example, an image containing a representative 2D geometric dose feature (GDF), or other additional information, such as one or more indicators of size, position, measurement, or similar information related to one or more features included in the base image. In other words, according to some configurations, a mapping of the additional information available in relation to the summary 2D image can be applied so that the additional information can be equivalently related to the appropriate plane or section of the 3D imaging dataset. Thus, in some configurations, T(GDF) is evaluated.

[0080] According to some configurations, the transformation of the additional information image can be combined with a plane or slice of the 3D dataset determined to best correspond to the base image data. In one example, an appropriately transformed or mapped additional information image can be overlaid or combined with a plane or slice of the 3D dataset determined to be substantially equivalent to the base image of the 2D summary image. In some configurations, for example, T(GDF) is overlaid on P, thereby allowing direct visualization of, for example, dose features on P. Similarly, by overlaying T(GDF) on P, the P image can be viewed by a medical professional and can include planned dose level information.

[0081] 2D Image Decomposition 1A and 1B show two exemplary 2D images of a region of interest of a subject.

[0082] FIG. 1A shows a schematic representation of an image of a slice of a subject's torso. The 2D image in FIG. 1A schematically illustrates an image that may result from a subject with histologically proven non-small cell lung carcinoma. The carcinoma is marked on the image as a contour 100. The general region of interest is indicated by contour line 200 (entire torso). Also marked on the image of FIG. 1A are various other features of interest, including the lungs 110, spinal cord 120, esophagus 130, and trachea 140.

[0083] FIG. 1B shows a slice of the subject's torso from FIG. 1A, in which the dose distribution of a radiation therapy treatment plan is shown. The treatment plan may include, for example, a stereotactic body radiation therapy plan. The treatment plan involves a VMAT technique, with 5 Gy in 10 fractions providing a total dose of 50 Gy. Iso-dose lines (% of total dose) 300 are shown, and a color overlay indicates the dose distribution, i.e., regions 400, 410 with high-dose areas surrounding the tumor, step-down dose regions 420, 430, and a large low-dose region 440.

[0084] It will be appreciated that either Figure 1A or Figure 1B can be used as a 2D image that includes both base image data (grayscale image data set) and additional information. Figure 1A includes additional information related to the contours of features of interest. Figure 1B includes additional information related to the contours of features of interest and additional information related to the treatment plan, i.e., in this example, isodose lines and color scale dose information overlaid on the base image data.

[0085] According to the arrangement, a composite image such as that shown in FIGS. 1A and 1B is "unblended" or "decomposed" to produce at least two separate images.

[0086] 1A and 1B can be understood to include base image data in the form of slice images of interest generated from a 3D CT image dataset. In the case of FIG. 1B, that base image data is overlaid (i.e., alpha-blended) with at least a second image at a given transparency. The second image, in this example, represents the planned or intended radiation therapy treatment dose. Typically, the base image (in this case, a CT image) includes a grayscale image, and the treatment dose image or additional information included in the composite image may typically include one or more color features or color images.

[0087] To address image decomposition (i.e., unblending), when presented with a composite image such as that shown in Figure 1A or Figure 1B, the goal is to obtain versions of the two original images from a single overlaid composite image.

[0088] In general, such decomposition may only be possible if certain assumptions are made about the properties of the two original images. In a typical configuration, the color values ​​of pixels in each of the participating images are represented as RGB triples. The mapping of CT intensity to the corresponding gray in each pixel of the grayscale image is achieved through a grayscale color map, which is assumed to be known. The mapping of radiation therapy dose levels to image color pixels is achieved through an unknown RGB color mapping.

[0089] Therefore, one possible approach to performing decomposition of an overlaid RGB image on a grayscale base image can be based on estimating the unknown color mapping (used to generate the original dose planning image) from the overlaid image pixel data. Such color map estimation can be straightforward for a color map implemented in a subspace orthogonal to the grayscale of the CT image. In other color map subspaces, such estimation is more complex and generally cannot be uniquely solved.

[0090] In practice, the most commonly used RGB color map represents connected curves (e.g., broken lines) in RGB space. By assuming a general linear color map (or a set of multiple linear maps), some suitable statistical methods for estimation of vector space basis sets (e.g., based on independent component analysis (ICA)) can be applied to estimate the color map as basis vectors from the RGB values ​​of image pixels. If the color map must be assumed to be nonlinear, some more sophisticated nonlinear vector space estimation methods are needed.

[0091] In the general case, solving the image unblending problem may be impractical when no assumptions can be made about the underlying colormaps (or other image properties). However, when more information is given (e.g., one or both colormaps may be known, or the subspace of the colormaps may be provided exactly), image blending becomes more tractable.

[0092] Evaluating the conversion The problem of finding the corresponding cross-sectional slice for a 2D slice image in a 3D volume can be formulated as a slice-to-volume registration problem. Given a 2D slice image S and a 3D volume V, the slice-volume registration method operates to identify a 2D-2D transformation function T̂ and a plane P̂[V] (i.e., a 2D slice from volume V).

[0093] In the most general case, T̂ and P̂ minimize the following objective function: T^,P^=argmin T, P C(S T(x),P[V](x))+ R(T,P)

[0094] where: T^ is the "optimal" transformation, P^ is the "optimal" surface, C is the image similarity function, R is a regularization term.

[0095] The mapping T may be rigid, affine, or non-rigid. The image similarity function C is a quantitative assessment of the similarity between the transformed 2D image and the corresponding slice in the 3D volume V.

[0096] Various types of image similarity functions C can be used, for example, a similarity function defined using only intensity values ​​in S and V (e.g., normalized cross-correlation, mutual information), or a similarity function defined using only geometric landmarks, or a combination of both image intensity values ​​in S and V and geometric landmarks.

[0097] A regularization term R can be used to make the problem well-posed. The regularizer R may impose constraints on the solution (e.g., minimize out-of-plane deformations, minimize deformation magnitudes, impose elasticity constraints, etc.).

[0098] Alternatively, the problem of finding corresponding cross-sectional slices for a 2D image in a 3D volume can be formulated as an iterative 2D registration problem in which 2D images are iteratively aligned with slices of the 3D volume. The best or optimal corresponding slice can be identified by the index of the slice found to have the lowest value of the similarity function.

[0099] It will therefore be appreciated that the steps of identifying a representative slice and evaluating a transformation may be combined or performed separately.

[0100] By implementing the method according to the one described above, the configuration facilitates the user's continued interaction with the new imaging data combined with information from the summary. The user may interact with the output of some configurations, for example, via a graphical user interface (GUI). The configuration allows direct user interaction with at least one of a 2D image or multiple images provided as a summary of a previous or planned treatment or study, or a transformed image in which an equivalent plane or slice of a 3D imaging dataset is identified as equivalent to the 2D image or multiple images, or a transformed additional information image containing information about the 2D image or multiple images, including, for example, geometric features of the delivered dose and / or transformed measurements (lines, regions of interest). The user may: The original 2D image, an equivalent transformed image in the form of a plane or slice of the 3D imaging dataset that has been assessed as equivalent to one or more 2D images; The transformed additional information image features contained in the original 2D image, The user may be able to interact with the various available 2D images, including the 2D overlays, to activate or deactivate them.

[0101] Thus, configurations and implementations may provide systems and methods for supporting tools for visualizing additional information, such as one or more geometric features, measurements, annotations, or the like, related to previous findings and / or treatments associated with newly acquired imaging datasets of the same region of interest. The additional information may form part of a representative 2D image or set of images that form part of a treatment or findings summary report in an electronic medical record system. The 2D image or set of images may comprise one or more files stored in a standard image format, such as jpeg or a similar format.

[0102] The configuration may enable a user, for example a medical imaging professional, to make a more rapid assessment of newly acquired imaging information based on a combination of the newly acquired imaging information with a summary report of previous findings or treatment provided in relation to one or more representative slices or planes of the region of interest.

[0103] Example 1: Treatment Summary Figure 2 shows a schematic representation of the inputs and outputs of the method according to one configuration. Having generally described the method, a specific implementation relating to the application of such a method in the case of representative 2D images from radiation treatment or 2D images with annotations from a previous tumor assessment exam (as described in relation to Figure 3) will now be described in more detail.

[0104] 2 shows a representative 2D image 2000 that may be extracted from a storage medium. The 2D image may include a portion of a patient report, a screen grab, a portion of an electronic medical record (EMR), etc. In this case, the 2D image may include a treatment plan summary. Image 2000 includes a composite image formed from base image data and a treatment plan for radiation therapy. Treatment isodose lines 2100 can be considered to form part of image 2000. The base image includes one or more features 2200 of the region under investigation.

[0105] According to the construction method, a representative 2D RGB image 2000 (e.g., of a previously acquired CT containing a color dose distribution) is decomposed into (i) grayscale slices (S) representing the planning image and (ii) color geometric dose features (GDFs). For example, color isodose lines can be extracted from the representative 2D RGB image using a simple RGB decomposition of the 2D RGB image and by detecting pixels with unequal R, G, and B values. Transparent color patches of the alpha-blended image can be extracted using appropriate mathematical optimization techniques.

[0106] 2 shows a schematic representation of a volume of 3D imaging data 2500 available relating to a region under investigation. The volume of image data 2500 may comprise, for example, a series of image "slices" obtained via a suitable CT or MRI technique.

[0107] According to the construction method, the base image data from the 2D image 2000 and the 3D volume of tracked images are processed so that a transformation T can be calculated. T maps the base image data S for the 2D image 2000 to the 3D volume of tracked images (V) 2500. T is calculated based on determining a plane P in V that can be considered to correspond to the representative slice S. The transformation T can be determined using slice-to-volume registration or simple 2D-to-2D registration, each technique operating with the goal of determining the index of the slice in V that best matches S. It will be appreciated that it may be impossible to find a slice in V 2500 that perfectly matches a feature 2200 of the base image data of the 2D image 2000. This may be because the acquisition of the 2D image and the 3D volume of image data was separated in time, resulting in changes in the feature 2200. In the example shown in FIG. 2, for example, additional feature 2600 is present. Nevertheless, according to the construction, a best-fit slice or plane of the image data 2500 is found.

[0108] Once calculated, T can then be applied to the representative 2D color geometric dose features 2100 (or other annotations) of the 2D image 2000. That is, T(GDF) can be calculated.

[0109] T(GDF) can be overlaid on the determined plane or slice P, as shown in FIG. 2 as a newly created composite image 2700. The newly created 2D composite image 2700 allows visualization of known dose features for the 2D image 2000 on P, which is a representative slice of the 3D volumetric image data 2500 that corresponds to the base image of the 2D image 2000. The T(GDF) may contain additional annotations or other information in addition to the transformation 2800 of the dose information isodose line 2200, and thus overlaying the T(GDF) on P may support labeling of the determined plane or slice P with the planned dose levels and / or those additional annotations or other information.

[0110] The configuration may allow a user, e.g., a radiologist, to view geometric dose features corresponding to a representative CT planning image overlaid on newly acquired imaging information. For example, the configuration may allow a user to label P images forming part of the newly acquired imaging data according to the previously planned and applied treatment doses. The labeling may allow the radiologist to distinguish regions to be treated at various dose levels (indicated by isodose lines). Overlaying the isodose lines of previous treatments on follow-up images can improve the radiologist's decision-making.

[0111] Example 2: Previous Check While examples have been described in the context of radiation therapy applied to a patient, representative 2D images from a prior tumor assessment examination or imaging event can be treated similarly. Figure 3 shows a schematic of the inputs and outputs of the method according to one configuration. According to such an approach, a 2D summary image of a prior examination or subject assessment is decomposed into a base image and an additional information image, where the additional information image includes detected and extracted annotations, including at least one of text, arrows, lines, colored region of interest (ROI) contours, or similar features of potential interest.

[0112] 3 shows a representative 2D image 3000 that may be extracted from a storage medium. The 2D image may include a portion of a patient report, a screen grab, a portion of an electronic medical record (EMR), etc. In this case, the 2D image may include image annotations 3100. The image 3000 includes a composite image formed from base image data and additional information in the form of annotations 3100. The base image includes one or more features 3200 of the area under investigation.

[0113] According to one method, a representative 2D RGB image 3000 (e.g., of a previously acquired CT slice containing additional information in the form of image annotations) is decomposed into (i) a grayscale slice (S) representing the planning image and (ii) an additional information image (ADD). For example, colored annotations can be extracted from the representative 2D RGB image using a simple RGB decomposition of the 2D RGB image and detecting pixels with unequal R, G, and B values.

[0114] 3 shows a schematic representation of a volume of 3D imaging data 3500 available relating to a region under investigation. The volume of image data 3500 may comprise, for example, a series of image "slices" obtained via a suitable CT or MRI technique.

[0115] According to the construction method, the base image data from the 2D image 3000 and the 3D volume of tracked images are processed so that a transformation T can be calculated. T maps the base image data S for the 2D image 3000 to the 3D volume of tracked images (V) 3500. T is calculated based on determining a plane P in V that can be considered to correspond to a representative slice S. The transformation T can be determined using slice-to-volume registration or simple 2D-to-2D registration, each technique operating with the goal of determining the index of the slice in V that best matches S. It will be appreciated that it may be impossible to find a slice in V 3500 that perfectly matches a feature 3200 of the base image data of the 2D image 3000. This may be because the acquisition of the 2D image and the 3D volume of image data was separated in time, causing changes in the feature 3200. According to the construction, the best-fit slice or plane of the image data 3500 is found.

[0116] Once calculated, T can then be applied to the representative 2D additional information 3100 (or other annotations) of the 2D image 3000. That is, T(ADD) can be calculated.

[0117] T(ADD) can be overlaid on the determined plane or slice P, as shown in FIG. 2 as a newly created composite image 3700. The newly created 2D composite image 3700 allows visualization of dose features known in relation to the 2D image 3000 on P, which is a representative slice of the 3D volumetric image data 3500 that corresponds to the base image of the 2D image 3000. T(ADD) may have additional annotations or other information in addition to the additional information transform 3800, and thus overlaying T(ADD) on P may support labeling of the determined plane or slice P with the additional annotations or other information known in relation to the 2D image 3000.

[0118] The configuration may allow a user, e.g., a radiologist, to view geometric dose features corresponding to representative 2D images of a previous examination or evaluation overlaid on newly acquired imaging information. For example, the configuration may allow a user to label P images forming part of the newly acquired imaging data according to previously known annotations or measurements. Labeling may allow a radiologist to assess the development of disease or changes to the area under investigation, thus supporting improved decision-making by medical professionals.

[0119] FIG. 4 is a schematic representation of a system for mapping additional information about a 2D representation of a region of interest onto a 3D representation volume of the region of interest to generate a new composite 2D representation of the region of interest as described or otherwise contemplated herein.

[0120] The system 4000 includes one or more of a processor 4100 , a memory 4200 , a user interface 4300 , a communication interface 4500 , and a storage device 4600 interconnected via one or more system buses 4700 .

[0121] In some embodiments, such as when the system is part of or in communication with imaging hardware or an imaging platform, the hardware may include additional imaging hardware (not shown). It will be understood that Figure 4 constitutes an abstraction of system 4000, and that the actual organization of the components of system 4000 may differ from that shown.

[0122] According to one embodiment, system 4000 includes a processor 4100 that can execute instructions stored in memory 4200 or storage device 4600 or otherwise process data. Processor 4100 may be configured to perform one or more steps of the method illustrated generally in FIG. 5 and may include a module described or otherwise contemplated herein. Processor 4100 may be formed from one or more modules and may include, for example, memory 4200. Processor 4100 may take any suitable form, including, but not limited to, a microprocessor, a microcontroller, multiple microcontrollers, a circuit, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a single processor, or multiple processors.

[0123] The memory 4200 may take any suitable form, including non-volatile memory and / or RAM. The memory 4200 may include various memories, such as, for example, cache or system memory. Thus, the memory 4200 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read-only memory (ROM), or other similar memory devices. The memory may store, among other things, an operating system. The RAM is used by the processor for temporary storage of data. According to one embodiment, the operating system may include code that, when executed by the processor, controls the operation of one or more components of the system 4000. It will be apparent that in embodiments in which the processor implements one or more of the functions described herein in hardware, software described as corresponding to such functions in other embodiments may be omitted.

[0124] The user interface 4300 may comprise one or more devices for enabling communication with a user, such as an administrator, imaging or medical professional. The user interface may comprise any device or system that allows information to be transmitted and / or received and may include a display, mouse, and / or keyboard for receiving user commands. In some embodiments, the user interface 4300 may include a command line interface or a graphical user interface that may be presented to a remote terminal via the communication interface 4500. The user interface 4300 may be collocated with one or more other components of the system or may be located remotely from the system and communicate via a wired and / or wireless communication network.

[0125] The communication interface 4500 may include one or more devices for enabling communication with other hardware devices. For example, the communication interface 4500 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. In addition, the communication interface 4500 may implement a TCP / IP stack for communicating according to the TCP / IP protocol. Various alternative or additional hardware or configurations for the communication interface 4500 will be apparent.

[0126] The storage device 4600 may include one or more machine-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, or similar storage media. In various embodiments, the storage device 4600 may store instructions for execution by the processor 4100 or data on which the processor 4100 may operate. For example, the storage device 4600 may store an operating system for controlling various operations of the system 4000.

[0127] Although system 4000 is shown as including one of each of the described components, various components may be duplicated in various embodiments. For example, processor 4100 may include multiple microprocessors configured to independently perform the methods described herein, or configured to perform steps or subroutines of the methods described herein, such that the multiple processors cooperate to achieve the functionality described herein. Furthermore, when system 4000 is implemented in a cloud computing system, various hardware components may reside in separate physical systems. For example, processor 4100 may include a first processor in a first server and a second processor in a second server. Many other variations and configurations are possible.

[0128] According to one embodiment, the processor 4100 includes one or more modules for performing one or more functions or steps of the methods described or otherwise contemplated herein. For example, the processor 4100 may include a decomposition module 4700, a slice evaluation module 4750, a transformation module 4800, and / or an image generation module 4850.

[0129] FIG. 5 is a flowchart of a method 5000 for mapping additional information about a 2D image of a region of interest onto a 3D image volume of the region of interest to generate a new composite 2D image of the region of interest.

[0130] In step 5100, a system for such mapping and image generation is provided. The system may be any of the systems described or otherwise contemplated herein and may have any of the components or modules described or otherwise contemplated herein.

[0131] In method step 5200, one or more 2D composite images of a region of interest are received by or provided to the system.

[0132] In step 5300 of the method, a 3D image volume of a region of interest is received by or provided to the system.

[0133] In step 5400 of the method, the system is configured to decompose the 2D composite image into at least a base image and an additional image containing the additional information, for example through steps performed by a decomposition module.

[0134] In method step 5500, the system is configured to identify a representative 2D slice of the 3D image volume corresponding to the base image based on an evaluation of the base image and the 3D image volume, for example, via steps performed by a slice evaluation module.

[0135] In method step 5600, the system is configured to evaluate a transformation of the base image that maps the base image to the representative 2D slices, for example, via steps performed by a transformation module.

[0136] In step 5700 of the method, the system is configured to combine the representative 2D slice with an image formed by applying the evaluated transformation to the additional image to generate a new composite 2D image of the region of interest, for example, through steps performed by an image generation module.

[0137] Those skilled in the art will readily recognize that the steps of the various methods described above can be performed by a programmed computer. As used herein, some embodiments are also intended to cover program storage devices, e.g., digital data storage media, that are machine-readable or computer-readable and encode a machine-executable or computer-executable program of instructions, which perform some or all of the steps of the above-described methods. The program storage device may be, for example, a digital memory, a magnetic storage medium such as a magnetic disk or magnetic tape, a hard drive, or an optically readable digital data storage medium. Embodiments are also intended to cover computers programmed to perform the steps of the above-described methods. The term "non-transitory," as used herein, is a limitation of the medium itself (i.e., tangible, not a signal), as opposed to a limitation to data storage persistence (e.g., RAM vs. ROM).

[0138] While exemplary embodiments of the present invention have been described in the preceding paragraphs with reference to various examples, it should be understood that modifications to the examples given can be made without departing from the scope of the invention as claimed.

[0139] Features set forth in the foregoing description may be used in combinations other than those expressly set forth.

[0140] Although functions are described with reference to particular features, those functions may be performed by other features, whether or not described.

[0141] Although features have been described with reference to particular embodiments, these features may also be present in other embodiments, whether or not described.

[0142] Although efforts have been made in the foregoing specification to draw attention to those features of the invention which are considered to be particularly important, it will be understood that the applicant claims protection for any patentable feature or combination of features mentioned above and / or shown in the drawings, whether or not specific emphasis is given to it.

[0143] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprise" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. A computer program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, provided together with or as part of other hardware, but also in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. An apparatus configured to map additional information relating to a 2D medical image of a region of interest onto a 3D medical image volume of said region of interest, comprising: at least one processor; When executed by the at least one processor, the apparatus comprises at least: receiving a 2D synthetic medical image of the region of interest; receiving a 3D medical image volume of the region of interest; The 2D synthetic medical image comprises at least Base image, and an additional image containing said additional information; Decompose it into identifying a representative 2D slice of the 3D medical image volume corresponding to the base image based on evaluation of the base image and the 3D image volume; evaluating a transformation of the base image that maps the base image to the representative 2D slice; combining the representative 2D slice with an image formed by applying the estimated transformation to the additional image to generate a new composite 2D medical image of the region of interest. at least one memory for storing instructions; An apparatus having:

2. The apparatus of claim 1 , wherein the 2D medical image of the region of interest comprises a color image.

3. The apparatus of claim 1 or claim 2, wherein the composite 2D medical image comprises a 2D summary image of a 3D medical image volume representing the region of interest.

4. The apparatus of claim 1 , wherein the synthetic 2D medical image includes additional information regarding a treatment or procedure performed on the region of interest.

5. The apparatus according to claim 1 , wherein the additional information comprises a geometric distribution of the therapeutic dose to be applied to the region of interest.

6. The device of claim 5 , wherein the treatment comprises radiation therapy.

7. 7. The apparatus of claim 5 or claim 6, wherein the geometric distribution comprises isodose lines or color map dose information.

8. The apparatus of claim 1 , wherein the additional information comprises one or more annotations comprising an outline of a feature of interest, or a marker of a feature of interest, or a measurement of a feature of interest.

9. The apparatus of claim 1 , wherein decomposing the 2D composite medical image comprises decomposing the 2D composite image into at least a grayscale base image and an additional image containing the additional information.

10. 10. The apparatus of claim 1, wherein decomposing the 2D composite medical image comprises analyzing pixels of the 2D composite medical image to determine the grayscale base image and an additional image including the additional information.

11. 11. The apparatus of claim 1, wherein identifying a representative 2D slice of the 3D medical image volume corresponding to the base image comprises evaluating one or more planar images forming the 3D image volume against the base image, and selecting a planar image from the evaluated one or more planar images that best matches the base image.

12. 12. The apparatus of claim 1, wherein identifying the representative 2D slice of the 3D medical image volume and evaluating a transformation of the base image that maps the base image to the representative 2D slice comprises using a slice-to-volume or 2D-to-2D registration image registration technique.

13. 13. The apparatus of claim 1, wherein evaluating a transformation of the base image that maps the base image to the representative 2D slice comprises evaluating one or more differences between the images resulting from one or more of rotation, translation, deformation, and scaling.

14. 1. A computer-implemented method for mapping additional information about a 2D image of a region of interest onto a 3D medical image volume of said region of interest, comprising: receiving a 2D composite medical image of the region of interest and a 3D medical image volume of the region of interest; The 2D synthetic medical image comprises at least Base image, and an additional image containing said additional information; and identifying a representative 2D slice of the 3D medical image volume that corresponds to the base image based on evaluation of the base image and the 3D medical image volume; evaluating a transformation of the base image that maps the base image to the representative 2D slice; combining the representative 2D slice with an image formed by applying the estimated transformation to the additional image to generate a new composite 2D medical image of the region of interest; A method comprising:

15. A computer program operable to perform the method of claim 14 when executed on a computer.

Images