System and method for synchronous navigation and annotation of a set of interrelated heterogeneous medical images

Abstract

The present disclosure relates to a method for synchronous navigation and annotation of a set of interrelated heterogeneous medical images.SOLUTION: The method includes correlating the first and second sets of sequential tomographic medical images of the subject based on the first metadata and the second metadata to generate a plurality of correlated image pairs, superimposing each correlated image pair to generate a fused image corresponding to the correlated image pair, selectively and sequentially rendering the correlated image pair, the fused image, and / or a third tomographic medical image of the subject, and selectively rendering a visual indication of a pixel of interest and / or a region of interest for the correlated image pair, the fused image, and / or the third tomographic medical image. A machine learning algorithm is used to identify pixels in the medical image that are associated with the medical condition. A graphical annotation is rendered at or near the location of the pixel.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to medical imaging, and more particularly to a system and method for synchronized navigation and annotation of a set of interrelated heterogeneous medical images. [Background technology]

[0002] Medical imaging is the technology and process of capturing images inside the body for clinical analysis and intervention, as well as for visual display of organ and tissue function (physiology). Medical imaging seeks to reveal internal structures hidden beneath the skin and bone, as well as diagnose and treat disease. Medical imaging also builds a database of normal anatomy and physiology and allows for the identification of abnormalities. Imaging of excised organs and tissues can also be performed for medical purposes. Medical images obtained from imaging devices such as computed tomography (CT), positron emission tomography (PET), and magnetic resonance imaging (MRI) can be used to make medical diagnoses. Whole-body maximum intensity projection (MIP) imaging can also be used for medical diagnoses.

[0003] However, the amount of information presented to medical professionals from medical images and computer-aided diagnosis (CAD) analysis is increasing as the performance of imaging devices and CAD analysis improves. When large numbers of images are acquired and subjected to analysis, the navigation and annotation of the results and images can complicate the medical professional's review task. Summary of the Invention [Problem to be solved by the invention]

[0004] Thus, there is a need for a method and apparatus for navigating and annotating a set of interrelated, heterogeneous medical images. The present invention addresses this need, as well as other needs. [Means for solving the problem]

[0005] Accordingly, one aspect of the present invention provides a computer system for synchronized navigation and annotation of heterogeneous interrelated medical images, the system comprising a computer processor, a computer-readable storage medium, and program instructions stored on the computer-readable storage medium for execution by the processor, the program instructions including: correlating a first set of sequential tomographic medical (STM) images of a subject with a second set of STM images of the subject based on first metadata and second metadata to generate a plurality of interrelated image pairs.

[0006] A first set of STM images each include first metadata, depict a first anatomical plane, and are captured by a first medical imaging technique (MIT), and a second set of STM images each include second metadata, depict the first anatomical plane, and are captured by a second medical imaging technique. The program instructions further include: overlaying each of the interrelated image pairs to generate a plurality of fused images, each corresponding to one of the interrelated image pairs; selectively and sequentially rendering, when input is received from the input / output device, one or more of the interrelated image pairs, the fused image, and a third tomographic medical (TTM) image of the subject by the display, wherein the third tomographic medical image depicts a second anatomical plane and is captured by a third medical imaging technique; and selectively rendering, when input is received from the input / output device, a visual indication of one or more of a pixel of interest and a region of interest by the display for one or more of the interrelated image pairs, the fused image, and the third tomographic medical image.

[0007] The program instructions may further include program instructions for using a machine learning algorithm to identify pixels in one of the first STM images or one of the second STM images associated with a medical condition and for drawing, by the display, a graphical annotation at the pixel location of the pixel. The machine learning algorithms are trained on predetermined medical images, each depicting the presence or absence of a medical condition within that anatomical region. The presence of the medical condition is determined by a pixel value within the anatomical region being greater than or less than a predetermined value. The program instructions may further include program instructions for selectively drawing, by the display, a second graphical annotation surrounding the pixel location, thereby visually representing the region of interest.

[0008] Another aspect of the invention provides a computer-implemented method for synchronized navigation and annotation of a set of heterogeneous interrelated medical images. The method uses first and second metadata to correlate a first set of serial tomographic medical (STM) images of a subject with a second set of STM images of the subject to generate a plurality of interrelated image pairs. The first set of STM images each include first metadata, depict a first anatomical plane, and are captured by a first medical imaging technique. The second set of STM images each include second metadata, depict the first anatomical plane, and are captured by a second medical imaging technique. Each interrelated image pair is registered to generate a plurality of fused images, each corresponding to one of the interrelated image pairs.

[0009] When input is received from the input / output device, one or more of the correlated image pair, the fused image, and a third tomographic medical image are selectively and sequentially rendered. The third tomographic medical image depicts a second anatomical plane and is captured by a third medical imaging technique. When input is received from the input / output device, a visual representation of one or more of a pixel of interest and a region of interest is selectively rendered by the display for one or more of the correlated image pair, the fused image, and the third tomographic medical image. The one or more of the pixel of interest and the region of interest are associated with a medical condition.

[0010] A particular aspect of the present invention provides a computer program product comprising a computer-readable storage medium having program code embodied therein that is executable by a processor to correlate a first set of serial tomographic medical (STM) images of a subject with a second set of STM images of the subject based on first and second metadata to generate a plurality of correlated image pairs, the first set of STM images each including first metadata, depicting a first anatomical plane, and being captured by a first medical imaging technique, and the second set of STM images each including second metadata, depicting the first anatomical plane, and being captured by a second medical imaging technique. The processor executes the program code to overlay each correlated image pair to generate a plurality of fused images, each corresponding to one of the correlated image pairs.

[0011] The processor executes program code for, when input is received from the input / output device, selectively and sequentially rendering, by the display, one or more of the correlated image pair, the fused image, and a third tomographic medical image of the subject. The third tomographic medical image depicts a second anatomical plane and is captured by a third medical imaging technique. The processor executes program code for, when input is received from the input / output device, selectively rendering, by the display, a visual representation of one or more of a pixel of interest and a region of interest for one or more of the correlated image pair, the fused image, and the third tomographic medical image. The one or more of the pixel of interest and the region of interest are associated with a medical condition.

[0012] Other aspects, advantages and novel features of the invention will be described in part below and in part will become apparent by practice of the invention when considered in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0013] The invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1A] FIG. 1A is a block diagram illustrating an information processing environment for synchronization, navigation, and annotation of a set of interrelated heterogeneous medical images, according to some embodiments. [Figure 1B] FIG. 1B illustrates transaxial, coronal, and sagittal planes of a subject, according to another embodiment. [Figure 2] FIG. 2 is a flowchart illustrating the operational steps of an information processing program on a computing device within the information processing environment of FIG. 1 for synchronization, navigation, and annotation of a set of interrelated heterogeneous medical images, according to certain embodiments. [Figure 3] FIG. 3 illustrates a graphical user interface (GUI) for rendering medical images within the information processing environment of FIG. 1, according to yet another embodiment. [Figure 4]4 shows a block diagram of components of a computing device executing an information processing program, according to some embodiments. Corresponding reference characters indicate corresponding parts throughout the drawings. The examples set forth herein represent presently preferred embodiments of the invention, and such examples should not be construed as limiting the scope of the invention in any way. DETAILED DESCRIPTION OF THE INVENTION

[0014] As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may be generally referred to herein as a "circuit," "module," or "system." Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code / instructions embodied thereon.

[0015] Any combination of computer-readable media may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the context of this specification, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0016] The program code embodied on the computer readable medium may be transmitted using any suitable medium, such as, but not limited to, wireless, wired, fiber optic cable, or RF, or any suitable combination thereof.

[0017] Computer program code for carrying out operations of aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, or C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., over the Internet using an Internet Service Provider).

[0018] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine. These instructions, when executed by the processor of the computer or other programmable data processing apparatus, generate means for implementing the functions / acts specified in the flowcharts and / or blocks of the block diagrams.

[0019] These computer program instructions may be stored on a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device to function in a particular way, such that the instructions stored on the computer-readable medium produce an article of manufacture including instructions that implement the functions / acts specified in the flowcharts and / or block diagrams.

[0020] The computer program instructions, when loaded into a computer, other programmable data processing apparatus, or other device, cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process such that the instructions, when executed on the computer or other programmable apparatus, provide a process for implementing the functions / acts specified in the flowcharts and / or block diagram blocks.

[0021] Medical imaging is the technology and process of capturing images inside the body for clinical analysis and intervention, as well as for visual display of organ and tissue function (physiology). Medical imaging seeks to reveal internal structures hidden beneath the skin and bone, as well as diagnose and treat disease. Medical imaging also builds a database of normal anatomy and physiology and allows for the identification of abnormalities. Imaging of excised organs and tissues can also be performed for medical purposes. Medical images obtained from imaging devices such as computed tomography (CT), positron emission tomography (PET), and magnetic resonance imaging (MRI) can be used to make medical diagnoses. Whole-body maximum intensity projection (MIP) imaging can also be used for medical diagnoses.

[0022] However, the amount of information provided to medical professionals from medical images and computer-aided diagnosis (CAD) analysis is increasing as the performance of imaging devices and CAD analysis improves. When large numbers of images are acquired and subjected to analysis, the navigation and annotation of the results and images can complicate the review task for medical professionals. Therefore, there is a need for a method and apparatus for navigating and annotating a set of heterogeneous, interrelated medical images. The present invention addresses this need, as well as others.

[0023] Referring initially to FIG. 1A, a block diagram illustrating an information processing environment (generally 100) for capturing, storing, and / or analyzing medical images according to an embodiment of the invention is provided. The information processing environment may include one or more of an imaging device 102, one or more servers (e.g., an image server 105 and a report server 107), and computing devices (e.g., a medical workstation (WS) 104 and an analysis WS 103) communicatively connected via a network 109. Two or more of the above components may be combined into a single unit. The network 109 may be comprised of wired, optical, and / or wireless radio frequency-based communication network technologies that may be arranged in a variety of network topologies. Typically, the network 109 may be any set of digital interconnections that allows computing devices to use a common communication protocol to communicate with each other.

[0024] The imaging device 102 may be one or more imaging devices including an emitter 130 and a sensor 120. The imaging device 102 is configured to scan the subject's body by exposing the subject to a signal emitted by the emitter 130 and capturing this signal (or signal reflections) with the sensor 120, thereby obtaining detailed internal images of the subject's body (e.g., images of anatomical structures and physiological processes within the body). Medical images generated by the imaging device 102 may be stored in the image database 106 and / or other databases communicatively connected to the network 109. For example, the imaging device 102 is configured to generate serial tomographic images of the subject. The imaging device 102 may capture tomographic images along one or more anatomical planes of the subject (e.g., transaxial, coronal, sagittal, median, parasagittal, etc.).

[0025] Turning now to FIG. 1B , for example, each subject 142 (e.g., a human, mammal, other biological entity, or portion thereof) has a first body axis, the z-axis, that extends from the pelvis through the center of the body to the crown of the head. The transverse axial plane 140, also known as the axial plane or transverse plane, lies perpendicular to the z-axis in the x-y plane and can occur anywhere along the z-axis. The coronal (or frontal) plane 150 extends left-right in the x-z plane and can occur anywhere along the y-axis. The sagittal plane 160 extends front-to-back in the y-z plane and can occur anywhere along the x-axis. When the body is divided into two halves, left and right, the sagittal plane 160 is commonly referred to as the midsagittal plane. The x- and y-axes lie within the transverse axial plane. Although medical image display systems allow the operator to view any anatomical plane, transaxial, coronal, and sagittal planes are routinely used in medical imaging to visualize anatomical structures and description of findings.

[0026] Applicable signals emitted by emitter 130 include, but are not limited to, strong magnetic fields, magnetic field gradients, radio waves, x-rays, and ionizing radiation. Imaging device 102 preferably generates heterogeneous, continuous, tomographic images of the subject. Imaging device 102 may generate medical images using one or more medical imaging techniques, including, but not limited to, x-rays, computed tomography (CT) scans, magnetic resonance imaging (MRI), positron emission tomography (PET), maximum intensity projection (MIP), and / or other medical imaging techniques capable of generating internal images of the subject's body or portions thereof.

[0027] Each of the servers may include a software program that provides an installed database management system (DBMS). In some embodiments, the image server 105 and the report server 107 may be included in one or more computing devices. The servers may include one or more storage media (e.g., flash memory, a solid-state drive (SSD), or a hard disk drive (HDD)) for storing information. For example, the image server 105 may include an image database (DB) 106 for storing at least medical images. The report server 107 may include a report DB 108 for storing at least processed reports. The medical WS 104 may be used by medical personnel (e.g., doctors and nurses) to review at least medical images, view analysis reports, and / or generate electronic medical records.

[0028] The Medical WS 104 may send requests to view medical images to the Image Server 105, display the received medical images, send requests to view associated analysis reports to the Report Server 107, and / or display the received analysis reports. The Medical WS 104 may perform the above-described processes by executing software programs for each process. The steps described in this disclosure may be performed by one or more control circuits and / or processors. The analysis reports may include images analyzed by an information processing program 110 included in the Analysis WS 103 (or other computing device communicatively connected to the network 109).

[0029] The analysis WS 103 is a computing device that may be configured to analyze or interpret one or more medical images (e.g., stored in the image DB 106 or other data store communicatively connected to the network 109) as described in this disclosure. The analysis WS 103 may be used by a user (e.g., a medical professional) to analyze one or more medical images and / or generate an analysis report. The analysis WS 103, or another processor communicatively connected to the network 109, may perform steps defined by an information processing program 110. As described in more detail below, the information processing program 110 includes program code that enables synchronized navigation and annotation of a set of heterogeneous, interrelated medical images. The information processing program 110 may be stored in a database communicatively connected to the analysis WS 103. The information processing program 110 may also be stored in one or more databases communicatively connected to the network 109.

[0030] As shown in Figure 2, a flowchart is provided. The flowchart illustrates operational steps of an information processing program 110 on a computing device within the information processing environment of Figure 1 for synchronizing, navigating, and annotating a set of interrelated heterogeneous medical images in accordance with an embodiment of the present invention. In step 210, a first set of STM images is correlated with a second set of STM images based on the first metadata and the second metadata, thereby generating a plurality of interrelated image pairs. Applicable metadata may include identification information such as, but not limited to, an image identification (ID) for identifying the medical image, Digital Imaging and Communications in Medicine (DICOM) information, a tomography ID assigned to each cross-sectional image included in the medical images, a subject ID for identifying the subject, and a study ID for identifying the study.

[0031] Additionally, applicable metadata may include, but is not limited to, imaging information regarding imaging method, imaging conditions, imaging sequence information, anatomical view information, and image capture date and time. The cross-sectional medical images provided by the present disclosure can be digitally "stacked" to provide a user with a sequential view of a subject's body or portion thereof along an anatomical plane. The metadata is used to correlate a set of medical images of a subject according to their sequential positions, regardless of the anatomical locations at which they were captured.

[0032] A first set of STM images each include first metadata, depict a first anatomical plane, and are captured by a first medical imaging technique. A second set of STM images each include second metadata, depict the first anatomical plane, and are captured by a second medical imaging technique.

[0033] For example, the first set of STM images and the second set of STM images are retrieved from image database 106. Still referring to FIG. 3 , the first set of STM images may be transaxial images generated by CT scans, including image 306. CT scan images readily identify anatomical structures such as the pelvis, ribs, spine, liver, esophagus, and subcutaneous adipose tissue. The second set of STM images may be transaxial images generated by PET, including image 311. PET scan images readily identify measured changes in metabolic processes and other physiological activities, including blood flow, local chemical composition, and absorption (e.g., to identify tumors, metastases, brain lesions, vascular disease, etc.). The first and second metadata are used to positionally correlate each transaxial CT image in the first set of STM images with the transaxial PET images in the second set of STM images.

[0034] In step 220, each interrelated image pair (e.g., image 306 and image 311) is overlaid to generate multiple fused images, each corresponding to one of the interrelated image pairs. For example, fused image 316 is interrelated with the image pair consisting of image 306 overlaid on image 311. FIG. 3 illustrates a graphical user interface (GUI) (generally 300) for rendering medical images within the information processing environment of FIG. 1, according to another embodiment. GUI 300 is rendered on a display (e.g., display 620) and may include one or more display panes (hereinafter "panes") in which the medical images are rendered. GUI 300 may include, but is not limited to, pane 305, pane 310, pane 315, and pane 320.

[0035] In step 230, when input is received from an input / output device (see external component 600 below), one or more of each cross-correlated image pair, each fused image, and a third tomographic medical (TTM) image of the subject (e.g., image 321) are selectively rendered by a display (e.g., display 620). The third cross-sectional medical image depicts a second anatomical plane and is captured by a third medical imaging technique. Image 321 is an MIP image depicting a coronal view of the subject. As shown, the cross-correlated image pairs of images 306 and 311 are rendered in panes 305 and 310, respectively. A fused image 316 of images 306 and 311 is rendered in pane 315. Image 321 is rendered in pane 320. Thus, a user can use GUI 300 to view the results of three different medical imaging techniques captured by two different anatomical planes.

[0036] In another embodiment, GUI 300 includes annotation icon 350 and / or annotation icon 360 (depicted in pane 305, pane 310, and pane 315). Annotation icon 350 represents the anatomical location of image 321 relative to fused image 316, image 306, and image 311. Annotation icon 360 represents the anatomical location of fused image 316, image 306, and / or image 311 relative to image 321. A user can positionally manipulate annotation icon 350 via one or more of external components 600 to continuously change the third tomographic medical image depicted in pane 320. Alternatively, a user can positionally manipulate annotation icon 360 up or down via one or more of external components 600 to continuously change the fused image, the first STM image, and / or the second STM image depicted in pane 315, pane 305, and pane 310, respectively.

[0037] In step 240, to implement step 230, when an input signal is received from an input / output device, one or more of the correlated image pairs and the fused image are sequentially rendered. For example, one or more panes of GUI 300 may include a GUI element 325 that allows a user to selectively and sequentially view the medical images rendered in one or more of the panes. In other words, a user can manipulate GUI element 325 to visually traverse along the anatomical planes through which the medical images were captured. In step 250, when an input is received from the input / output device, the display selectively renders a visual representation of one or more of a pixel of interest (e.g., pixel of interest 335) and a region of interest (e.g., region of interest 330) for one or more of the correlated image pairs, the fused image, and the third tomographic medical image. The pixel of interest and the region of interest represent regions of the image (e.g., pixel locations) where a medical condition may be depicted. Each pixel has a pixel location. A region of interest typically has one or more of a length, width, and / or height measured in pixels.

[0038] In certain embodiments, step 250 is performed using a machine learning algorithm. A machine learning algorithm is a type of mathematical model that is trained to predict or classify new data (e.g., medical images) from a given dataset (e.g., predetermined medical images depicting the presence or absence of a medical condition described above). During training, the learning algorithm iteratively adjusts the model's internal parameters to minimize the error of the prediction. In a preferred embodiment, the machine learning algorithm is trained with predetermined medical images, each depicting the presence or absence of a medical condition within its anatomical region. The presence of a medical condition can be determined by pixel values ​​within the anatomical region being greater than or less than a predetermined value (or range of values). In other words, image pixels, referred to as pixels of interest, having pixel values ​​greater than or less than a predetermined value (or range of values) are considered by the machine learning algorithm to be associated with the medical condition that the machine learning algorithm is trained to detect.

[0039] When step 250 is implemented to render a visual representation of a pixel of interest (e.g., an annotation icon or pixel color change), in step 260, a pixel in one of the first STM images or one of the second STM images associated with a medical condition is identified using a machine learning algorithm. The pixel has a pixel location in the associated medical image. GUI 300 draws a graphical annotation at the pixel location (or the nearest location). If the pixel is not identified, GUI 300 can draw a notice to the user conveying the absence of the pixel. In this example, the machine learning algorithm identifies a pixel of interest, and GUI 300 draws graphical annotation 335 at the corresponding pixel location (or the nearest location) in image 311. When step 250 is implemented to render a visual representation of a region of interest, in step 260, a second graphical annotation surrounding (or nearest located at) the pixel location is drawn. Thus, once the pixel is identified, GUI 300 draws graphical annotation 330 surrounding graphical annotation 335 located at the pixel location. Applicable graphical annotations include, but are not limited to, graphical colors, geometric shapes, and / or icons that can be used to visually identify pixels, pixel locations, pixels of interest, and / or regions of interest.

[0040] Because image 311 is correlated with image 306, GUI 300 also renders annotation icon 350 on image 306 and fused image 316 in their respective display panes. A user can positionally manipulate annotation icon 350 within pane 305, pane 310, and / or pane 315 to sequentially traverse each of the third cross-sectional medical images rendered in pane 320, thereby exploring a region of interest captured in the second anatomical plane. Alternatively, a user can positionally manipulate annotation icon 360 within pane 320 to sequentially traverse each of the medical images rendered in pane 305, pane 310, and / or pane 315, thereby exploring a region of interest captured in the first anatomical plane.

[0041] 4 shows a block diagram of components of analysis WS 103, medical WS 104, and / or imaging device 102, in accordance with an embodiment of the present invention. Data processing system 500 or 600 represents an electronic device capable of executing machine-readable program instructions. Data processing system 500 or 600 may represent a smartphone, computer system, PDA, or other electronic device. Examples of computing systems, environments, and / or configurations that may be represented by data processing system 500 or 600 include, but are not limited to, a personal computer system, a server computer system, a thin client, a thick client, a wearable computer, a handheld or laptop device, a multiprocessor system, a microprocessor-based system, a network PC, a minicomputer system, and a distributed cloud computing environment including any of the above systems or devices.

[0042] The analysis WS 103, the medical WS 104, and / or the imaging device 102 each include a set of internal components 500 and a set of external components 600, as shown in FIG. 4. Each of the set of internal components 500 may include one or more processors 520, one or more computer-readable RAMs 522 and one or more computer-readable ROMs 524 on one or more buses 526, one or more operating systems 528, and one or more computer-readable tangible storage devices 530. The information processing program 110 is stored in one or more of the respective computer-readable tangible storage devices 530 for execution by one or more of the respective processors 520 via one or more of the respective RAMs 522 (which typically include cache memory). In the embodiment shown in FIG. 4, each of the computer-readable tangible storage devices 530 may be a magnetic disk storage device of an internal hard drive. Alternatively, each of the computer-readable tangible storage devices 530 may be a semiconductor storage device such as ROM 524, EPROM, flash memory, or other computer-readable tangible storage device capable of storing computer programs and digital information.

[0043] The internal component 500 also includes a R / W drive or interface 532 for reading from / writing to one or more portable computer-readable tangible storage devices 636, such as a CD-ROM, a DVD, a memory stick, a magnetic tape, a magnetic disk, an optical disk, or a semiconductor storage device. The information processing program 110 can be stored in one or more of the portable computer-readable tangible storage devices 636, read from each R / W drive or interface 532, and loaded into each computer-readable tangible storage device 530.

[0044] Each set of internal components 500 also includes a network adapter or interface 536, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, a 3G or 4G wireless interface card, or other wired or wireless communication link. The information processing program 110 can be downloaded from an external computer to the analysis WS 103, the medical WS 104, and / or the imaging device 102 via a network (e.g., the Internet, a local area network, a wide area network) and each network adapter or interface 536. From the network adapter or interface 536, the information processing program 110 may be loaded into each computer-readable tangible storage device 530. The network may include copper wire, optical fiber, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers.

[0045] Each set of external components 600 may include a computer display monitor 620, a keyboard 630, and a computer mouse 634. The external components 600 may also include a touch screen, a virtual keyboard, a touchpad, a pointing device, and other human interface devices. The internal components 500 also include a device driver 540 to interface with the computer display monitor 620, the keyboard 630, and the computer mouse 634. The device driver 540, the R / W drive or interface 532, and the network adapter or interface 536 comprise hardware and software (stored in the storage device 530 and / or the ROM 524).

[0046] Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, or C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network ("LAN") or a wide area network ("WAN"), or may be connected to the external computer (e.g., over the Internet using an Internet Service Provider).

[0047] Based on the foregoing, a computer system, method, and program product according to the present invention have been disclosed. However, numerous modifications and substitutions can be made without departing from the scope of the present invention. Accordingly, the present invention has been disclosed by way of example only, and not by way of limitation.

Claims

1. 1. A computer system for synchronized navigation and annotation of a set of interrelated heterogeneous medical images, comprising: one or more computer processors; one or more computer-readable storage media; program instructions stored on the computer-readable storage medium for execution by at least one of the one or more computer processors; Equipped with The program instructions include: correlating a first set of serial tomographic medical images (STM images) of the subject with a second set of STM images of the subject based on the first metadata and the second metadata to generate a plurality of correlated image pairs; the first set of STM images each including the first metadata, depicting a first anatomical plane, and captured by a first medical imaging technique; a second set of STM images each including the second metadata, depicting the first anatomical plane, and captured by a second medical imaging technique; overlaying each of the plurality of interrelated image pairs to generate a plurality of fused images, each corresponding to one of the interrelated image pairs; selectively and sequentially rendering, upon receiving an input signal from an input / output device, one or more of the plurality of interrelated image pairs, the fused image, and a third tomographic medical image of the subject, the third tomographic medical image depicting a second anatomical plane and captured by a third medical imaging technique; selectively rendering, by the display, a visual representation of one or more of a pixel of interest and a region of interest for one or more of the plurality of interrelated image pairs, the fused image, and the third tomographic medical image when input is received from the input / output device, wherein the one or more of the pixel of interest and the region of interest are associated with a medical condition; a system including program instructions for:

2. 2. The system of claim 1, wherein one of the first medical imaging technique, the second medical imaging technique, and the third medical imaging technique is one of an X-ray, a computed tomography scan (CT scan), a magnetic resonance imaging scan (MRI scan), a positron emission tomography scan (PET scan), and a maximum intensity projection scan (MIP scan).

3. The system of claim 2 , wherein the first medical imaging technique is the CT scan.

4. The system of claim 2 , wherein the second medical imaging technique is the PET scan.

5. The system of claim 2 , wherein the third medical imaging technique is the MIP scan.

6. The program instructions for selectively rendering one or more of each of the correlated image pairs, the fused image, and the third tomographic medical image include: sequentially rendering each of the plurality of interrelated image pairs and one or more of the fused images when the input signal is received from the input / output device. The system of claim 2 further comprising program instructions for:

7. The program instructions for selectively rendering the visual representation of the pixel of interest include: using a machine learning algorithm to identify a pixel in one of the first STM images or one of the second STM images associated with the medical condition, the pixel including a pixel location; drawing a graphical annotation by said display at said pixel location; and further comprising program instructions for: the machine learning algorithms are trained with predetermined medical images, each depicting the presence or absence of the medical condition within its anatomical region; the presence of the medical condition is determined by pixel values ​​within the anatomical region being greater than or less than a predetermined value; The system of claim 6.

8. The program instructions for selectively rendering the visual representation of the region of interest include: Rendering, by said display, a second graphical annotation surrounding said pixel location. further comprising program instructions for: The system of claim 7.

9. 1. A computer-implemented method for synchronized navigation and annotation of a set of interrelated heterogeneous medical images, comprising: correlating a first set of serial tomographic medical images (STM images) of a subject with a second set of STM images of the subject based on first metadata and second metadata to generate a plurality of correlated image pairs, wherein the first set of STM images each include the first metadata and the second set of STM images each include the second metadata; overlaying each of the plurality of interrelated image pairs to generate a plurality of fused images, each corresponding to one of the plurality of interrelated image pairs; selectively and sequentially rendering, on a display, one or more of the plurality of interrelated image pairs, the fused image, and a third tomographic medical image of the subject when an input signal is received from an input / output device; selectively rendering, by the display, a visual representation of one or more of a pixel of interest and a region of interest for one or more of the plurality of interrelated image pairs, the fused image, and the third tomographic medical image when input is received from the input / output device; Including, the first set of STM images depicting a first anatomical plane and captured by a first medical imaging technique; a second set of STM images depicting the first anatomical plane and captured by a second medical imaging technique; the third tomographic medical image depicts a second anatomical plane and is captured by a third medical imaging technique; The method, wherein one or more of the pixel of interest and the region of interest is associated with a medical condition.

10. 10. The computer-implemented method of claim 9, wherein one of the first medical imaging technique, the second medical imaging technique, and the third medical imaging technique is one of an X-ray, a computed tomography scan (CT scan), a magnetic resonance imaging scan (MRI scan), a positron emission tomography scan (PET scan), and a maximum intensity projection scan (MIP scan).

11. 11. The computer-implemented method of claim 10, wherein the first medical imaging technique is the CT scan.

12. 11. The computer-implemented method of claim 10, wherein the second medical imaging technique is the PET scan.

13. The computer-implemented method of claim 10 , wherein the third medical imaging technique is the MIP scan.

14. Selectively rendering one or more of each of the plurality of interrelated image pairs, the fused image, and the third tomographic medical image, includes: sequentially rendering each of the plurality of interrelated image pairs and one or more of the fused images when the input signal is received from the input / output device.

11. The computer-implemented method of claim 10, further comprising:

15. The step of selectively rendering the visual representation of the pixel of interest comprises: using a machine learning algorithm to identify a pixel in one of the first STM images or one of the second STM images associated with the medical condition, the pixel including a pixel location; drawing a graphical annotation by said display at said pixel location; Further comprising: the machine learning algorithms are trained with predetermined medical images, each depicting the presence or absence of the medical condition within its anatomical region; the presence of the medical condition is determined by pixel values ​​within the anatomical region being greater than or less than a predetermined value; 15. The method of claim 14, wherein the method is computer-implemented.

16. The step of selectively rendering the visual representation of the region of interest comprises: Rendering, by said display, a second graphical annotation surrounding said pixel location. Further comprising:

16. The method of claim 15, wherein the method is computer-implemented.

17. 1. A computer program product comprising a computer-readable storage medium having program code embodied thereon, The program code correlating a first set of serial tomographic medical images (STM images) of the subject with a second set of STM images of the subject based on the first metadata and the second metadata to generate a plurality of correlated image pairs; overlaying each of the plurality of interrelated image pairs to generate a plurality of fused images, each corresponding to one of the plurality of interrelated image pairs; selectively and sequentially rendering, upon receiving an input signal from an input / output device, one or more of the plurality of interrelated image pairs, the fused image, and a third tomographic medical image of the subject, the third tomographic medical image depicting a second anatomical plane and captured by a third medical imaging technique; selectively rendering, by the display, a visual representation of one or more of a pixel of interest and a region of interest for one or more of the plurality of interrelated image pairs, the fused image, and the third tomographic medical image when input is received from the input / output device; a processor-executable method for performing the first set of STM images each including the first metadata, depicting a first anatomical plane, and captured by a first medical imaging technique; a second set of STM images each including the second metadata, depicting the first anatomical plane, and captured by a second medical imaging technique; one or more of the pixel of interest and the region of interest are associated with a medical condition; Computer program products.

18. 18. The computer program product of claim 17, wherein one of the first medical imaging technique, the second medical imaging technique, and the third medical imaging technique is one of an X-ray, a computed tomography scan (CT scan), a magnetic resonance imaging scan (MRI scan), a positron emission tomography scan (PET scan), and a maximum intensity projection scan (MIP scan).

19. The program code for selectively rendering the visual representation of the pixel of interest comprises: using a machine learning algorithm to identify a pixel in one of the first STM images or one of the second STM images associated with the medical condition, the pixel including a pixel location; drawing a graphical annotation by said display at said pixel location; further comprising program code executable by the processor to perform the machine learning algorithms are trained with predetermined medical images, each depicting the presence or absence of the medical condition within its anatomical region; the presence of the medical condition is determined by pixel values ​​within the anatomical region being greater than or less than a predetermined value; 20. The computer program product of claim 18.

20. The program code for selectively rendering the visual representation of the region of interest comprises: Rendering, by said display, a second graphical annotation surrounding said pixel location. further comprising program code executable by the processor to perform 20. The computer program product of claim 19.

Images