Method and apparatus for fusing multimodal images into fluorescence fluoroscopy images

Abstract

The present invention includes an apparatus for transforming a 3D image dataset such that the transformed image is registered to a 2D fluoroscopic image dataset. The process of the present invention is considered image registration in that it finds a transformation that aligns features of the 3D image with the 2D image. The fluoroscopic image registration system has a multi-stage framework that incrementally improves the image registration results. Each stage utilizes a fluoroscopic registration engine that iteratively searches the parameter space. The core registration engine generates a candidate transformation and uses it to generate a digitally reconstructed radiograph, which is then compared to the supplied fluoroscopic image, and the iterative registration engine either terminates the process or uses it to construct a new candidate transformation. The similarity measure is based on intensity and some custom data related to the registration scenario.

Description

Technical Field

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 120,476, filed on December 2, 2020, which is hereby incorporated by reference in its entirety.

[0002] The present invention relates to a novel image fusion of three-dimensional (3D) diagnostic images and navigation two-dimensional (2D) images, such as 2D X-ray images. More particularly, the present invention relates to an apparatus and method for modifying and fusing pre-acquired 3D images and combining those 3D images with (live / real-time) 2D images obtained during a procedure from a 2D image generation device, such as an X-ray fluoroscopy system.

Background Art

[0003] Fluoroscopy is an imaging technique that uses X-rays to obtain real-time dynamic images (videos) of the interior of a subject, such as the human body. In a basic application of fluoroscopy in medical imaging, a fluoroscopy device enables a physician to view the internal structures and functions of a patient, and for example, the pumping action of the heart or the swallowing motion can be observed. This technique is useful for both diagnosis and treatment and is applied in general radiology, radiation oncology, interventional radiology procedures, interventional cardiology procedures, surgery, particularly image-guided surgery, and other fields of medicine.

[0004] One of the simplest forms of this technique is a fluoroscopy device that includes an X-ray source and an image intensifier / detector between which a patient is positioned. Most fluoroscopy devices also include an X-ray image intensifier and a camera to improve the visibility of the image and enable the image to be viewed on a local display or a remote display screen.

[0005] For decades, fluoroscopy has tended to produce live videos that are not recorded, but since the 1960s, with technological improvements, recording and playback of specific parts of an examination have become standard.

[0006] Fluoroscopy has many limitations that prevent it from becoming more useful for clinicians. Firstly, because fluoroscopy uses ionizing radiation, each time the fluoroscopy machine is operated, there is a risk of radiation exposure to the patient and others in the treatment room. The consensus of experts and numerous studies have linked increased radiation exposure to an increased risk of cancer or direct or indirect tissue damage. These effects effectively limit the amount of radiation a patient can be exposed to, which in turn limits the number of fluoroscopy images that can be taken in a given period, and consequently, the quality of those images.

[0007] Secondly, unlike computed tomography (CT) and magnetic resonance imaging (MRI), X-ray fluorescence fluoroscopy has limited sensitivity to certain structures within the body. For example, bones can be seen with the naked eye in most cases, while soft tissue structures are often much more difficult to visualize. Some structures, such as blood vessels, can be easily identified by injecting a contrast agent that increases the color density, thereby enhancing the contrast of the fluorescence image. Iodine-based contrast agents are usually used for vascular interventions, but these iodine-based contrast agents can cause liver toxicity.

[0008] These limitations may cause physicians to limit the visualization of procedures, potentially leading to longer procedure times. This could expose sedated patients to additional risks related to awakening from sedation / anesthesia. For patients under general anesthesia, longer procedure times result in longer hospital stays and longer recovery periods. More generally, limitations in visualization can impair real-time navigation decision-making and lead to unsatisfactory procedure outcomes due to a lack of meaningful anatomical information available during the procedure.

[0009] Therefore, this field requires novel visualization methods that can provide meaningful image information in an accurate and timely manner. Some aspects of the above problem have already been addressed by several techniques. For example, digitally reconstructed radiographs (DRRs) simulate fluoroscopic images from three-dimensional computed tomography (CT) images. Tracking hardware can be used to determine the parameters necessary to construct a DRR that precisely overlaps with the fluoroscopy engine.

[0010] While the above methods and others exist as solutions, there is no reference-free ("markerless") tracking method that can find the conversion parameters necessary to create an enhanced DRR that overlays structures from 3D images scanned by CT, MRI, positron emission tomography (PET), etc., onto fluorescence fluoroscopy images in a common manner across the entire human body. [Overview of the project]

[0011] This invention relates to a novel image fusion of three-dimensional (3D) diagnostic images and navigation two-dimensional (2D) projection images, such as 2D X-ray images. More specifically, the invention relates to an apparatus and method for modifying and fusing pre-acquired 3D images and combining them with (live / real-time) 2D images obtained during treatment from a 2D image generation device such as an X-ray fluorescence fluoroscopy system. This allows for easy confirmation and dynamic tracking of structures such as the abdominal aorta, which may not have been visible in conventional 2D fluorescence fluoroscopy images, by overlaying them with aorta images obtained from CT, MRI, 3D, volumetric ultrasound, or other cross-sectional / 3D image scans.

[0012] In one embodiment, the present invention includes an apparatus for transforming a 3D image dataset so that the transformed image is aligned to a 2D fluorescence imaging dataset. In discovering a transformation that aligns the features of a 3D image with those of a 2D image, the process of the present invention is considered by the inventors to be "image registration".

[0013] In one embodiment, the present invention relates to a series of stages for discovering the optimal combination of conversion parameters that promote the matching of digitally reconstructed radiographic images (DRRs) and fluoroscopic images, which the inventors refer to as "fluoroscopic image registration."

[0014] More specifically, the fluorescence fluoroscopy registration system of the present invention has at its core a multi-stage framework for progressively improving the image registration results. In one embodiment, each stage utilizes the same core registration engine, or module, which iteratively searches the parameter space defined by that stage. In one embodiment, the core registration engine creates a candidate transformation, uses it to create a DRR, then compares it to a supplied fluorescence fluoroscopy image and terminates the process, or uses it to construct a new candidate transformation. In one embodiment, the measurement of candidate transformation similarity is based on some custom data related to brightness and registration scenarios.

[0015] Generally speaking, in one embodiment, the algorithm of the present invention is started in the first stage by a core registration engine. The core registration engine uses starting parameters (i.e., an image and some pre-transformed image) determined by the initial user position (e.g., anterior-posterior (AP) abdomen). The core registration engine uses the initial user position to construct DRR candidates. However, much of the functionality to achieve this result is provided by the event classifier of the iterative registration engine.

[0016] In one embodiment, the similarity between the DRR and the actual fluoroscopic image is recorded in memory by the core registration engine, and the image volume, image frame, and candidate transformations are transferred to the iterative registration engine, where the algorithm repeats processing using perturbed candidate transformations in a second stage controlled by the iterative registration engine ("digital radiograph reconstruction").

[0017] In one embodiment, the similarity results determined by the iterative registration engine (third stage, "similarity") are continuously recorded by the core registration engine's optimization engine. In the fourth stage, the optimization engine induces the creation of new candidate transformations from these results ("optimization engine"). These results are then used to generate a DRR image, and the generated DRR image is evaluated for its match with the latest fluorescence fluoroscopy navigation image.

[0018] In one embodiment, if a transformation is not acceptable based on predetermined parameters by the iterative alignment engine, the iterative alignment engine generates a new candidate transformation. The iterations controlled by the iterative alignment engine continue until a convergence criterion is reached, and the best candidate transformation is applied to the relevant construct in volumetric space, and the result is fused with the original fluorescence image frame.

[0019] In one embodiment, the image registration system includes a fluorescence fluoroscopy registration engine that iteratively improves a digitally reconstructed radiographic image (DRR) from a PACS image source using a fluorescence fluoroscopy image source, and matches the DRR image with the fluorescence fluoroscopy image, wherein the fluorescence fluoroscopy registration engine is located on either a client computer or a server, and comprises a core registration module that creates candidate transformations to obtain the DRR image and compares the DRR image with the fluorescence fluoroscopy image, a graphics memory module, and an iterative alignment module, wherein the graphics memory module receives volume data from the PACS image source. The system comprises a volumetric data storage for storing data and a video data storage for storing video data from the fluorescence fluoroscopy source. The core registration module controls the acquisition of the data from the PACS image source and the fluorescence fluoroscopy device source, and the transmission and storage of the data to the graphics memory module. The iterative alignment module evaluates the DRR image for matching with the fluorescence fluoroscopy image and repeatedly performs a process to generate a new DRR image using a new candidate transformation from the core registration module until a match is achieved with the new DRR image. The new candidate transformation is applied to structures in volumetric space, fused with the fluorescence fluoroscopy image to become a transformed image, and image registration is performed.

[0020] In one embodiment, the image registration system further comprises an input memory module that receives and stores the data from the PACS image source and the fluoroscopy device source prior to transmitting the data to the graphics memory module, the input memory module further comprises a volume data storage for storing volume data from the PACS image source and a video data storage for storing video data from the fluoroscopy device source.

[0021] In one embodiment, the image registration system further comprises a frame acquisition module connected to the fluorescence fluoroscopy apparatus source, the frame acquisition module acquires individual digital frames from the video data stream from the fluorescence fluoroscopy apparatus source, and transmits the video data to the video data storage of the input memory module.

[0022] In one embodiment, the core registration module further comprises an event classifier that autonomously determines the status of the image registration, the event classifier acquires a plurality of fluorescence fluoroscopy image frames acquired from the fluorescence fluoroscopy imaging device source by the frame acquisition module as input, the core registration module analyzes the characteristics of the plurality of fluorescence fluoroscopy image frames at that time, compares them with the DRR image, confirms the match, and dynamically performs the image registration.

[0023] In one embodiment, the image registration system further comprises a display unit of a computer system, wherein the core registration module comprises a graphical user interface (GUI) module, the graphical user interface (GUI) module assists in reading, displaying, and communicating with fluorescence fluoroscopy images received from the input memory module, enables communication between the user and the fluorescence fluoroscopy registration engine, and allows display of at least one of the fluorescence fluoroscopy image and the converted image on the screen of the display unit.

[0024] In one embodiment, the graphics memory module further includes an output frame buffer module that receives the converted image from the iterative alignment module and transmits the converted image as a rendered image to the display unit.

[0025] In one embodiment, the iterative alignment module further includes a candidate transformation module that uses the start parameters of the DRR image and one of the previously converted DRR images to construct the new candidate transformation, a digital reconstruction module that obtains the new candidate transformation and generates the new DRR image, and a similarity module that determines the similarity between the new DRR image and the fluoroscopic image and determines the match.

[0026] In one embodiment, the core registration module further includes an optimization module that induces the creation of the new candidate transformation of the candidate transformation module from the similarity result confirmed by the similarity module.

[0027] In one embodiment, the core registration module performs a three-parameter registration indicating a 2D rigid body transformation on the new DRR image.

[0028] In one embodiment, after the three-parameter registration, the core registration module performs a six-parameter registration to grasp the out-of-plane rotation of the patient's body and the variation in the distance from the fluoroscopic imaging device.

[0029] In one embodiment, in order to grasp the non-linear motion of the structure, non-rigid registration is performed on the new DRR image, and each parameter of the multi-parameter non-rigid registration enables local deformation within the new DRR image to be modeled.

[0030] In one embodiment, an image registration method is a method for performing image registration. Using a user interface and a graphics user interface (GUI) module that provides a fluoroscopic image registration system, on a display unit of a computer system, start image registration, and activate a fluoroscopic registration engine of the fluoroscopic image registration system, which is arranged in either a client computer or a server. The fluoroscopic registration engine performs the following steps: creating a candidate transformation by a core registration module of the fluoroscopic registration engine to obtain a digitally reconstructed radiograph (DRR) image; comparing the DRR image with the fluoroscopic image using an iterative alignment module of the fluoroscopic registration engine; evaluating the DRR image for conformity with the fluoroscopic image; repeatedly improving the DRR image using a new candidate transformation from the core registration module and generating a new DRR image until conformity with the new DRR image is achieved; applying the candidate transformation that has achieved conformity to a structure in a volumetric space; and fusing the new candidate transformation with the fluoroscopic image to form a transformed image and perform image registration.

[0031] In one embodiment, the core registration module controls the acquisition of the data from the PACS image source and the fluoroscopic image capture device source, and the transmission and storage of the data to the graphics memory module.

[0032] In one embodiment, the image registration method is a method for performing image registration, further comprising the steps of: capturing individual digital frames from a stream of video data from the fluorescence imaging device source using a frame acquisition module connected to the fluorescence imaging device source; and transmitting the video data to the video data storage of the input memory module of the fluorescence imaging registration engine.

[0033] In one embodiment, the image registration method further includes the step of autonomously determining the status of the image registration using an event classifier of the iterative alignment engine module, wherein the event classifier takes as input a plurality of fluorescence fluoroscopy image frames acquired by the frame acquirer from the fluorescence fluoroscopy image acquisition device source, and the core registration module analyzes the characteristics of the fluorescence fluoroscopy image frame at that time among the plurality of fluorescence fluoroscopy image frames, compares it with the DRR image for matching, and dynamically performs the image registration.

[0034] In one embodiment, the image registration method further includes the step of enabling the user to read, display, and communicate with the fluorescence image by providing a display unit that displays at least one of the fluorescence image received from the input memory module and the converted image received from the output frame buffer module, wherein the output frame buffer module receives the converted image from the iterative alignment module and transmits the converted image as a rendered image to the display unit.

[0035] In one embodiment, the image registration method further includes the steps of: configuring the new candidate transformation using the start parameter of the DRR image and one of the previously transformed DRR images, and using the candidate transformation module of the iterative registration module; generating the new DRR image using the digital reconstruction module of the iterative registration engine; and determining the similarity between the new DRR image and the fluorescence fluoroscopy image and determining the match using the similarity module of the iterative registration engine.

[0036] In one embodiment, the image registration method further includes using the optimization module of the core registration module to create the new candidate transformation of the candidate transformation module from the similarity results confirmed by the similarity module.

[0037] In one embodiment, the image registration method further includes the steps of performing a 3-parameter registration showing a 2D rigid body transformation on the new DRR image using the core registration module, and, after the 3-parameter registration, performing a 6-parameter registration using the core registration module to grasp the out-of-plane rotation of the patient's body and the variation in the distance from the fluoroscopy imaging device.

[0038] In one embodiment, the image registration method further includes the step of performing a non-rigid registration on the new DRR image in order to capture the non-linear motion of the components, wherein each parameter of the multi-parameter non-rigid registration can model local deformations within the new DRR image. [Effects of the Invention]

[0039] In one embodiment, the fluorescence fluoroscopy registration engine enables a unique method for labeling and indexing fluorescence fluoroscopy video data, and the core registration engine, relating to a typical workflow, acquires the aforementioned video images and other annotations created from the 3D image data (i.e., automatically annotated labels based on registration content and existing maps), and, if annotations exist, associates those annotations with a set of pixels in each video frame.

[0040] Another use case in one embodiment would be to create a semantic index of interventional procedures using the core registration engine of the fluorescence fluoroscopy registration engine, so that the most important / clinically relevant parts of therapeutic procedures such as stent insertion, needle biopsy, ablation, or contrast agent injection can be identified.

[0041] In one embodiment, the fluorescence fluoroscopy registration engine can stabilize the motion of fluorescence images through target tracking, fixing a specific object in the same position within the video stream even if it was moving in the unedited video stream. This method could also be applied to radiotherapy, where patient movement is tracked using the fluorescence fluoroscopy registration engine and fed back to the treatment delivery system as updated tracking information about the actual movement of the tumor. In one embodiment, the movement of the object is used to guide the radiation beam used for the patient treatment, and the radiation beam moves in conjunction with the tumor. Alternatively, the radiation beam is modulated or irradiated only when the tumor is located in a specific area. This maximizes treatment accuracy and minimizes unnecessary radiation to surrounding soft tissues and bone, regardless of patient / tumor movement.

[0042] In one embodiment, the fluoroscopy registration system uses the target's motion history to more accurately predict its location. In one embodiment, previous motion recorded by the fluoroscopy registration engine is modeled as a 3D path, and the modeled path is positioned relative to the motion tracked by the fluoroscopy image. In one embodiment, when a trend is detected by the fluoroscopy registration engine in adjusting the target path, predictive adjustments are further applied to the target tracking model, and the core registration engine enables dynamic correction of the 3D motion path from the 2D fluoroscopy image. According to the cyclic motion adjustment based on the fluoroscopy registration results of the present invention, the treatment margin is adjusted, i.e., reduced, and healthy tissue is preserved.

[0043] In one embodiment, by using fluorescence fluoroscopy registration system technology, the positioning of lung lesions in the respiratory cycle can be more precise, the radiation beam of the procedure can be adjusted, and a certain amount of radiation is irradiated only when it is known that the target is moving through one of the target paths. In one embodiment, by narrowing the margin, the amount of radiation reaching healthy cells is reduced, reducing complications and improving prognosis.

[0044] Therefore, several features consistent with the present invention have been outlined above to allow for a better understanding of the detailed description of the invention that follows and to a better understanding of the invention's contribution to the art. Of course, there are additional features consistent with the present invention, which are described below and form the subject matter of the claims appended to this document.

[0045] In this regard, before detailing at least one embodiment consistent with the present invention, it should be understood that the present invention is not limited by the configuration details and arrangement of components described or illustrated later. Methods and apparatus consistent with the present invention may take other embodiments and may be implemented and performed in various ways. It should also be understood that the expressions and wording used in this document, as in the abstract below, are for illustrative purposes only and should not be construed as restrictive.

[0046] Therefore, those skilled in the art will understand that the ideas of this disclosure can be readily utilized as a basis for designing other configurations, methods, and systems to achieve the objectives of the present invention. It is therefore important that such equivalents be considered included in the claims, insofar as they do not depart from the spirit and scope of the methods and apparatus consistent with the present invention. [Brief explanation of the drawing]

[0047] The description of the drawings includes examples of embodiments of the present disclosure, but is not intended to limit the scope of the present disclosure. [Figure 1] Figure 1 is a schematic diagram of a fluorescence fluoroscopy registration system including specialized computer resources according to one embodiment consistent with the present invention. [Figure 2] Figure 2 is a flowchart of a fluoroscopic registration for overlaying an aortic segment with a live fluoroscopic image, used in a typical workflow according to one embodiment consistent with the present invention. [Figure 3] Figure 3 is a schematic diagram showing each stage of the fusion process of a fluorescence fluoroscopy registration system according to one embodiment consistent with the present invention. [Figure 4] Figure 4A is a schematic diagram of the registration process of a fluorescence fluoroscopy registration system according to one embodiment consistent with the present invention. Figure 4B is a schematic diagram of the registration process of a fluorescence fluoroscopy registration system according to one embodiment consistent with the present invention. [Figure 5]Figure 5 is a schematic diagram of a typical workflow for overlaying an aortic segment with a live fluoroscopic image according to one embodiment consistent with the present invention. [Figure 6] Figure 6 is a schematic diagram of a typical workflow for rich annotation and indexing according to one embodiment consistent with the present invention. [Figure 7] Figure 7 shows a schematic diagram and processing example of motion stabilization according to one embodiment consistent with the present invention. [Modes for carrying out the invention]

[0048] The terms used herein are for the sole purpose of describing specific embodiments and are not intended to limit the invention. The phrase "and / or" as used herein includes all combinations of items described therein. The singular forms "a," "an," and "the" as used herein are intended to include the plural as well as the singular unless otherwise explicitly stated in the context. Furthermore, the terms "includes," "comprises," and / or "including" and "comprising," when used herein, specify the features, steps, operations, elements, and / or components described, but are understood not to exclude the existence and addition of other features, steps, operations, elements, components, and / or groups thereof.

[0049] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as those generally understood by those skilled in the art relating to this invention. Furthermore, terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with the relevant art and the context of this disclosure, and should not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0050] It is understood that many techniques and steps are disclosed in the description of this invention. Each of these has its own merits and can be used in combination with one or more, or possibly all, of the other disclosed techniques. Furthermore, each step may be performed in a different order than disclosed. Therefore, for the sake of clarity, this specification refrains from unnecessarily repeating all potential combinations of the individual steps. However, this specification should be read with the understanding that such combinations are entirely within the scope of the invention.

[0051] Medical applications are executed by systems connected to existing information systems. These existing information systems include hospital information systems (HIS), radiology information systems (RIS), radiography equipment, and / or other information systems, such as PACS and / or other systems. These systems may be designed to comply with relevant standards, such as Digital Imaging and Communications in Medicine (DICOM). Bidirectional communication between these systems enables information retrieval and / or provision between them, allows for the updating of information stored in the information systems, and enables the generation of desired reports and / or other information.

[0052] This section discusses fluorescence fluoroscopy fusion apparatuses, devices, and methods for the alignment and fusion of non-fluorescent fluoroscopic images. Numerous specific details are provided below for illustrative purposes to fully understand the invention. However, it will be apparent to those skilled in the art that the invention can be implemented without detailed examples, and that any examples not described in detail are not intentionally omitted but are considered to be known in the art.

[0053] The fluorescence fluoroscopy registration system of the present invention may include, for example, a client computer such as a personal computer (PC), and the client computer may or may not be linked to or integrated with the PACS, and may include, for example, an image display device capable of providing high-resolution digital images in 2D or 3D. According to one embodiment of the present invention, the client computer may be a mobile terminal if the image resolution is sufficiently high. The mobile terminal can be operated by the user remotely connecting to its program, and examples include a mobile computer terminal, a portable data organizer (PDA), a tablet, a smartphone, or other mobile terminal.

[0054] Input devices or other selection devices may be provided for selecting clickable icons, selection buttons, and / or other selectors. These can be displayed within the user interface using menus, dialog boxes, pull-down windows, or other user interfaces. The user interface may be displayed on the client computer, and the user may input commands into the user interface via a (multifunction) programmable stylus, keyboard, mouse, voice processing device, laser pointer, touchscreen, or other input device. Input devices or other selection devices may be implemented by dedicated hardware components, or their functions may be performed by code instructions executed on the client processor. For example, input devices or other selection devices may be implemented in a display device to show a selection window for inputting selections using a stylus or keyboard.

[0055] A client computer comprises a processor (internal or external), which includes a central processing unit (CPU) or graphics processing unit (GPU), parallel processors, input / output (I / O) interfaces, memory with programs having data structures, and / or other components. The components of the processor may be specialized to perform specific functions executed by programs. A client computer may also include input devices, image display devices, and one or more secondary storage devices.

[0056] Image display devices can display images such as X-ray images and / or other images clearly, simply, and accurately. Image display devices may be implemented using other touch-sensitive devices. Examples of other touch-sensitive devices include tablet personal computers, pocket personal computers, and plasma screens. High-resolution goggles may be used as a graphical display to provide end-users with the ability to view images.

[0057] A client computer may have applications that reside on the client computer and are written to run an existing computer operating system or a specialized system. Users may communicate with the applications via a graphical user interface. Client computer applications may be ported to other personal computer (PC) software, portable information terminals (PDAs), mobile phones, and / or any other digital device having a graphical user interface and a corresponding storage volume.

[0058] A processor can execute a program configured to perform a predetermined operation. According to one embodiment of the present invention, the processor is accessible to memory. The memory stores at least one sequence of code instructions. The code includes data structures and programs for performing a predetermined operation. The memory and programs may be located within or outside the client computer. While the system of the present invention may be described as performing a particular function, those skilled in the art will readily understand that this function is performed by a program rather than by the system itself.

[0059] A program may include a segmented program having code to perform a desired operation, a plurality of specialized modules that perform sub-operations of a certain operation, or be part of a single module of a larger program that provides that operation. A processor may be configured to access and / or execute multiple programs corresponding to multiple operations, such as assisting a user interface, providing communication functions, performing data mining functions, performing email operations, and / or other operations.

[0060] The data storage device may include databases such as centralized and / or distributed databases, and these databases may be relational databases connected via a network. The data storage device may be directly connected to servers and / or client computers, or indirectly connected via a LAN, WAN, and / or other network, or a communication network such as the Internet. The data storage device may be an internal storage device or an external storage device.

[0061] A client computer may be connected to other client computers or servers via a communication link. The communication link may include a wired communication link, a wireless communication link, or a switched circuit communication link. The communication link may also include a network of data processing devices, such as a LAN, WAN, the Internet, or a combination thereof. The communication link may connect email systems, fax systems, telephone systems, wireless communication systems such as pagers and mobile phones, wireless PDA systems, and other communication systems. The communication link may be implemented using dedicated hardware components or using a general-purpose CPU that executes instructions from a program. According to one embodiment of the present invention, the communication link may be at least partially included in a processor that executes instructions from a program.

[0062] The server is provided in a centralized environment. The server may be similar in structure and operation to a client computer. Alternatively, a distributed client computer may be provided with multiple individual processors, which may be located on one or more machines. The server may consist of a single unit or a distributed system having multiple servers or data processing units. These servers may be shared by multiple users who are directly or indirectly connected to each other. These servers may be connected to a communication link, which is preferably configured to communicate with multiple client computers.

[0063] The present invention may be executed using software applications residing in a client computer and / or server environment, or using software applications residing in a distributed system across numerous client computer systems via a computerized network. Therefore, individual calculations may be performed on either a client computer or a server, or both. Calculations consistent with the present invention may be performed on either a client computer or a server, or both. If a server is used, that server may be accessible by client computers via the internet.

[0064] A user interface, including graphical user interfaces (GUIs), may be provided to support multiple interfaces, including a display screen, a speech recognition system, a speaker, a microphone, input buttons, and / or other interfaces.

[0065] Although the above physical architecture is described as either a client computer component or a server component, those skilled in the art will understand that the components of the physical architecture may be located on either a client computer or a server, or in a distributed environment. Furthermore, the above features and arithmetic operations may be implemented by specialized hardware or as a program having code instructions executed on a data processing unit, and some parts of the above sequence of operations may be executed in hardware, while other parts of the arithmetic operations may be executed using software.

[0066] This underlying technology can be replicated to various other sites. Each new site may maintain communication with its surroundings so that, in the event of a catastrophic failure, one or more servers can continue running the application, and the system can geographically distribute the application load as needed.

[0067] Furthermore, although it is stated that an embodiment of one embodiment of the present invention is recorded in memory, those skilled in the art will understand that all or part of the present invention may be stored on or read from other computer-readable media, such as secondary storage devices. These secondary storage devices may be any known or future-developed ones, such as CD-ROMs, hard drives, flash drives, or other forms of ROM or RAM. Furthermore, although certain components of a system are described, those skilled in the art will understand that a system suitable for use with the method and system of the present invention may include additional or different components.

[0068] This disclosure is intended to be illustrative of the present invention and is not intended to limit the invention to any specific embodiments described in the following figures or description.

[0069] Herein, preferred embodiments and specific examples are referenced and the invention is described and explained, but it will be apparent to those skilled in the art that other embodiments and examples may perform similar functions and / or produce similar results. All such equivalent embodiments and examples are included in the spirit and scope of the invention, are considered therein and are intended to be included in this disclosure.

[0070] In one embodiment, the present invention relates to a novel image fusion of three-dimensional (3D) diagnostic images and two-dimensional (2D) navigation images, such as 2D X-ray images. More specifically, in one embodiment, the present invention relates to an apparatus and method for modifying and fusing pre-acquired 3D images and combining them with (live / real-time) 2D images obtained during treatment from a 2D image generation device such as an X-ray fluorescence fluoroscopy system. This allows for easy confirmation and dynamic tracking of structures such as the abdominal aorta, which could not be seen in conventional 2D fluorescence fluoroscopy images, by overlaying them with the aorta obtained from CT, MRI, 3D, volumetric ultrasound, or other cross-sectional / 3D image scans.

[0071] In one embodiment, the present invention includes an apparatus for transforming a 3D image dataset so that the transformed image is aligned to a 2D fluorescence fluoroscopy image dataset during a fusion session. In discovering a transformation that aligns the features of a 3D image with those of a 2D image, the process of the present invention is considered by the inventors to be "image registration".

[0072] More specifically, in one embodiment, in a single fusion session, there is one floating (or moving) image, which is transformed and fused with all subsequent images loaded during the session under fluorescence fluoroscopy registration processing. These subsequent images are referred to as reference images (or fixed images), and each time a new reference image is loaded, the floating image for that session is transformed and aligned with the reference image.

[0073] In one embodiment, the present invention relates to a series of stages for discovering the optimal combination of conversion parameters that promote the matching of digitally reconstructed radiographic images (DRRs) and fluoroscopic images, which the inventors refer to as "fluoroscopic image registration."

[0074] More specifically, the fluorescence fluoroscopy registration system of the present invention has at its core a multi-stage framework for progressively improving the registration results. In one embodiment, each stage utilizes the same core registration engine, or module, which iteratively searches the parameter space defined by that stage. In one embodiment, the core registration engine creates a candidate transformation, uses it to create a digitally reconstructed radiographic image (DRR), then compares it to the supplied fluorescence fluoroscopy image, and either terminates the process or uses it to construct a new candidate transformation. In one embodiment, the measurement of candidate transformation similarity is based on some custom data related to brightness and registration scenarios.

[0075] In one embodiment, the present invention uses a computer equipped with a specialized processor, which improves the fundamental operation of the computer and leads to further advancements in the medical technology field, as described later.

[0076] In one embodiment of the present invention, Figures 1 and 2 comprehensively illustrate the specialized computer resources, the flowchart of the algorithm, and the individual steps of the flowchart when data from a video source 102 and a network source 106 is input into the memory 104 of the fluorescence fluoroscopy registration system 100 of the present invention, processed by an optimization engine, i.e., module 117, and a graphics engine, i.e., module 113, and finally rendered to the display unit 120.

[0077] More specifically, as shown in Figure 1, in one embodiment, the fluorescence fluoroscopy registration system 100 of the present invention comprises a fluorescence fluoroscopy registration engine 101, i.e., a processing board 101, and an independent electronic equipment frame acquisition module 103. In one embodiment, the fluorescence fluoroscopy registration engine (FRE) 101 iteratively registers fluorescence fluoroscopy images and DRRs based on brightness with 3 to N degrees of freedom (DOF), the DOF of the FRE ranging from 2D linear in-plane transformation (3DOF) to 3D linear transformation (6DOF) and transformations of any number of dimensions required for modeling non-rigid deformation (NDOF). In one embodiment, the fluorescence fluoroscopy registration engine 101 may be located on a client computer or server and may be a distributed system.

[0078] In one embodiment, as shown in Figure 1, the fluorescence fluoroscopy system 102 includes an X-ray source and a fluorescence screen in which the patient is positioned during medical image acquisition, and acquires real-time moving images of the inside of a patient using X-rays within a hospital or medical facility. The fluorescence fluoroscopy system 101 can use a computer that provides data acquisition, data storage, and image analysis software to make the images available on a display screen, and similarly, other equipment such as X-ray image enhancers and cameras can be used to improve image visibility.

[0079] In one embodiment, the frame acquisition module 103 of the fluorescence fluoroscopy registration system 100 is mounted together with the fluorescence fluoroscopy imaging apparatus 102 or mounted separately and connected to the fluorescence fluoroscopy imaging apparatus 102, for example, by an HDMI cable. In one embodiment, the frame acquisition module 103 acquires individual digital still frames from the digital video stream from the fluorescence fluoroscopy imaging apparatus 102 (see step 200 in Figure 2). In one embodiment, the frame acquisition module 103 transmits the information of each individual digital still frame to the input memory module 104 of the fluorescence fluoroscopy registration engine 101 (step 201 in Figure 2) via a network, i.e., cable, and the information is stored as video data storage 105.

[0080] In one embodiment, the PACS 106 communicates images bidirectionally according to the DICOM standard, controls the storage and transmission of medical images in a hospital or medical facility, and is integrated with medical imaging equipment such as scanners, servers, workstations, printers, network hardware, and networks, and receives 2D volumetric data from a fluoroscopy imaging device 102. In one embodiment, the fluoroscopy registration engine 101 of the present invention collects 3D volumetric data transmitted from the PACS 106 via a network or cable (step 202 in Figure 2) and stores the volumetric image data in an input memory module 104 and a volume data storage 107 (step 203 in Figure 2).

[0081] In one embodiment, the specialized core registration engine, or module 108, of the fluorescence fluoroscopy registration engine 101 of the present invention comprises at least one specialized processor 108 (see Figure 1) having a memory 123, and this specialized processor 108 controls the acquisition of data by the frame acquisition module 103 and PACS 106, and the transmission and storage of that data to the input memory module 104. The core registration engine 108 processes the received data from the video data storage 105 and volume data storage 107, and stores that data in the video data storage 109 and volume data storage 110 of the specialized graphics processor memory module 111 of the fluorescence fluoroscopy registration engine 101, respectively.

[0082] In one embodiment, the fluorescence fluoroscopy registration engine 108 includes at least one specialized graphics processor 113 as an iterative registration engine 109 (see Figure 1). In one embodiment, the iterative registration engine 113 includes an event classifier 112, a candidate transformation module 114, a digital radiographic image reconstruction module 115, and a similarity module 116.

[0083] In one embodiment, an event classifier 112 (see Figure 1) is used to autonomously determine which registration scenario the fluorescence fluoroscopy registration engine 101 should be in. In one embodiment, this classification is based solely on fluorescence fluoroscopy video data 105, but uses one or more frames to make the decision. In one embodiment, the event classifier 112 takes frames as input, either captured from the fluorescence fluoroscopy apparatus 102 by the frame capturer 103 or directly captured. Each latest frame is transferred by the core registration engine 108 and stored in the video data storage 105 of the input memory module 104, and transmitted to the video data storage 110 of the specialized graphics processor memory module 111, which also stores previous frames. In one embodiment, the core registration engine 113 analyzes the features of the image frame at that time (and the last frame) and performs dynamic image fusion to provide the user with seamless, non-interactive fusion.

[0084] In one embodiment, the fluorescence fluoroscopy registration engine 101 is divided into a series of stages 300, as shown in Figure 3. Generally, in one embodiment, the algorithm of the present invention (Figures 1 to 4B) is started in the first stage by the core registration engine 108 (step 204, Figure 2). The core registration engine 108 uses starting parameters (i.e., images and previously transformed images) defined by the initial user position (e.g., anterior-posterior (AP) abdomen) to construct DRR candidates ("candidate transformation parameters" 301). Much of the functionality to achieve this result is provided by the event classifier 112 of the iterative alignment engine 113 (see Figure 1).

[0085] In one embodiment, the similarity between the DRR and the actual fluoroscopic image is recorded in memory 125 by the core registration engine 108, and the image volume, image frame, and candidate transformations are transferred to the iterative registration engine 113 (step 205, Figure 2). Then, in the second stage, the algorithm is controlled by the iterative registration engine 113 and iteratively performs processing with perturbed candidate transformations ("digital radiographic image reconstruction" 302).

[0086] In one embodiment, the similarity results (third stage, “similarity” 303) are determined by the iterative registration engine 113 and continue to be recorded by the optimization engine 117 of the core registration engine 108. In the fourth stage, the optimization engine 117 induces the creation of new candidate transformations from those results (“optimization engine” 304), and those results are subsequently used to generate a DRR image. The DRR image is evaluated for its match with the latest fluorescence fluoroscopy navigation image (step 206, Figure 2).

[0087] In one embodiment, if a transformation is unacceptable based on predetermined parameters by the iterative alignment engine 113 (step 208, Figure 2), the iterative alignment engine 113 generates a new candidate transformation (step 209, Figure 2). The iterations controlled by the iterative alignment engine 113 continue until a convergence criterion is reached, and the best candidate transformation is applied to the relevant constructs in volumetric space, and the result is fused with the original fluorescence image frame (step 210, Figure 2).

[0088] In particular, regarding the creation of candidate transformation parameters in the first stage 301 / 400 (see Figure 4A), the 3-parameter investigation, rigid refinement using 6-parameters, and non-rigid refinement resulting in the final image transformation are shown in Figures 4A and 4B. In one embodiment, the parameter investigation is repeated based on brightness for the 3-parameter, 6-parameter, and non-rigid refinement.

[0089] In one embodiment, with respect to the first stage, the core registration engine 108 creates a DRR of the imaged body within the approximate angle of fluorescence fluoroscopy and the field of view of the magnified display. In one embodiment, the core registration engine 108 constitutes a novel graphics processor based on a simplex method based on a hill-climbing approach. In the graphics processor, a series of candidate solutions constitute a polyhedron in N-dimensional space. The simplex method works well when there is noise found in actual medical images.

[0090] In one embodiment, the core registration engine 108 connects each DRR to create a whole-body DRR. This whole-body DRR is found by 3-parameter registration by the core registration engine 108 (see Figure 4A, "Basic position setting, 3-parameter investigation" 401). The above 3 parameters represent 2D rigid body transformations (in-plane rotation and in-plane [xy] translation). In one embodiment, once a matching position is found by the core registration engine 108, the DRR parameters are updated by the core registration engine 108 at that position.

[0091] This step of the present invention creates a rough overlap between the CT image and the DRR, and does not account for out-of-plane rotation of the body or variations in distance from the camera. In one embodiment of the present invention, to address these positional misalignments, the core registration engine 108 performs 6-parameter registration based on the starting position from the previous step (see "Rigid Body Refinement, 6 Parameters" 402).

[0092] The results of rigid body registration do not capture the nonlinear behavior of structures in an image, such as soft tissues stretching and deforming from different locations or different respiratory stages. Therefore, in one embodiment of the present invention, the core registration engine 108 creates a multi-parameter registration in place of a 6-parameter registration, where each parameter models local deformations, and the resulting non-rigid body registration addresses these deformations (see "Non-rigid body refinement, non-rigid body registration" 403). In one embodiment, these multi-parameters are adjusted to define the entire deformation region. In that transformation region, each voxel in the volume being transformed has an eigenvector that defines where it belongs in the new image. In one embodiment, various 3D transformation models exist (e.g., a uniform grid connected by b-splines, a non-uniform grid connected by b-splines, a hierarchical subvolume connected by quaternion interpolation, etc.). In this phase 403, the same basic configuration as the previous registration phases 401 and 402 is maintained, and candidate transformations are created and iteratively evaluated by the iterative alignment engine 404.

[0093] In one embodiment, as shown in Figure 3, the fluorescence fluoroscopy registration engine 101 evaluates the DRR image for each match using similarity and determines whether the transformation is clinically acceptable based on a pre-trained threshold (step 208, Figure 2). In one embodiment, the final result of stages 301-304 is a transformation field for deforming features within the volume and enhancing and fusing the alignment between the DRR and the fluorescence fluoroscopy image (step 210, Figure 2).

[0094] In one embodiment, in the first stage, once the volume and initial values ​​are determined by the core registration engine 108, the parameters are transferred by the core registration engine 108 to the iterative registration engine 113 / 404 (see Figure 4A). In one embodiment, the iterative registration engine 404 performs 3-parameter registration and 6-parameter registration, as well as non-rigid registration. Non-rigid registration adds a hierarchy to the parameter investigation to reduce complexity, keeping each volume with non-rigid registration problems in a lower dimension. In one embodiment, the iterative registration engine 404 uses a common iterative registration method common to each phase 401, 402, and 403. In one embodiment, in each iteration of the iterative registration engine 404, the time value of each point determines a candidate solution to be next evaluated in space, based on extensive heuristic calculations.

[0095] More specifically, in one embodiment, the event classifier 112 filters out empty frames in the video data storage 110 of the graphics processor memory module 111 by entropy calculations performed in parallel on the event classifier 112's multithreaded processor. In one embodiment, the iterative alignment engine 113 sends the remaining real frames to a macroblock (MB) calculation, which matches and infers the movement of subblocks across the remaining real frames. In one embodiment, the MB calculation is performed between the event classifier 112, which spans the event classifier 112's multithreaded processors, and the local memory modules 122 of each multiprocessor 108, 113 that hold the image regions.

[0096] In one embodiment, the event classifier 112 uses previous frames residing in the same local memory 12 to influence the calculation of MB motion vectors, and the calculation results are sent back and integrated into the main global memory 123 by the event classifier 112. In one embodiment, the event classifier 112 sends the data results from memory 123 to a pre-trained classifier running on the event classifier 112's multithreaded hardware to determine whether the motion vectors are sufficiently diverse and whether partial or complete re-registration of the data can be initiated (step 208, Figure 2).

[0097] In one embodiment, the determination by the iterative alignment engine 113 is classified into three categories: (1) minimal change in motion vectors, no re-registration required; (2) significant change in motion vectors, partial re-registration required; and (3) little to no common MBs, requiring full re-registration.

[0098] In one embodiment, in the case of the first classification (1), the iterative alignment engine 113 determines that no parameters need to be recalculated. In one embodiment, in the case of the second classification (2), the iterative alignment engine 113 determines that only half of the parameters, i.e., only the in-plane parameters, need to be recalculated. In one embodiment, by avoiding overall registration for each frame, the classifier of the iterative alignment engine 113 enables a higher frame rate by switching between (1) and (2), which is significantly faster than (3).

[0099] In one embodiment, the iterative function of the iterative alignment engine 404 generally aims to maximize the number of new similar feature points by the similarity module 116, and replaces points within the polyhedron when a better point is found.

[0100] In one embodiment, the similarity module 116 uses mutual information (MI) to analyze information common to the brightness values ​​between two sets of pixels. This does not presuppose any priori pixel matching, which would be an ideal candidate for matching the actual image and the composite image in the present invention. In one embodiment, MI was selected because it has been shown to be a robust criterion in various registration scenarios where the two images are considered completely different. In one embodiment, MI is used to calculate the similarity between the DRR and the fluoroscopic image (see step 207, Figure 2). Furthermore, in one embodiment, MI ignores structures that are not common to the two images, which means it is robust when instruments or contrast enhancers are used.

[0101] In one embodiment, the image registration algorithm utilizes the parallel computing capabilities of the graphics processor 113. In particular, in one embodiment, the calculation of the standardized mutual information (NMI) between the original fixed image (i.e., the fluorescence fluoroscopy image) and the transformed "floating" image (i.e., the DRR image) is a computationally intensive step in the algorithm, facilitated by the graphics processor 113 kernel.

[0102] In one embodiment, a voxel within a certain subvolume shares memory resources within a certain multiprocessor 113 and optimizes computational resources for an algorithm, while another subvolume does not do so and is therefore positioned as a subvolume for a different multiprocessor.

[0103] In one embodiment, once the iterative alignment engine 113 finds an optimized combination of 6DOF rigid body transformations for all subvolumes, the resampling graphics processor 113 kernel acquires those final transformations, interpolates them to derive a smooth transformation region, applies that region to a floating image, and creates a final registration image (step 210, Figure 2). In one embodiment, the 6DOF rigid body transformation for each voxel is estimated by the iterative alignment engine 113, which interpolates the transformation of the center of the subvolume surrounding that voxel. In one embodiment, the independent components of the transformation are interpolated separately by the iterative alignment engine 113, with three translations along the coordinate axes determined by tricubic interpolation, while the 3D rotational orientation is determined by spherical cubic quaternion interpolation. In one embodiment, the transformation, interpolation, and final resampling application are independent at the voxel level, and each voxel can be positioned in a single thread. In one embodiment, both of these kernels utilize thread groups and threads for algorithm execution to ensure that the iterative alignment engine 113 is used extensively for these time-consuming kernels.

[0104] In one embodiment, the converted image is displayed on the display unit 120 of the client computer system 121. In one embodiment, the iterative alignment engine 113 includes a dedicated frame buffer 119, since the screen of the display unit 120 must have each defined pixel with each reload. In one embodiment, the frame buffer 119 is an area of ​​memory 111 used to hold frames of data continuously transmitted to the display unit 120.

[0105] In one embodiment, the frame buffer 119 is the size of the largest displayable image and may be an independent memory bank on the graphics card (display adapter) or a portion allocated in standard memory 111. In one embodiment, when fusion is complete, the converted image is transferred to this frame buffer 119, and the display unit 120 displays the rendered enhanced image from the output frame buffer 119 on the screen.

[0106] In one embodiment, the core registration engine 108 includes a graphical user interface (GUI) module 118 (see Figure 1), which enables the user to communicate with the fluorescence fluoroscopy registration system 100 via the display unit 120 of the client computer system 121. In one embodiment, the GUI 116 displays on the screen 120 of the display unit an image received from the input memory 104 and an image created by fusion processing via the fluorescence fluoroscopy registration system 100 and transmitted from the specialized graphics processor memory module 111 and the output frame buffer module 114. In one embodiment, a fusion session is defined by a single execution of the GUI 116, that is, from the start of the GUI 116 to the end of the GUI 116.

[0107] In one embodiment, the GUI module 118 assists in reading, displaying, and communicating with medical images received from the input memory 104 and images created by fusion processing via the fluorescence fluoroscopy registration system 100, and collects information from the user to set the fusion processing, such as selecting appropriate images and instructing the timing of fusion start. In one embodiment, the information is collected in a configuration text file 124 and transferred from the input memory 104 to the same main memory 111 along with the image input. In one embodiment, the GUI module 118 renders elements to be output via the frame buffer 119 using a specialized graphics processor 113.

[0108] With regard to the calculations of the present invention, as described above, the present invention assists users (i.e., medical professionals) through the visual evaluation, comparison, and fusion of information between anatomical and functional images from patients. The present invention provides additional information to the user's existing workflow for patient evaluation and a method for comparing medical image data from multiple DICOM-enabled image modality sources. The present invention enables the display, fluoroscopic registration, and fusion of medical images in diagnostic radiology, oncology, radiotherapy planning, interventional radiology procedures, interventional cardiology procedures, and other medical fields.

[0109] In one embodiment, a user launches a graphics user interface (GUI) 118 from a client computer system 121, and the graphics user interface (GUI) 118 creates a new GUI window on the screen of the display unit 120, in which no image has been loaded. The fluorescence fluoroscopy registration engine 101 launches network configuration, tests the underlying computer platform, and initiates internal processing to perform the fluorescence fluoroscopy image registration of the present invention.

[0110] In one embodiment, the core registration engine 108 displays at least an overview of the status of other user-facing communication buttons on the display unit 120, and indicates on the display unit 120, using text or color codes, that the user can start a new session when ready.

[0111] In one embodiment, a session creates two processes: GUI module 118 interface processing and fluorescence fluoroscopy registration engine 101 processing. In one embodiment, the user communicates with GUI 118, and simultaneously, GUI 118 monitors newly arriving fluorescence fluoroscopy images determined by the processing of the fluorescence fluoroscopy registration system 100 described above, as shown in Figures 1 and 3. The present invention is designed to minimize the amount of communication required to confirm the fusion results from the fluorescence fluoroscopy registration process.

[0112] In one embodiment, prior to the start of a session, two image sources (fluorescence fluoroscopy imaging device 102 and PACS 106) are pre-configured by the core registration engine 108, and the fluorescence fluoroscopy images are sent to the fluorescence fluoroscopy registration engine 101 processing board of the client computer or server. In one embodiment, the core registration engine 108 pushes the image (image 1 - floating image) from the PACS 106 image source, and the GUI module 118 receives the image via the frame buffer 119 and automatically displays it on the display unit 120.

[0113] In one embodiment, for a new image, the image frame is received by the client computer system 120 and rendered by the frame buffer 119 on the screen 120 of the main display pane / display unit of the GUI module 118. At this point, in one embodiment, the user can perform actions required for the image to be used by the fluorescence registration engine 101 for the remainder of the session, such as changing the window level (increasing, decreasing, shrinking, or enlarging) or accessing the color map (action indicator).

[0114] In one embodiment, the core registration engine 108 pushes another image (image 2 - reference image) from another image source, such as a fluorescence fluoroscopy imaging device 102. Image 2 is received via the frame buffer 119 using the GUI module 118 and displayed on the screen 120 of the display unit. In one embodiment, if there are two images, the GUI module 118 creates a registration operation for the two images, and the fluorescence fluoroscopy registration process is started by the fluorescence fluoroscopy registration system 101 of the present invention. As described above, in this process, the core registration engine 108 treats image 1 as a floating image and image 2 as a reference image.

[0115] In one embodiment, Image 1 is transformed by the core registration engine 108 and the iterative alignment engine 113 (for the purpose of correcting any changes found between Image 1 and Image 2), and the iterative alignment engine 113 fuses Image 1 with Image 2 by the process described above in relation to the fluorescence fluoroscopy registration system 100. In one embodiment, after the fluorescence fluoroscopy registration engine 101 has completed and the fusion has been performed, the transformed DRR image is sent back to the GUI 118, which sequentially renders the latest images via the frame buffer 119. Thus, in one embodiment, the user may communicate with the images again via the GUI 118 to read the images and perform operations such as zooming, changing the window level, fusion (changing the transparency of the superimposed images), and panning.

[0116] In one embodiment, if fine adjustments are needed to the image registration, the user can move the registration image by using the arrow keys on the display unit 120 or the keyboard on the computer system 121 to move the overlapping images in place by a few pixels at a time.

[0117] In one embodiment, errors in image registration are reported to the administrator. Furthermore, in one embodiment, if image registration needs to be redone due to misalignment or a significant change in position, registration can be restarted, and the latest frame becomes the new final image for registration.

[0118] In this embodiment, after some time has elapsed, another image is captured from the fluorescence fluoroscopy apparatus image source 102 (image 3), received via the frame buffer 119, rendered, and displayed by the display unit 120 of the client computer 121. In one embodiment, image 3 replaces image 2, which was discarded by the core registration engine 108 for the session. In one embodiment, the core registration engine 108 and the iterative alignment engine 113 transform image 1 in relation to image 3, but since states such as window level, fusion level, and zoom are initially based on the display of the fused image of image 1 and image 2, the user may, at this point, communicate with the image on the display unit 120 again to change the display of the fused image as needed.

[0119] In one embodiment, if additional images are captured in this session, the core registration engine 108 continues to replace the current reference image in this pattern, while the first image pushed as a floating image in the session is retained. In this embodiment, if no further images are captured, the user closes the GUI 118 and returns to the start screen on the display unit 120. The fusion session ends at this point, and in one embodiment, the images used in the previous session become directly usable (or unusable) in subsequent sessions.

[0120] In one embodiment, the workflow and GUI 118 as a whole are facilitated by a core registration engine 108, which forms the basis of the fluorescence fluoroscopy registration engine 101, and an iterative alignment engine 113, which performs volumetric image conversion for fluorescence fluoroscopy images. In one embodiment, the fluorescence fluoroscopy registration engine 101 can directly convert one modality image to a different modality image. On the other hand, in one embodiment, if a previous frame has been received and fused, the fluorescence fluoroscopy registration engine 101 of the present invention uses the previous registration results to induce the current registration by attempting heuristic methods.

[0121] In one embodiment, a detailed log of all actions performed by the user is recorded in the memory 125 of the core registration engine 108.

[0122] In one embodiment, the workflow of this embodiment can be modified in various ways to accommodate different needs. For example, in one embodiment, images may be uploaded directly from a USB or CD by reading a DICOM PACS 106 via GUI 118. In another embodiment, a single image source may generate all images used in one session, while in another embodiment, there may be multiple image sources. Also, in another embodiment, fusion may not be performed automatically but can be initiated by a user using a button on GUI 118.

[0123] In one embodiment, a typical workflow for registering a 3D image volume to a fluorescence fluoroscopy image is described below when there is no solution from a previous registration related to the current registration. Various other scenarios may occur during fluorescence fluoroscopy, and prior registrations from that session may have conversion results that the present invention can use due to failures in the current registration.

[0124] In one embodiment, before the stage requiring fluorescence fluoroscopy begins, all existing 3D images are registered together in the space of the 3DCT images to be registered together with the fluorescence images using the non-rigid registration technique in the core registration engine 108 (described above). In one embodiment, the 3DCT images are used for aligning the reference line with the fluorescence images.

[0125] More specifically, in one embodiment, a typical workflow of the autonomous fusion process of the fluorescence fluoroscopy registration engine 101 includes the use of the push interface of the DICOM PACS 106 (see Figure 5) in a situation where an aortic segment is superimposed with a live fluorescence fluoroscopy image.

[0126] In one embodiment of a typical workflow 500, first, the contrast-enhanced CT image 501 is segmented (using appropriate medical equipment) for the aorta 502, which is the target vessel of the procedure. In one embodiment, the aortic segment 502 is stored as a label map in the volume data 110 of the graphics memory 111 by the core registration engine 108, and a new image volume is created in the same volumetric space as the original CT image 501. In one embodiment, both the aortic segment 502 and the original CT image 501 are pushed to the image selection process by the core registration engine 108. The core registration engine 108 selects images from the input memory 104 as the corresponding volume (reference image) and the visualization volume (floating image), respectively.

[0127] In one embodiment, the user selects the abdomen as the internal location and the initial direction as the AP anterior-posterior direction. At this point, in one embodiment, the iterative alignment engine 113 of the fluorescence fluoroscopy registration engine 503 finds corresponding locations between the fluorescence fluoroscopy image 504 and the CT image 501 (see image 505), and using these parameters, the image 502 of the aortic segment is superimposed on the fluorescence fluoroscopy image 504 (see superimposed image 506), clarifying the position within the frame and thus fusion is promoted.

[0128] In one embodiment, once the arrangement of the 3DCT image 501 and the fluorescence fluoroscopy image 504 is determined, other image data aligned with the 3DCT image 501 may be used for refinement or for advanced visualization using the same conversion method as described above.

[0129] In one embodiment, once the part related to the visualization of the process is completed, the fluorescence transmission image frame on the display unit 120 becomes empty, causing the core registration engine 108 to stop the fusion process.

[0130] In one embodiment, some stages of the fluorescence fluoroscopy registration process may be modified by the fluorescence fluoroscopy registration engine 101 under appropriate conditions. For example, once the first visualization image appears, the fusion process of the fluorescence fluoroscopy registration system 100 continues to update subsequent fluorescence fluoroscopy images. However, the conventional general operation of a fluorescence fluoroscopy camera is patient-aligned movement / adjustment to focus on a different area of ​​the patient's body. In one embodiment, the image tool applied by the user moves within the image, but the patient's body remains stationary, and therefore the fused image is in the same position.

[0131] However, in one embodiment, at some point, user operation causes a discontinuous pan of the camera frame to different parts of the body. In one embodiment, the core registration engine 108 detects this and restarts the fluorescence fluoroscopy image registration process, and after a few seconds, the iterative alignment engine 113 creates a new fused image. Thus, in one embodiment, the results of the previous image registration process then only require adjustment by the same transformation of the fluorescence fluoroscopy registration engine 101.

[0132] In one embodiment, when the fluorescence fluoroscopic images overlap (i.e., when the camera is smoothly panned across the subject), the fluorescence fluoroscopic registration engine 101 can make this transformation apparent in the registration of the two registration images. In another embodiment, the above transformation can similarly be applied to an image during the fusion process, and by moving the image, rapid and accurate image updates can be performed without requiring all registration stages (i.e., similarity, optimization, etc.).

[0133] Regarding the patient's position, in other embodiments, angiographic images or digital subtraction images, typically acquired during contrast agent administration, generate 2D projections of the patient's blood vessels and other structures. In one embodiment, these structures are highlighted, providing new information about changes in the shape and position of internally deformable structures, although the subject's position is not altered. Similarly, information about structural deformation can also be obtained from user devices visible in the fluoroscopic image corresponding to the structure, such as catheters introduced into blood vessels. However, in one embodiment, the fluoroscopic registration engine 101 of the present invention updates the corresponding superimposed image by performing the final stage of non-rigid registration and updating only the nonlinear transformation, thereby creating a more accurate superimposed image.

[0134] In one embodiment, the event classifier 112 is used in conjunction with the core registration engine 108 as described above to invoke each of the above scenarios without requiring additional communication from the clinician. In one embodiment, as described above, the core registration engine 108 takes frames directly captured from the fluoroscopy apparatus as input, analyzes the features of the image frame at that time (and the last frame), and determines which scenario to invoke. In one embodiment, the result is a system that dynamically invokes image fusion, providing the user with uninterrupted, communication-free fusion.

[0135] In one embodiment, as shown in Figure 6, the fluorescence fluoroscopy registration engine 101 enables a unique method for labeling and indexing fluorescence fluoroscopy video data. In one embodiment, the core registration engine 108, in relation to a typical workflow, retrieves the aforementioned captured video and other annotations created from 3D image data (i.e., automatically annotated labels based on registration content or existing maps), and associates them, if present, with a set of pixels in each video frame. In one embodiment, the association between 3D labels and 2D video data is performed through a transformation discovered by the fluorescence fluoroscopy registration engine 101. In one embodiment, annotations associated with a particular frame allow user queries against the video stream (see "Example Questions") to include these labels, such as anatomical structures. These queries are processed by the core registration engine 108 through semantic processing (images and labels are added to a semantic record), immediately retrieving and returning images from memory 111, enabling rapid and comprehensive case verification.

[0136] For example, a clinician searching for multiple frames related to a specific organ can find them all without having to review the entire video sequence, eliminating the need for extensive review of long videos. In one embodiment, as shown in Figure 4B, this method is combined with the labels of the event classifier 112 to further facilitate the incorporation of labels extracted from the fluoroscopic video sequence itself by the core registration engine 108 into user queries.

[0137] Another use case in one embodiment would be to create a semantic index of interventional procedures using the core registration engine of the fluorescence fluoroscopy registration engine, so that the most important / clinically relevant parts of therapeutic procedures such as stent insertion, needle biopsy, ablation, or contrast agent injection can be identified.

[0138] Another embodiment involves motion stabilization of the fluorescence fluoroscopy image by target tracking 700 (see Figure 7), a unique method enabled by the fluorescence fluoroscopy registration engine 101. In one embodiment, the fluorescence fluoroscopy stabilization of the present invention fixes a specific object in the same position within the video stream, even if that object was moving in the unedited video stream. For example, spinal procedures often involve conscious sedation, and the patient may move during the procedure. In the prior art, the target lumbar vertebrae move in the fluorescence fluoroscopy image along with the patient's overall movement during the procedure, requiring the clinician to make adjustments each time. However, with the fluorescence fluoroscopy registration engine 101 of the present invention, the object of the procedure can be fixed within the frame regardless of the actual patient movement, and only the relative movement of surrounding structures is displayed.

[0139] In other embodiments, the same method is applied to radiotherapy in which a tumor identified by fluoroscopy is fixed as the target location. In one embodiment, patient movement is tracked using a fluoroscopy registration engine 101 and fed back to the treatment delivery system as updated tracking information of the actual movement of the tumor. In one embodiment, the movement of the target captured by a core registration engine 108 is used to guide the radiation beam used for the treatment of the patient, and the radiation beam moves in conjunction with the tumor. Alternatively, the radiation beam is modulated or irradiated only when the tumor is located in a specific area. This maximizes the precision of the treatment and minimizes unnecessary radiation to surrounding soft tissues and bone, regardless of patient / tumor movement.

[0140] In particular, when it is not possible to accurately interpret all degrees of movement in real time, the fluorescence fluoroscopy registration system 100 of the present invention can use the movement history of the target to more accurately predict its position. In one embodiment, previous movements recorded by the fluorescence fluoroscopy registration engine 701 are modeled as a 3D path 702. In one embodiment, this path is positioned relative to the movement tracked by the fluorescence fluoroscopy image 703. In one embodiment, image registration creates a bridge between the modeled 3D motion and the 2D motion observed in the fluorescence fluoroscopy image 703. In one embodiment, a four-dimensional (4D) image sequence (a periodic time sequence array of 3D volumes) is used to create a 3D track that models the movement of the object. Conversely, in order to model the movement of a 3D target derived from fluorescence fluoroscopy images 703 from multiple angles, periodic movements in 2D are synchronized and the path of the object is triangulated. This is particularly useful for periodically repeating movements such as heartbeats or continuous breathing.

[0141] In one embodiment, if there is a change in the periodic movement of the target during a session (for example, rapid breathing), the 3D target tracking model is adjusted by the fluorescence fluoroscopy registration engine 701 (see reference numeral 704). In one embodiment, when a trend is detected by the fluorescence fluoroscopy registration engine 701 during the adjustment of the target path 705, subsequent adjustments are pre-applied to the target tracking model, enabling dynamic correction of the 3D motion path from the 2D fluorescence fluoroscopy image by the core registration engine 108.

[0142] Adjusting for periodic motion is useful in radiotherapy when the target is affected by respiratory motion. While current standard medical practices provide sufficient margins for treatment regardless of whether the procedure is performed using a fixed beam, motion gate, or image-guided robot, there is still a risk of harmful radiation over-treating healthy tissue. The periodic motion adjustment based on fluorescence fluoroscopy registration results of the present invention allows for adjustment, i.e., reduction of these treatment margins, thus preserving healthy tissue.

[0143] In other embodiments relating to lung diseases, for some of the path of periodic movement of the lesion, the target moves within a loop where the initial path and the return path of the target are different. For example, in patients with chronic obstructive pulmonary disease (COPD), the third most common respiratory diagnosis, the path of target movement is often asymmetrical due to parenchymal lung injury seen with lung refraction and air bubbles due to scarring. Such asymmetric movement complicates current radiotherapy planning, adding margins to the entire area that may be treated. The shape and intensity of the radiation are configured to ensure that sufficient radiation reaches the lesion, but sufficient radiation doses are also delivered to healthy tissue, causing complications and worsening the prognosis.

[0144] However, as described above with reference to Figure 6, the technology of the fluorescence fluoroscopy registration system 100 of the present invention allows for more precise positioning of lesions in the respiratory cycle. In one embodiment, simply knowing which phase of the patient's respiration is in at that time allows for adjustment of the radiation beam during treatment, and a certain amount of radiation is irradiated only when it is known that the target is moving through one of the target paths. In one embodiment, narrowing the margin reduces the amount of radiation reaching healthy cells, thereby reducing complications and improving prognosis.

[0145] It is emphasized that the above embodiments of the present invention merely describe possible examples of implementation to clearly illustrate the principles of the present invention. The above embodiments of the present invention may be modified or improved in a manner that does not depart from the spirit or principles of the invention. All such modifications and improvements are intended to be within the scope of the present invention and protected by the following claims.

Claims

1. An image registration system including a fluorescence fluoroscopy registration engine that repeatedly improves a digitally reconstructed radiographic image (DRR) from a PACS image source using a fluorescence fluoroscopy image source, and matches the DRR image with the fluorescence fluoroscopy image, The aforementioned fluorescence fluoroscopy registration engine is located on either the client computer or the server. The aforementioned fluorescence fluoroscopy registration engine is A core registration module that creates candidate transformations to obtain the aforementioned DRR image and compares the DRR image with the fluorescence fluoroscopy image, Graphics memory module and Equipped with a repeating alignment module, The graphics memory module is A volume data storage that stores volume data from the PACS image source, A video data storage device for storing video data from the aforementioned fluorescence fluoroscopy imaging device source, Equipped with, The core registration module controls the acquisition of data from the PACS image source and the fluorescence fluoroscopy device source, and the transmission and storage of such data to the graphics memory module. The aforementioned repeating alignment module is The DRR image was evaluated for its agreement with the fluorescence fluoroscopic image. The process for generating a new DRR image using a new candidate transformation from the core registration module is repeatedly performed until a match is achieved with the new DRR image. The aforementioned new candidate transformation is applied to a structure in volumetric space, fused with the fluorescence fluoroscopy image to form a transformed image, and image registration is performed. The aforementioned core registration module, A 3-parameter registration demonstrating 2D rigid body transformation is performed on the aforementioned new DRR image. An image registration system that performs 6-parameter registration after the aforementioned 3-parameter registration to understand the variation in the patient's body rotation out of plane and the distance from the fluoroscopy imaging device source.

2. The system further comprises an input memory module that receives and stores the data from the PACS image source and the fluorescence fluoroscopy device source prior to transmitting the data to the graphics memory module, The aforementioned input memory module is A volume data storage for storing volume data from the PACS image source, and The image registration system according to claim 1, further comprising a video data storage for storing video data from the aforementioned fluorescence fluoroscopy imaging device source.

3. The device further comprises a frame acquisition module connected to the source of the aforementioned fluorescence fluoroscopy imaging apparatus, The image registration system according to claim 2, wherein the frame acquisition module acquires individual digital frames from the video data stream from the fluorescence imaging device source and transmits the video data to the video data storage of the input memory module.

4. The aforementioned core registration module, The system further includes an event classifier that autonomously determines the status of the aforementioned image registration. The event classifier acquires multiple fluorescence fluoroscopy image frames acquired from the fluorescence fluoroscopy image acquisition device source by the frame acquisition module as input, The image registration system according to claim 3, wherein the core registration module analyzes the characteristics of the plurality of fluorescence fluoroscopy frames at that time, compares them with the DRR image, confirms the match, and dynamically performs the image registration.

5. The computer system is further equipped with a display unit, The core registration module includes a graphical user interface (GUI) module, The graphical user interface (GUI) module is It assists in reading, displaying, and communicating with the fluorescence fluoroscopic images received from the input memory module. The image registration system according to claim 4, which enables the user to communicate with the fluorescence fluoroscopy registration engine and to display at least one of the fluorescence fluoroscopy image and the converted image on the screen of the display unit.

6. The image registration system according to claim 5, wherein the graphics memory module further comprises an output frame buffer module that receives the converted image from the iterative alignment module and transmits the converted image as a rendered image to the display unit.

7. The aforementioned repeating alignment module, A candidate transformation module that constitutes the new candidate transformation, using the start parameters of the DRR image and one of the previously transformed DRR images, A digital reconstruction module that acquires the new candidate transformation and generates the new DRR image, and The image registration system according to claim 6, further comprising a similarity module for determining the similarity between the new DRR image and the fluorescence fluoroscopy image and determining whether the image matches.

8. The aforementioned core registration module, The image registration system according to claim 7, further comprising an optimization module that guides the creation of the new candidate transformation by the candidate transformation module based on the similarity results confirmed by the similarity module.

9. To understand the nonlinear motion of the constituent elements, Non-rigid registration is performed on the aforementioned new DRR image. The image registration system according to claim 8, wherein each parameter of the multi-parameter non-rigid registration is capable of modeling local deformations within the new DRR image.

10. A method for performing image registration, Using a graphics user interface (GUI) module that provides a user interface and a fluorescence fluoroscopy image registration system, image registration is initiated on the display unit of the computer system. The fluorescence fluoroscopy registration engine of the fluorescence fluoroscopy registration system, located on either the client computer or the server, is started. The aforementioned fluorescence fluoroscopy registration engine performs the following steps: The core registration module of the aforementioned fluorescence fluoroscopy registration engine creates candidate transformations and obtains a digitally reconstructed radiographic (DRR) image. A step of comparing the DRR image with the fluorescence fluoroscopy image using the repeating alignment module of the fluorescence fluoroscopy registration engine, A step of evaluating the DRR image for its agreement with the aforementioned fluorescence fluoroscopic image, The step of repeatedly improving the DRR image using a new candidate transformation from the core registration module and generating a new DRR image until a match is achieved with the new DRR image, The steps include applying the new candidate transformation that resulted in the aforementioned match to a structure in volumetric space, The steps involve fusing the new candidate transformation with the fluorescence fluoroscopy image to obtain a transformed image and performing image registration. Perform The steps include: performing a 3-parameter registration representing a 2D rigid body transformation on the new DRR image using the core registration module; and Following the 3-parameter registration, a 6-parameter registration is performed using the core registration module to understand the variation in the patient's out-of-plane rotation and the distance from the fluoroscopy imaging device source. An image registration method further including the following.

11. The aforementioned core registration module, Acquisition of data from PACS image source and fluorescence fluoroscopy device source, The image registration method according to claim 10, which controls the transmission and storage of such data to a graphics memory module.

12. The steps include: capturing individual digital frames from a stream of video data from the fluorescence fluoroscopy device source using a frame acquisition module connected to the fluorescence fluoroscopy device source; and The step of transmitting the aforementioned video data to the video data storage of the input memory module of the fluorescence fluoroscopy registration engine. The image registration method according to claim 11, further comprising:

13. Using the event classifier of the aforementioned iterative alignment engine module, The process further includes a step of autonomously determining the status of the image registration, The event classifier takes as input multiple fluorescence fluoroscopy image frames acquired by the frame acquisition module from the fluorescence fluoroscopy image acquisition device source, The image registration method according to claim 12, wherein the core registration module analyzes the characteristics of the fluorescence fluoroscopy image frame at that time among the plurality of fluorescence fluoroscopy image frames, compares them with the DRR image for matching, and dynamically performs the image registration.

14. The fluorescence fluoroscopic image received from the input memory module and The converted image received from the output frame buffer module and The step of providing a display unit on which at least one of the above is displayed, thereby enabling the user to read, display, and communicate with the fluorescent fluoroscopic image. It further includes, The image registration method according to claim 13, wherein the output frame buffer module receives the converted image from the iterative alignment module and transmits the converted image to the display unit as a rendered image.

15. To construct the aforementioned new candidate transformation The steps include: using the start parameters of the DRR image and any of the previously transformed DRR images, and using the candidate transformation module of the iterative alignment module to construct the new candidate transformation; The steps include generating the new DRR image using the digital reconstruction module of the iterative registration engine, The steps include: determining the similarity between the new DRR image and the fluorescence fluoroscopy image using the similarity module of the iterative alignment engine, and determining whether the image matches; The image registration method according to claim 14, further comprising:

16. The image registration method according to claim 15, further comprising using the optimization module of the core registration module to create the new candidate transformation of the candidate transformation module from the similarity results confirmed by the similarity module.

17. To understand the nonlinear motion of the constituent elements, The step of performing non-rigid registration on the new DRR image. It further includes, The image registration method according to claim 16, wherein each parameter of the multi-parameter non-rigid registration is capable of modeling local deformations in the new DRR image.

Images