Intraoperative two-dimensional to three-dimensional registration adjustment method and system

Abstract

A method for correcting a position of a virtual target in a patient during a surgical site procedure includes segmenting an organ and a real target from an initial pre-surgical image dataset; receiving a patient real-time surgical image dataset including organs and real targets and camera tracking information; registering the initial preoperative image data set and the real-time image data set; determining a candidate position of a virtual target of the real target based on the initial registration; generating a first image under a first view angle for displaying the virtual target and the real target; adjusting the candidate position of the virtual target to match the actual position of the real target in the first image; and calculating the correction position of the virtual target according to the adjusting step. Related systems are also disclosed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to provisional application No. 63 / 400,044, filed on August 22, 2022, and entitled “Intraoperative Two-Dimensional to Three-Dimensional Registration Adjustment Method and System.”

[0003] Technical field of the invention

[0004] The present invention relates to surgical procedures, and more particularly to assisting physicians in tracking and guiding during surgical procedures. Background Art

[0005] Minimally invasive surgery is surgery done with only a small incision or no incision at all, usually using an endoscope, bronchoscope, laparoscope, or similar instrument.

[0006] For example, in bronchoscopic surgery, a bronchoscope is inserted through the patient's nose or mouth, advanced through the trachea and into the desired airway. The surgery can then be performed through the working lumen of the bronchoscope. A light source and camera at the tip of the bronchoscope enable the doctor to observe the airway wall in real time. A skilled doctor can identify their location in the airway and navigate along the airway wall to the desired location.

[0007] However, it is often necessary to supplement endoscopic visualization with radiographic guidance (e.g., by taking real-time X-ray images of the area using a fluoroscopy device). In certain procedures, radiographic guidance is necessary.

[0008] For example, in a transbronchial needle aspiration (TBNA) procedure, a long flexible catheter with a needle at the tip is advanced through the working lumen of a bronchoscope to the target's planned entry site. The needle is then advanced through the airway wall to a location outside the bronchoscope's field of view to aspirate a tissue sample. Once the needle is outside the bronchoscope's field of view, it is highly desirable or necessary to use fluoroscopy or alternative means to view and track the needle.

[0009] There are existing route planning, guidance and tracking techniques to help doctors reach the target site. Examples of such techniques are described in the literature. See, for example, U.S. Pat. No. 9,675,420 by Higgins et al. and U.S. Pat. No. 9,265,468 by Rai et al., each of which is incorporated herein by reference in its entirety. In order to perform such techniques, real-time image data from a fluoroscopy apparatus of the organ concerned must be registered with data of pre-collected three-dimensional image data of the organ. Matching can be performed using image-based feature matching. An example of fast two-dimensional-three-dimensional registration is described in U.S. Pat. No. 9,886,760 by Liu et al., the entirety of which is incorporated herein by reference.

[0010] Nevertheless, registration is still prone to errors, that is, the three-dimensional model of the organ calculated in real time does not match the pre-acquired data. Such errors are particularly prevalent in the registration of relatively non-rigid organs such as the lungs. In addition, during surgery, the target tissue (e.g., a suspected lesion) identified in the three-dimensional model of the anatomical structure based on the pre-acquired image data may not align with the actual location of the target tissue. Errors may occur for a variety of reasons, including, for example: (a) the patient position of the pre-acquired image data is different from the patient position of the real-time image data; (b) the point in the patient's respiratory cycle of the pre-acquired image data is different from the point in the real-time image data; and / or (c) the patient is unable to hold his breath completely when the pre-acquired image data is acquired (e.g., a rigid endoscope may prevent complete breath holding or the endoscope adapter has leaks). These errors are undesirable for many reasons, not just inaccurate display of the virtual route to the virtual target.

[0011] Therefore, a method and system for solving the above errors are needed. Summary of the invention

[0012] A method and system for correcting the position of a virtual target in a patient's body during an on-site surgical operation, comprising: receiving a preoperative image dataset (e.g., a CT image dataset) of a patient including an organ and a real target; segmenting the organ and the real target from the preoperative image dataset; receiving a real-time image dataset of the patient (e.g., surgical fluoroscopy image data including camera tracking information corresponding to the image); registering the pre-recorded image dataset and the real-time image dataset; determining a candidate position of a virtual target of the real target based on the initial registration; generating a first image at a first perspective showing the virtual target and the real target; adjusting the candidate position of the virtual target to match the actual position of the real target in the first image; and calculating a corrected position of the virtual target based on the adjustment steps.

[0013] In an embodiment, the method also includes generating a second image at a second perspective showing the virtual target and the real target; adjusting the candidate position of the virtual target to match the actual position of the real target in the second image; and, wherein, the corrected position of the virtual target is calculated based on the adjustment step performed on the first image and the adjustment step performed on the second image.

[0014] In another embodiment of the present invention, a system or workstation includes at least one processor, which is operable to receive a preoperative image dataset of a patient including an organ and a real target; segment the organ and the real target from the preoperative image dataset; receive a real-time image dataset of the patient, which includes camera tracking information corresponding to the image; align the pre-recorded image dataset and the real-time image dataset; determine a candidate position of a virtual target of the real target based on an initial alignment; and calculate a corrected position of the virtual target based on receiving an updated or adjusted position of the virtual target.

[0015] In an embodiment, the processor is also operable to: generate a second image showing the real target and the virtual target at a second perspective; and calculate the corrected position of the virtual target based on the user adjustment of the candidate position performed on the first image and the user adjustment of the candidate position performed on the second image.

[0016] In an embodiment, the system or workstation is provided in the form of a desktop computer, a portable computer or a laptop computer.

[0017] In an embodiment, the graphical user interface comprises a plurality of windows displaying different image views of the real object and the virtual object, and at least one panel for accepting user input and instructions.

[0018] The description, objects and advantages of the present invention will become clear from the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic diagram of an operating room environment including a workstation according to an embodiment of the present invention;

[0020] Figure 2 is a block diagram of a surgical tracking system according to an embodiment of the present invention;

[0021] Figure 3 is a flow chart of a method for registering real-time two-dimensional fluorescence images and camera tracking data with pre-acquired three-dimensional image data;

[0022] Figure 4A is a top perspective view of a patient plate including a cross-shaped patient plate tool according to an embodiment of the present invention;

[0023] Figure 4B is an enlarged view of the patient plate tool shown in FIG. 4 a according to an embodiment of the present invention;

[0024] Figure 4C is an exploded view of the patient plate according to the embodiment of the present invention shown in FIG. 4 a;

[0025] Figure 4D is an enlarged view of a portion of a patient plate according to an embodiment of the present invention shown in FIG. 4c;

[0026] Figure 5 is a flow chart of a virtual target calibration method according to an embodiment of the present invention;

[0027] Figure 6 is a screenshot of a 2D fluoroscopy view showing misalignment of the actual and virtual targets;

[0028] Figure 7 is a screenshot taken at the first C-arm position showing the actual target being marked;

[0029] Fig. 8A is a screenshot taken at the second C-arm position showing the actual target being marked;

[0030] Figure 8B is another screen shot at a selected C-arm position showing the VTC panel window;

[0031] Fig. 9 Schematic diagram of real and virtual targets in CT coordinate space and fluorescence coordinate space at two different C-arm positions;

[0032] Fig.10 is a screenshot of the first C-arm position showing the alignment of the corrected virtual target with the actual target;

[0033] Fig.11 is a screenshot of the second C-arm position showing the alignment of the corrected virtual target with the actual target;

[0034] Fig.12 is an anteroposterior (A / P) view of the patient's airway model; and

[0035] Figures 13A-13B are screen shots of various graphical user interfaces according to embodiments of the present invention. DETAILED DESCRIPTION

[0036] Before describing the present invention in detail, it should be understood that the present invention is not limited to the specific variations set forth herein, because various changes or modifications may be made to the described invention and may be replaced by equivalents without departing from the spirit and scope of the present invention. It will be apparent to those skilled in the art that, after reading this disclosure, each of the various embodiments described and illustrated herein has discrete components and features that can be easily separated or combined with the features of any one of the other several embodiments without departing from the scope or spirit of the present invention. In addition, many modifications may be made to adapt specific circumstances, materials, material compositions, processes, process actions or steps to the goals, spirit or scope of the present invention. All of these modifications are intended to fall within the scope of the claims set forth herein.

[0037] The methods described herein may be performed in any order of events described that is logically possible, as well as in the order of events described. In addition, where a numerical range is provided, it is understood that each intermediate value between the upper and lower limits of the range, as well as any other specified value or intermediate value within the specified range, is encompassed within the present invention. Furthermore, it is contemplated that any optional features of the described inventive variations may be independently formulated and claimed, or combined with any one or more of the features described herein.

[0038] All prior subject matter mentioned herein (eg, publications, patents, patent applications, and hardware) is incorporated herein by reference in its entirety, except to the extent that such subject matter might conflict with the subject matter of the present invention, in which case the subject matter herein controls.

[0039] Reference to a single item includes the possibility that there are multiple identical items. More specifically, as used herein and in the appended claims, the singular forms "a", "an", "said", and "the" include plural referents unless the context clearly indicates otherwise. It should also be noted that the claims can be written to exclude any optional elements. Therefore, this statement is intended to serve as a prior basis for the use of exclusive terms such as "only", "only" and the like in conjunction with the narration of the claim elements or the use of a "negative" limitation. It should be understood that unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the invention belongs.

[0040] Operating Room and Workstation Overview

[0041] Figure 1A schematic diagram of a surgical environment in an operating room including a workstation 50 according to the present invention is shown. A subject 10 is shown on a calibrated patient board 32 on an operating table 20, with a surgical device 30 positioned and extended into the subject's lungs. The surgical device 30 has a distal working end or tip 31 that has passed through the subject's mouth, trachea, bronchi, and into the lungs. Although Figure 1 The illustrated surgical device 30 is intended to represent an endoscope, ie, a bronchoscope, but the invention is not limited thereto.

[0042] The surgical device may be any device, instrument, implant, or marker that is visible under fluoroscopy, has a portion that is visible under fluoroscopy, or is modifiable so that it is visible under fluoroscopy. Examples include, but are not limited to, catheters, sheaths, needles, ablation devices, forceps, brushes, biopsy needles, stents, valves, coils, seeds, and fiducial markers.

[0043] Fluoroscope 40 captures real-time fluoroscopic video of the subject. Video frames of the video or images are collected or received by workstation 50 for processing. The real-time image can also be displayed on video monitor 62.

[0044] The position and attitude of the fluoroscopy camera 42 is tracked using a calibrated tracking sensor 44. Figure 1 In the fluoroscopy unit shown, an optical tracking sensor 44 observes an optically visible symbol 46 to provide information about the position, direction and posture of the fluoroscopy camera 42 to the workstation 50 in real time (unless the context clearly indicates otherwise, sometimes referred to herein as "camera tracking information"). The optical tracking sensor 44 can be positioned on any fixed surface, including a floor, wall, ceiling or table. Optionally, the optical tracking sensor 44 can be positioned on a movable platform that can be locked in place. An example of an external sensor is a Polaris Spectra tracker manufactured by NDI (Northern Digital Inc., Waterloo, Ontario, Canada). The tracking sensor 44 is configured to detect the position of the detection tool 46 (optionally, the detection tool 90 is to track the posture of the patient board 32), which can be in the form of a cross symbol as shown. Although the tracking sensor shown in this embodiment is based on optics, other technologies (e.g., electromagnetic) according to the present invention can be used to track the fluoroscopy camera or patient board.

[0045] Next, as discussed in greater detail herein, the location of the target tissue shown in the real-time video from the fluoroscopy unit 40 is registered or mapped to a three-dimensional location in a model of the organ generated from a previously acquired image dataset (eg, CT data) of the patient's organ.

[0046] The display 60 can operate in conjunction with the workstation 50 to display various types of images, including three-dimensional model views, two-dimensional model fluoroscopic views, real fluoroscopic views, real endoscopic views, model endoscopic views, and various information superimposed on the views, such as but not limited to planning information, regions of interest, virtual target markers, blood vessels, virtual obstacles, real equipment, virtual equipment, routes to reach the target, user-provided annotations and markers, etc.

[0047] System Block Diagram

[0048] Figure 2 An intraoperative image registration system 88 comprising a workstation or specially programmed computer 50 is shown. Figure 2 The illustrated workstation 50 includes at least one processor 70 operable to correct the three-dimensional position of a virtual object, as will be described in greater detail below.

[0049] The workstation 50 is also shown having a storage device 80 that holds or stores information including imaging, equipment, marking and program data. The storage device may be, for example, a hard drive.

[0050] Figure 2 The workstation 50 shown in is adapted to receive real-time images (e.g., fluoroscopic or endoscopy images) and camera tracking information through various input ports or connectors (e.g., USB ports, video ports, etc.). A frame grabber 72 is preset to acquire individual video frames or images for processing. Real-time fluoroscopic images can be obtained by the workstation 50, for example, from continuous fluoroscopic video, video clips, and / or still frames. The workstation is also adapted to send image data to a display using a video card 82. An example of a workstation is a Dell computer model 7820, a dual Intel Xeon Gold 6136 processor, and an Nvidia Quadro RTX 4000 video card.

[0051] Figure 2 The illustrated system 88 also includes a display 84 that can present reports, data, images, results, and models in various formats including, but not limited to, graphical, tabular, and pictorial forms. In one embodiment of the invention, a virtual target is superimposed on a real-time two-dimensional fluorescent image of an organ.

[0052] However, it should be understood that although Figure 2 The system in is shown as having a memory 80 for receiving and storing various information, but the invention is not limited thereto. In alternative embodiments, the system may be configured to access only a storage device, such as a USB stick, a CD or other media storage device.

[0053] Figure 2 The illustrated system 88 also includes a user input device 86, such as a keyboard, joystick, or mouse. The user input device allows a user, such as a physician, to add or enter data and information, draw, summarize, select, and modify planning information, and make notes in files and records.

[0054] In another embodiment, the processor 70 can be connected to the memory device via the Internet or through another communication line (e.g., Ethernet) to access the network. For example, patient CT scan data can be stored on a server in a hospital, and the processor of the present application is suitable for accessing the data via the communication module 98 and processing the data. Class examples of communication module types include wireless (e.g., Bluetooth, Wi-Fi) as well as landline and Ethernet.

[0055] The display 84 may be incorporated with the processor in an integrated system (e.g., a laptop or larger tablet computer), or the display may cooperate with the processor from a remote location. The processor may be adapted to provide a display to one or more displays, tablet computers, or portable computer devices or smart phones (e.g., Apple Computer, Inc. manufactured by Cupertino, California, USA) via a network. ) to send or transmit data. In fact, although Figure 2 The computer system 88 shown in FIG. 8 includes a number of different components incorporated into the system, but the invention is not so limited. The present invention is intended to be limited only by the claims that follow.

[0056] Initial Registration Overview

[0057] Figure 3 1 is a flow chart showing a registration method 120 for assisting a physician during a surgical site operation according to the present invention. The steps may be performed on a computer or system and include: step 122, creating a three-dimensional model of a body organ; step 124, receiving at least one real-time fluoroscopic image of the body organ, including camera tracking information corresponding to the image; step 126, registering three-dimensional points from the model to two-dimensional fluoroscopic points in at least one fluoroscopic image; and step 128, deforming the three-dimensional model of the body organ to match the real-time organ.

[0058] In one embodiment, step 122 of creating a three-dimensional model of a body organ includes creating a three-dimensional model of a non-rigid body organ, such as a lung, in a first body position or a first patient position. The three-dimensional model of the body organ is created based on input including available image data from a subject, such as a high-resolution computed tomography (HRCT) scan. The three-dimensional model and related information are defined in a previously acquired image data (e.g., CT) coordinate space.

[0059] It should be understood that other acceptable data sets include, but are not limited to, MRI, PET, three-dimensional angiography, and X-ray data sets. In an embodiment, the workstation receives a three-dimensional image file, a three-dimensional image data set, or a set of two-dimensional images of an organ, from which a three-dimensional model of the organ can be calculated. For example, the workstation can communicate with a medical digital imaging and communications (DICOM) server to receive these data sets. Exemplary techniques for determining a three-dimensional model of a body organ are disclosed in U.S. Patent No. 7,756,316 entitled "Methods and Systems for Automatic Lung Segmentation". See also Patents Nos. 7,889,905 and 7,756,563, both to Higgins et al.; and Patent Publication No. 2008 / 0183073.

[0060] Next, step 124 details receiving real-time fluorescent image data. Camera tracking information corresponding to the real-time image is also collected. The real-time image data of the body organs and the fluoroscopy camera and patient board position is obtained when the subject is on the operating table, which may not be exactly the same as the first position. For example, the patient may be curled, overextended, at a different level of inspiration, or in a different body posture on the operating table compared to during the preoperative three-dimensional scan. Certain organs (e.g., lungs) may be deformed due to the subject's body posture, patient orientation, inspiration level and position. Therefore, the three-dimensional model of the object in the first position may not match the three-dimensional model of the object in the second real-time position.

[0061] Step 126 details registering the 3D based model image with the real-time fluoroscopic image. Exemplary types of 3D-2D registration may be intensity based or position based.

[0062] The intensity-based method is a preferred embodiment for matching projections (virtual fluoroscopic images) for a given real-time fluoroscopic image. The matching criteria can be mutual information, cross-correlation, etc. It should also be understood that step 126 provides for registering one or more points. In this way, patterns or point sets corresponding to targets or markers, for example, can be registered.

[0063] Position-based registration requires the user to go to a known location under fluoroscopy and perform registration based on known locations or anatomical landmarks. For example, the device can be moved to the main tracheal carina and the 3D model image is matched to the 2D fluorescence image at that known location. Registration techniques can also be found in patents Nos. 7,889,905 and 7,756,563.

[0064] The registration step 126 may additionally be based on applying a 3D deformable model (step 128) to the 3D image. The 3D deformable model is desirable because the 3D model of the object in the first position may not match the 3D model of the object in the second real-time position. The 3D deformable model may be previously available or estimated by step 128. The 3D deformable model applied to the 3D image generates a set of modified 3D positions, which are then used for registration.

[0065] In one embodiment, the loop 129 between step 126 and step 128 is continued until the maximum estimated deformation in step 128 is less than a threshold value. Figure 3 The threshold value is shown in step 130. An example range of the threshold value is 0.5 to 5 mm and can be set to about 1 mm. In addition, step 126 and step 128 can be performed separately or together as a joint estimation of the CT pose relative to the fluoroscopic camera and the deformable model.

[0066] The 3D deformable model can be refined so that any point in the fluoroscopic image can be matched to a point in the 3D model.

[0067] In one embodiment, a three-dimensional model of a body organ is deformed and a two-dimensional projection image of the three-dimensional model is compared to a two-dimensional projection image of a real-time two-dimensional fluoroscopic image. A two-dimensional delta or difference between the image created by the three-dimensional model and the real fluoroscopic image is calculated. This two-dimensional difference gives a constraint on the three-dimensional motion of a reference marker using the fluoroscopic camera pose. More such constraints can be generated from multiple fluoroscopic images to give the three-dimensional motion of the reference marker. This estimated three-dimensional motion can then be propagated to neighboring three-dimensional points using a smoothness constraint or an optical flow constraint. By Pinar Muyan-Ozcelik ( Muyan-Ozcelik's "Fast Deformable Registration on GPU: CUDA Implementation of Demons Algorithm" in 2008 gives an example of optical flow constraints applied to deformable models. Higgins et al.'s patent No. 7,889,905 also discloses a warping algorithm for warping a live image with a model image. See also patent No. 9,886,760 to Liu et al.

[0068] In an embodiment, a three-dimensional transformation matrix for organ and target locations is calculated by mapping each point in the three-dimensional virtual model (and typically in the CT coordinate system / space) to two-dimensional real-time or procedural image data (typically in the patient table or procedural coordinate system / space).

[0069] Optical parameters Fluorescence camera calibration

[0070] Although not specifically described in method 120, a fluoroscopic camera calibration step is typically performed on each new C-arm when a certain period of time (e.g., 6 months) has passed since the last C-arm calibration, or when maintenance is completed on the C-arm. The calibration data can be obtained offline and calculated by acquiring multiple fluoroscopic images of radiopaque markers to determine data such as the focal length and camera center of the fluoroscopic camera, as well as a characterization of the deformation pattern (a checkerboard pattern appears curved when viewed in a fluoroscopy instrument), and determine changes in these parameters when the fluoroscopy instrument is rotated throughout its range of motion. The calibration coefficients can be specific to each fluoroscopy instrument. Examples of calibration techniques include those described in Long, L., Dongri, S. (2019). Review of Camera Calibration Algorithms, in Bhatia, S., Tiwari, S., Mishra, K., Trivedi, M., eds., Advances in Computer Communication and Computational Sciences; volume 924 of the series Advances in Intelligent Systems and Computing, pp. 723–732, Springer, Singapore; https: / / link.springer.com / chapter / 10.1007 / 978-981-13-6861-5_61.

[0071] Fluoroscopic camera pose - patient table position - calibration

[0072] Throughout the present invention, reference is made to tracking of the camera, i.e., tracking the position of the camera using the sensor 44. When performing camera tracking, it is desirable to utilize a common coordinate system, and in an embodiment, the camera position is mapped to a patient table marker (LM) coordinate system or any available coordinate system. Again, referring to Figure 1In an embodiment, the fluoroscopy camera image coordinate system is calibrated with the patient table landmark (LM) coordinate system by using a custom patient plate 32, an external tracking sensor 44, a fluorescent tool 46, and a patient tool 90 attached to the patient plate 32 fixed to the operating table 20. Examples of external sensors 44 are Polaris Spectra or Vega trackers, both of which are manufactured by NDI located in Waterloo, Ontario, Canada. As described herein, the sensor 44 is operable to track the fluorescent tool 46 on the fluorescent camera in its three-dimensional space (hereinafter sometimes referred to as "NDI space").

[0073] In addition, referring to Figures 4a-4b, the sensor 44 is operable to position the patient tool 90 in the NDI space. The patient tool 90 is shown as being coupled to the patient plate 32 via a mounting member 96. Preferably, the patient tool 90 is fixed on the side closest to the workstation 50. The tool 90 is shown as having a plurality of arms 92 of different lengths, which are arranged perpendicular to each other. Markings 94 are shown at the end of each arm. These markings can be detected by the sensor 44 to accurately determine the position in the NDI space.

[0074] In addition, referring to Figures 4c-4d, the patient plate 32 is characterized by a patient landmark (LM) origin plate 33, which includes a set of non-transmissive markers 34a-34e to define the origin and XY axes of the landmark (LM) coordinate system. The markers 34a-34e can be detected by the fluoroscopy camera. The position of the markers 34a-34e relative to the patient tool 90 is fixed and known. Therefore, because the NDI sensor 44 can detect the position of the camera 42 through the camera markers 46 and the position of the patient plate through the patient tool 90, and the patient table LM origin (also known as the surgical field space) is known relative to the patient tool 90, the system is able to calculate the coordinate system from the NDI space to the fluorescent tool 46 space. and NDI space-to-patient tool 90 space transformation, and vice versa.

[0075] It should also be understood that, although various tools, sensors, specific coordinate transformations or mappings are described above, embodiments of the present invention may employ other optical and coordinate system calibration techniques. For example, another method for determining calibration parameters is described in U.S. Pat. No. 9,693,748 to Rai and Wibowo. Examples of patient tables and radiopaque markers are shown in Design Patent No. D765865, entitled “PATIENT POSITIONING TABLE”, filed on February 10, 2015, and Design Patent No. D820452, entitled “RADIO-OPAQUE MARKER”, filed on July 21, 2016, respectively.

[0076] Respiratory cycle compensation

[0077] A respiratory motion curve corresponding to the patient's respiratory cycle can also be created from the CT data or otherwise input to the workstation. This can be input or received in the form of image data of the bronchial tree and airways at multiple time points corresponding to inspiration, exhalation, and possibly one or more time points between inspiration and exhalation. This data can be processed to identify displacements of tissues and tissue surfaces. The image data at multiple time points is reviewed in a timely manner to accurately display the patient's respiratory motion curve. Pinar Muyan-Ozcelik ( An example procedure for doing this is described in "Fast Deformable Registration on the GPU: A CUDA Implementation of the Demons Algorithm" by Muyan-Ozcelik, 2008, where the authors describe using a pair of CT scans acquired from the same patient, one at full inspiration and the second at full expiration. The deformable registration technique described in this paper gives a mapping of the geometric position of each discrete point within the lungs from exhalation to inspiration and vice versa. From these mappings, the position of any region within the chest during the respiratory cycle can be estimated. Multiple pairs of scans (e.g., full inspiration and full expiration scans from multiple individuals) can also be used to create this dataset.

[0078] Respiratory motion can also be tracked by placing a tool on the chest, which is tracked by the sensor. This motion can be used as an input and processed within the system described herein to indicate when a fluorescent image should be taken.

[0079] Virtual target calibration

[0080] Figure 5 is a flow chart illustrating a virtual target calibration method 200 for assisting a physician during a surgical site operation according to an embodiment of the present invention.

[0081] Step 250 illustrates determining a candidate location of a virtual target. This step is performed based on the above combined Figure 3 The described deformation model is performed by mapping the target coordinates from the segmentation model to the intraoperative image dataset.

[0082] Step 260 illustrates generating a first image showing the virtual target and the real target. Figure 6 , a screenshot of a display is shown that includes an intraoperative anteroposterior (AP) view of a portion of a patient's lungs with a virtual target 310 superimposed thereon. The virtual target can be superimposed on the two-dimensional fluorescent image based on the known coordinates of the virtual target determined according to step 250 above. However, as shown, the virtual target 310 may be misaligned with the actual target 320. Some reasons for the virtual target 310 to be misaligned with the actual target 320 include patient positioning and breathing levels, whether by a ventilator or otherwise. Regardless of the cause, misalignment is undesirable because it makes the physician's job more difficult to determine the location of the actual target and may put the patient at risk.

[0083] Step 270 illustrates adjusting the candidate position to match the true target. Figure 7 , taking fluoroscopic images of organs and targets at a desired respiratory level (e.g., peak inspiration). The view can be in the anterior-posterior direction (AP), or rotated an angle (Ω) degrees clockwise (CW) or counterclockwise (CCW) along the anterior-posterior direction (AP), and the C-arm away from or toward the view. Ω (Omega) can vary. In an embodiment, Ω ranges from 20-40 degrees from the anterior-posterior direction (AP). In addition, the respiratory cycle level of the first image is also recorded.

[0084] Next, the doctor marks the actual target 330 in the first view. In an embodiment, the computer is operable to accept user input using a mouse, keyboard or touch screen to select the position of the correction. The user input can be in the form of highlighting, outlining, dragging (e.g., dragging a shape) or other visual tools.

[0085] In an embodiment, the computer is operable to present a virtual marker based on the position of the mouse pointer on the screen. The user moves the mouse pointer to the actual target on the screen (preferably the center) and clicks the mouse to record the desired position of the target 330, such as Figure 7 shown.

[0086] In an embodiment, based on clicks or other signals from the user, the system is programmed and operable to collect and calculate the following information for virtual target correction: three-dimensional position in the CT coordinate system, three-dimensional position in the fluorescence source coordinate system, and three-dimensional position in the patient table landmark (Table LM or surgical space) coordinate system.

[0087] In a preferred embodiment, reference Fig. 8A, the physician marks the target 340 in the second fluoroscopic image, preferably at least 30-60 degrees from the first image viewing angle of the fluoroscopy machine. Optionally, this step can be repeated from a plurality of different viewing angle ranges to provide updated position information for correcting the virtual target position discussed below. In an embodiment, the computer is programmed to suggest each viewing angle including -30° from the anteroposterior direction (AP), +30° from the anteroposterior direction (AP), and / or the anteroposterior direction (AP), or other angles if the C-arm is calibrated at other angles, such as + / -25° from the anteroposterior direction.

[0088] The second and additional images are preferably taken at a respiration level that is the same or similar to the respiration level recorded in the first image.For each additional image taken, a corresponding data set is collected and calculated in conjunction with the first C-arm position image data as described above.

[0089] Figure 8B is another screenshot showing target 340 marked in a second view (29 degrees counterclockwise). Figure 8B Also included is a VTC panel window 410 that opens when the Adjust Registration tab is selected. The VTC panel window 410 provides a user interface according to an embodiment of the present invention for a physician to conveniently calculate a corrected position of a virtual target as described herein.

[0090] Step 280 illustrates calculating the corrected position of the virtual target.

[0091] refer to Fig. 9 We hope to update the CT-Table LM transformation matrix so that the initial virtual target position is accurately mapped to the true target position in the fluorescence image coordinate system / space.

[0092] To do this, we need to calculate vectors v1 and v2 where:

[0093]

[0094] Then we calculate v1 based on the information collected above, where Represents the vector from the origin of the fluorescence source at C-arm position 1 to the center of the target in CT space. In order to transform multiple points in the fluorescence source coordinate system at C-arm position 1 into multiple points in CT space, we need to calculate the transformation M FS,1→CT . Convert M FS,1→CTIt is obtained by the following transformations: 1) The transformation from the fluorescence source to the fluorescence tool at C-arm position 1 is obtained by C-arm calibration and is represented by FS→FT. 2) The transformation from the fluorescence tool to the NDI tracker is obtained by tracking the fluorescence tool by the NDI tracker and is represented by FT→NDI. Similarly, the transformation from the NDI tracker to the patient tool is obtained by the NDI tracker and is represented by NDI→PT. 3) The transformation from the patient tool to the patient table landmark coordinate system is obtained by patient table calibration and is represented by PT→PTL. 4) The transformation from the patient table landmark to CT is obtained by registration calculation and is represented by PTL→CT. From the above transformations, it can be seen that we get the transformation M through FS→FT→NDI→PT→PTL→CT. FS,1→CT . Combined Figures 4A-4D , step 4 is performed as described above; and

[0095] Then calculate it by 1) Get the point where the center of the real target clicked by the user on the screen is located. The point clicked by the user is expressed in the screen space coordinate system. 2) Convert the point in the screen space coordinate system to the image coordinate system space, where the image coordinates are expressed in pixel units. 3) Based on the calibration of the C-arm at position 1, associate a ray with the image coordinates in step 2. This ray is defined in the fluorescence source coordinate system and must be converted in CT space by M FS,1→CT The resulting ray will be defined as 4) Scale the ray transformed in step 3 so that it intersects the plane defined by the normal of the fluorescence source including the center of the CT target. The vector v1 will lie in the plane defined by the normal of the fluorescence source and radiate from the center of the CT target to the scaled ray defined in step 3 (which intersects the plane formed by the normal of the fluorescence source). The equation defining the plane is in By using M FS,1 →The rotation component of CT is applied to the vector <0,0,1>, and the point p can be calculated using arrive The scaling factor is calculated by projection. Application produce

[0096] Next, we calculate v2, where v2 can be obtained as follows:

[0097] Preliminary confirmation and The shortest distance between As determined above, It can be determined by A similar method is used to obtain (except that the C-arm is located at position 2 or (OFS,2 ) situation).

[0098] if and intersect, the intersection point is located and the vector v2 is calculated.

[0099] if and do not intersect, then they are skew lines in three-dimensional space, and we can apply the least squares fitting method to find the point where the two lines are closest.

[0100] In an embodiment, a bisection method is used to match the clicked point to the C-arm position 2 .

[0101] After the translation is calculated as shown above, the projection of the virtual target will be located on the actual target. However, we also need to correct the rotational characteristics of the virtual fluorescence and the intraoperative fluorescence. In an embodiment, the correction of the rotational characteristics is performed by incrementally rotating the rotational component of the CT→Table LM transformation (e.g., by manual adjustment by the user) to visually align common features, preferably hard features such as the spine. In an embodiment, the user visually aligns the spine in the virtual fluorescence and the real fluorescence, even if the spine is parallel between the virtual fluorescence and the real fluorescence.

[0102] The rotation is performed in a plane parallel to the patient plate around the center of the target (keeping the target fixed). Current fluoroscopic recordings can be used for this procedure.

[0103] After we obtain the translation and rotation characteristics to map the virtual target center to the programmed fluorescent target center, and obtain v1 and v2 as described above, we can update the registration transformation matrix (CT coordinate system to table landmark coordinate system transformation, i.e., CT→table landmark (Table LM)).

[0104] In an embodiment, the method may further include displaying the corrected virtual target. Figure 10-11 , the corrected virtual targets 350, 360 are shown in two different C-arm positions, namely counterclockwise 30 degrees (CCW30) and anterior-posterior (AP), respectively. Among other benefits, the display can also provide tracking and guidance information to the physician. In one embodiment, the virtual target is superimposed on a three-dimensional view of the body organ. In addition, since the three-dimensional position of the virtual target is calculated as described above, it can be superimposed on a two-dimensional fluorescent perspective model projection. In fact, multiple views of the display target as well as any surgical devices, additional markers, routes, markers, and planning information can be displayed together with the model or real image of the organ.

[0105] Graphical User Interface

[0106] Reference again Figure 6 , various windows of a graphical user interface (GUI) 500 according to an embodiment of the present invention are shown. The GUI 500 is shown to include a static fluorescence window 510, a fused fluorescence window 520, suggested angle windows 530, 532, 534 and an options window 540.

[0107] The static fluoroscopic window 510 is configured to display a complete fluoroscopic image of the patient based on the CT image data of the C-arm position selected by the user. Fig.12 As shown, static fluoroscopic views can be displayed in the anteroposterior (A / P) direction or in addition to Figure 6 The static fluorescence window is shown.

[0108] Reference again Figure 6 , the fused fluorescence window 520 is configured to display a real-time two-dimensional fluorescence image, wherein the virtual target 310 is calculated based on the pre-acquired CT image data and output from the registration on the two-dimensional fluorescence image as described above. As described above, the calculated virtual target 310 may not be aligned with the actual target 320.

[0109] Windows 530, 532, and 534 display suggested angles for the C-arm. The categories of suggested angles to be selected include (a) C-arm suggested angles 532 for the best viewing angle for a visible target when using the VTC discussed herein, and (b) C-arm suggested angles 534 for the best access path to the target.

[0110] In an embodiment, the C-arm angles are provided in the order of first best, second best, and third best viewing angles from the airway wall to the target.

[0111] In an embodiment, the VTC suggested angles are based on the contrast difference between the virtual target and the surrounding environment. In an embodiment, the C-arm angle that provides the highest contrast difference is the first suggested angle 530, the C-arm angle that provides the second highest contrast difference is the second suggested angle 532, the C-arm angle that provides the third highest contrast difference is the third suggested angle 534, and so on.

[0112] Window 540 displays various user options. Options include animation, target outline, and registration adjustment / virtual target correction (VTC).

[0113] Selecting the Animation tab provides animation between the current C-arm position fluoroscopic image and the corresponding virtual fluoroscopic image. In a VTC workflow, this information can be used to orient live and virtual anatomical structures such as the spine and ribs.

[0114] Selecting the Outline tab may provide the user with tools to mark or identify real objects in the image. Examples of tools include tracking, pointing / clicking, and dragging. In an embodiment, when the Outline tab is selected, a virtual object 310 displayed with fused fluorescence may be dragged by the user to the real object.

[0115] refer to Fig.13A , an "Adjust Registration" tab is presented according to an embodiment of the present invention. When the Adjust Registration tab is selected, a VTC panel window 560 is opened. The VTC panel window 560 is shown to include an instruction window, a virtual target incremental adjustment control with an undo feature, a selected registration data set (corresponding to the first shot, the second shot, etc.) acquisition (apply), recalculation and reset (i.e., restart) tab. The VTC options window also includes options to return to the initial registration and exit the tab.

[0116] Fig. 13B Another VTC panel window 660 is shown according to an embodiment of the present invention, which is opened when the Adjust Registration tab is selected. The VTC panel window 660 is shown to include an "Options" window that lists user instructions, a "Calculate" tab that starts the calculation of the virtual target correction, an "Apply" tab that applies the selected registration data set (corresponding to the first shot, the second shot, etc.), a "Restart" tab that recalculates the virtual target calculation, and a "Close" tab that cancels the immediate or current VTC and returns to the original registration.

[0117] Additional windows and tabs with corresponding functions may be added to achieve the purposes and features described herein. In addition, other modifications and variations may be made to the disclosed embodiments without departing from the present invention.

Claims

1. A method for correcting the position of a virtual target in a patient's body during an on-site surgical operation, comprising: receiving a preoperative image dataset of a patient, wherein the preoperative image dataset includes an organ and a real object; segmenting the organ and the real object from the preoperative image dataset; receiving a real-time surgical image dataset of a patient, wherein the real-time surgical image dataset includes the organ and the real target at actual positions and camera tracking information; registering the pre-operative image dataset and the real-time surgical image dataset to determine an initial pre-operative to real-time surgical transformation matrix; Determining candidate positions of a virtual target of the real target based on the pre-operative to real-time surgical transformation matrix; Generate a first image showing the virtual target and the real target at a first viewing angle; adjusting the candidate position of the virtual object to match the actual position of the real object in the first image; and A corrected position of the virtual target is calculated according to the adjusting step.

2. The method according to claim 1, comprising: generating a second image at a second viewing angle showing the virtual target and the real target; adjusting the candidate position of the virtual object to match the actual position of the real object in the second image; as well as Wherein, the corrected position of the virtual object is calculated based on the adjustment step performed on the first image and the adjustment step performed on the second image.

3. The method according to claim 2, wherein: The first viewing angle differs from the second viewing angle by at least 20 degrees, and optionally, wherein along a first direction, the first viewing angle is at least 20 degrees from the front-to-back (AP) direction, and along a second direction opposite to the first direction, the second viewing angle is at least 20 degrees from the front-to-back (AP) direction.

4. The method according to any one of claims 1 to 3, wherein: Adjusting the candidate position of the virtual object to match the actual position of the real object is performed by dragging the virtual object to the actual object in the first image and the second image.

5. The method according to any one of claims 1 to 3, wherein: Adjusting the candidate position of the virtual target to match the actual position of the real target is performed by marking features in or near the actual target in the first image and / or the second image.

6. The method according to claim 5, wherein: The feature is the center of the actual object, or alternatively the periphery of the actual object.

7. The method according to claim 2, wherein: The second image is taken at the same patient breathing level as the first image, and optionally the user is prompted when to take the second image based on breathing phase tracking and the point in the breathing phase at which the first image was taken.

8. The method according to claim 2, wherein: The calculating step includes determining a translational characteristic of the correction position.

9. The method according to claim 8, wherein: A translation characteristic from the candidate position to the actual position is determined based on determining vectors v1 and v2 in an initial pre-operative coordinate space, wherein: (a) Vector v1 is the virtual target vector based on the candidate virtual position under the first viewing angle position and the true target vector of the actual position as well as (b) Vector v2 is a real target vector based on the vector v1 and the actual position under the second perspective.

10. The method according to claim 9, wherein: is based on: sensing a first position of a fluorescent tool associated with the camera in a sensor space; calibrating the first position of the fluorescence tool to the fluorescence coordinate system; sensing, in the sensor space, a position of a patient tool secured to a patient table supporting the patient; calibrating the position of the patient table to the fluorescence coordinate system; and The initial pre-operative to real-time surgical transformation matrix.

11. The method of claim 10, wherein v2 is based on determining and The shortest distance between.

12. The method according to claim 11, wherein: Except for the C-arm position 2, In a similar manner to the above determination relevant means; and if and intersects, the intersection point is located and the vector v2 is calculated; and if and Do not intersect, apply a least squares fit to find the point where the two lines are closest.

13. The method according to any one of claims 8 to 12, further comprising calculating a rotational characteristic of the corrected position.

14. The method according to claim 13, wherein: The rotation feature is based on incrementally rotating the intraoperative image relative to the real-time image and about the center of the target until features present in the two images are aligned.

15. The method of claim 14, comprising updating the initial transformation matrix based on translation and rotation characteristics.

16. A method according to any one of the preceding claims, wherein: The camera tracking information includes measuring the position of the camera using an optical sensor.

17. The method of any of the preceding claims, further comprising calibrating the fluoroscopy camera coordinate system using optical sensor space.

18. A method according to any one of the preceding claims, wherein: The organ is the lung and the actual target is a lung nodule or tumor.

19. A method according to any one of the preceding claims, wherein: The virtual target is represented in the first image as a line or a group of lines forming a closed shape, which may optionally be a circle.

20. The method according to claim 2, wherein: The corrected position of the virtual object is a three-dimensional position.

21. The method according to claim 20, wherein: The first image and the second image are two-dimensional images.

22. A system for correcting the position of a virtual target, comprising a processor programmed and operable to: Segmentation of organs and ground truth objects from preoperative image datasets; receiving a real-time image dataset of a patient including the organ, the real object, and camera tracking information; registering the preoperative image dataset and the real-time image dataset according to an initial two-dimensional-three-dimensional transformation matrix; generating a first image at a first viewing angle based on the initial two-dimensional-three-dimensional transformation matrix, wherein the first image shows the real target at the actual position and the virtual target at the candidate position superimposed thereon; as well as Based on the user's adjustment of the candidate position of the virtual object in the first image, a corrected position of the virtual object is calculated.

23. The system of claim 22, wherein: The processor may also be operable to: generating a second image at a second viewing angle showing the real target and the virtual target; The corrected position of the virtual object is calculated based on an adjustment of the candidate position performed by the user on the first image and an adjustment of the candidate position performed by the user on the second image.

24. The system of claim 23, wherein: Both the first image and the second image are two-dimensional images.

25. The system of claim 24, wherein: The corrected position of the virtual target is a three-dimensional position.

26. The system of claim 25, wherein: The processor is further operable to update the initial two-dimensional to three-dimensional transformation matrix after calculating the corrected three-dimensional position of the virtual object.

27. The system according to any one of claims 22 to 26, wherein: The user adjustment includes a user marking a feature in or near the actual target in the first image and / or the second image.

28. The system of claim 22, wherein: The processor is programmed and operable to calculate a translational characteristic of the correction position.

29. The system of claim 28, wherein: A translation characteristic from the candidate position to the actual position is determined based on determining vectors v1 and v2 in an initial pre-operative coordinate space, wherein: (a) Vector v1 is a virtual target vector of the candidate virtual position under the first viewing angle. and the true target vector of the actual position as well as (b) Vector v2 is a real target vector based on the vector v1 and the actual position under the second perspective.

30. The system of claim 29, wherein: is based on: sensing a first position of a fluorescent tool associated with the camera in a sensor space; calibrating the first position of the fluorescence tool to the fluorescence coordinate system; sensing, in a sensor space, a position of a patient tool secured to a patient table supporting the patient; calibrating the position of the patient table to the fluorescence coordinate system; as well as The initial pre-operative to real-time surgical transformation matrix.

31. The system of claim 30, wherein: v2 is based on determining and The shortest distance between.

32. The system of claim 31, wherein: Except for the C-arm position 2, Similar to the above determination relevant means; and if and intersects, the intersection point is located and the vector v2 is calculated; and if and Do not intersect, apply a least squares fit to find the point where the two lines are closest.

33. The system according to any one of claims 28 to 32, wherein: The processor is also programmed and operable to calculate a rotational characteristic of the corrected position.

34. The system according to any one of claims 22-33, wherein: The processor is programmed and operable to receive the user adjustment via a mouse, keyboard, or touch screen.

35. A system for correcting the position of a virtual target as described herein.

36. A method for correcting the position of a virtual target as described herein.

37. A non-transitory program storage device readable by a processor and comprising instructions stored thereon to cause one or more processors to correct the position of a virtual target as described herein.

Images