Method and navigation system for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest - Patents.com

The method and navigation system address the inefficiencies and inaccuracies of conventional spinal navigation systems by using spatial parameter-based registration and virtual camera adjustments to achieve rapid and accurate 2D-3D image alignment, reducing radiation exposure and improving surgical navigation.

JP2024541438AActive Publication Date: 2024-11-08REMEX MEDICAL CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024529814
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-07-01
Filing Date
2022-11-18
Publication Date
2024-11-08
Estimated Expiration
2042-11-18

AI Technical Summary

Technical Problem

Conventional spinal navigation systems require X-ray detection, exposing patients to radiation risks and suffer from inefficient and inaccurate two-dimensional (2D) and three-dimensional (3D) image registration processes that often take several minutes to hours and result in registration failures up to 20% of the time.

Method used

A method and navigation system for registering 2D and 3D image datasets by reproducing 3D images based on spatial parameters, calculating similarity values, and adjusting virtual cameras to achieve accurate and efficient registration, allowing for realignment when failures occur.

Benefits of technology

The method and system enable rapid and accurate registration of 2D and 3D images, reducing radiation exposure by 98% and improving surgical navigation efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541438000001_ABST
    Figure 2024541438000001_ABST
Patent Text Reader

Abstract

A method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest is provided. [Solution] The method includes a step of defining a first vector of a registered virtual camera in the coordinate system of a three-dimensional image dataset; a step of transforming the first vector of the registered virtual camera by at least one transformation matrix to obtain a first transformation vector of an unregistered virtual camera in the coordinate system of the three-dimensional image dataset; a step of defining a focal point of the unregistered virtual camera in the coordinate system of the three-dimensional image dataset to be located at a reference point of the two-dimensional image dataset; and a step of generating an updated reconstructed image after repositioning the unregistered virtual camera by repositioning the unregistered virtual camera based on the first transformation vector and the focal point of the unregistered virtual camera.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This application was filed as a PCT International Patent Application on November 18, 2022, and claims priority to and the benefit of U.S. Provisional Application No. 63 / 264250, filed November 18, 2021, and U.S. Non-Provisional Application Nos. 17 / 856158, 17 / 856289, 17 / 856289, and 17 / 856301, filed July 1, 2022, the disclosures of which are incorporated herein in their entireties. The present application relates to a method and navigation system for establishing a relationship between two-dimensional image data and three-dimensional image data corresponding to a region of interest, and in particular to a method and navigation system for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest. [Background technology]

[0002] At present, the incidence of spine-related diseases is increasing day by day, and people's health is seriously threatened. Spinal surgery is the main way to treat spinal diseases, and how to reduce the surgical wound is more and more important, so that the probability of infection occurrence can be reduced, and patients can recover quickly without needing hospitalization. In order to achieve the above goal, a navigation system with an image-navigation surgical procedure has been introduced and used in spinal surgery. In the image-navigation surgical procedure, an image of the patient's region of interest is displayed on the display of the navigation system. At the same time, the navigation system can track surgical instruments and / or implants and import the simulated images thereof together with the image of the patient's region of interest. The advantages of the above image navigation surgical procedure and navigation system allow the surgeon to know the location of the surgical instruments and / or implants relative to the surgical target without the need for frequent use of C-arm fluoroscopy to know the relative positions throughout the surgical procedure, and the above technology is mostly disclosed in U.S. Patent No. "US6,470,207" entitled "Navigational Guidance Via Computer-Assisted Fluoroscopic Imaging" and published on October 22, 2002. The above listed patent documents are incorporated herein by reference in their entirety.

[0003] However, the above spinal navigation system usually needs to use X-ray detection to obtain the internal structure of the patient's body, so the patient is exposed to the risk of radiation. From this, it can be seen that the above conventional method has obvious inconveniences and defects and needs improvement. To solve the above problem, the related field has been struggling to find a solution, but no suitable solution has been developed for a long time.

[0004] There are many methods that can realize two-dimensional (2D) and three-dimensional (3D) registration, for example, using well-known two-dimensional and three-dimensional image registration methods. Two-dimensional and three-dimensional image registration methods include contour algorithms, point registration algorithms, surface registration algorithms, density comparison algorithms, and pattern intensity registration algorithms. Since the above registration methods all require a large amount of calculation, it usually takes several minutes to complete two-dimensional and three-dimensional registration. In fact, some registrations require 20 minutes to an hour. Furthermore, after a long processing time, the above registration procedures may result in inaccurate registration.

[0005] As described above, the above conventional methods have obvious inconveniences and deficiencies and need improvement, so it is necessary to develop a method and apparatus for completing 2D and 3D registration more accurately and efficiently, and the 2D and 3D registration flow should also be improved.

[0006] Also, two-dimensional images (e.g., X-ray images) usually cover multiple layers of vertebrae. Matching between two-dimensional images and digitally reconstructed radiographs (DRRs) generated by computed tomography (CT) data is not only inefficient, but also prone to registration failure, errors, and inaccuracies due to interference in the region of interest. The interference includes interference due to pedicle screws, vertebral cages, ribs, and hip bones. Specialized algorithms focus on comparison between multiple partitions rather than overall matching, thus contributing to comparative analysis on anatomical images. When performing two-dimensional and three-dimensional registration, the results of failed two-dimensional and three-dimensional registration still occur at a rate of more than 20%. Therefore, how to efficiently complete two-dimensional and three-dimensional registration using a part of the obtained results rather than rerunning the entire process is a goal that those skilled in the art are working on. Summary of the Invention [Problem to be solved by the invention]

[0007] The present disclosure is intended to provide a simplified summary of the present disclosure so that readers can have a basic understanding of the present disclosure. This disclosure is not an exhaustive overview of the present disclosure, and is not intended to identify key / core elements of the embodiments of the present invention or to define the scope of the present invention. [Means for solving the problem]

[0008] One technical aspect of the present application relates to a method for registering a two-dimensional image dataset of a region of interest and a three-dimensional image dataset of the region of interest, the method including the steps of: generating a first reconstructed image from the three-dimensional image dataset based on a first spatial parameter; calculating a reference similarity value based on the first reconstructed image and the two-dimensional image dataset; generating a second reconstructed image from the three-dimensional image dataset based on a second spatial parameter; calculating a comparison similarity value based on the second reconstructed image and the two-dimensional image dataset; comparing the comparison similarity value with the reference similarity value; and, if the comparison similarity value is less than or equal to the reference similarity value, registering the two-dimensional image dataset and the three-dimensional image dataset, and using the registered two-dimensional image dataset and the three-dimensional image dataset for computer-assisted surgical navigation.

[0009] Another technical aspect of the present application relates to a navigation system for registering a two-dimensional image dataset of a region of interest and a three-dimensional image dataset of the region of interest, the navigation system including: a memory for storing a plurality of instructions; and a processor for executing the following steps by retrieving the plurality of instructions by the memory: generating a first reconstructed image from the three-dimensional image dataset based on a first spatial parameter; calculating a reference similarity value based on the first reconstructed image and the two-dimensional image dataset; generating a second reconstructed image from the three-dimensional image dataset based on a second spatial parameter; calculating a comparison similarity value based on the second reconstructed image and the two-dimensional image dataset; comparing the comparison similarity value with the reference similarity value; and, if the comparison similarity value is less than or equal to the reference similarity value, registering the two-dimensional image dataset and the three-dimensional image dataset, and using the registered two-dimensional image dataset and the three-dimensional image dataset for navigation of a computer-assisted surgery.

[0010] Another technical aspect of the present application relates to a method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, the method comprising: adjusting a first virtual camera based on a distance parameter calculated based on the two-dimensional image dataset and the region of interest; rotating the first virtual camera based on an angle difference between a first vector calculated from two spatial marks in the three-dimensional image dataset and a second vector calculated from two first planar marks in the two-dimensional image dataset; rotating the first virtual camera based on an angle, the angle corresponding to an angle difference between a reconstructed image having a maximum similarity parameter and the two-dimensional image dataset, the reconstructed image with the maximum similarity parameter having a maximum similarity parameter among a plurality of similarity parameters calculated for a plurality of reconstructed images in the three-dimensional image dataset, the plurality of reconstructed images including a reconstructed image generated by the first virtual camera and another reconstructed image generated by another virtual camera, the other reconstructed image and the reconstructed image generated by the first virtual camera having different angles or pixels; and after the adjustment and rotation, using the method in a navigation system to register the two-dimensional image dataset and the three-dimensional image dataset.

[0011] Another technical aspect of the present application is a navigation system for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, the navigation system including a memory for storing a plurality of instructions, and a step of acquiring the plurality of instructions by the memory, the step of adjusting a first virtual camera based on a distance parameter calculated based on the two-dimensional image dataset and the region of interest, rotating the first virtual camera based on an angle difference between a first vector calculated from two spatial marks in the three-dimensional image dataset and a second vector calculated from two first planar marks in the two-dimensional image dataset, and rotating the first virtual camera based on an angle, the angle being greater than or equal to a maximum similarity parameter. and a processor for executing the method according to claim 1, wherein the reconstructed image has a maximum similarity parameter corresponding to an angular difference between a reconstructed image having the maximum similarity parameter and the two-dimensional image dataset, the reconstructed image having the maximum similarity parameter being one of a plurality of similarity parameters calculated for a plurality of reconstructed images in the three-dimensional image dataset, the plurality of reconstructed images including a reconstructed image generated by the first virtual camera and another reconstructed image generated by another virtual camera, the other reconstructed images and the reconstructed image generated by the first virtual camera having different angles or pixels, after rectification and rotation, and using the method in a navigation system to register the two-dimensional image dataset and the three-dimensional image dataset.

[0012] One technical aspect of the present application relates to a method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, the method including the steps of: defining a first vector of a registered virtual camera in a coordinate system of the three-dimensional image dataset; transforming the first vector of the registered virtual camera by at least one transformation matrix to obtain a first transformation vector of an unregistered virtual camera in the coordinate system of the three-dimensional image dataset; defining a focal point of the unregistered virtual camera to be located at a reference point of the two-dimensional image dataset in the coordinate system of the three-dimensional image dataset; and generating an updated reconstructed image after repositioning the unregistered virtual camera by repositioning the unregistered virtual camera based on the first transformation vector and the focal point of the unregistered virtual camera.

[0013] One technical aspect of the present application relates to a navigation system for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, the navigation system including: a memory for storing a plurality of instructions; and a processor for executing the following steps by retrieving the plurality of instructions by the memory: defining a first vector of a registered virtual camera in a coordinate system of the three-dimensional image dataset; transforming the first vector of the registered virtual camera by at least one transformation matrix to obtain a first transformation vector of an unregistered virtual camera in the coordinate system of the three-dimensional image dataset; defining a focal point of the unregistered virtual camera to be located at a reference point of the two-dimensional image dataset in the coordinate system of the three-dimensional image dataset; and generating an updated reconstructed image after repositioning the unregistered virtual camera by repositioning the unregistered virtual camera based on the first transformation vector and the focus of the unregistered virtual camera.

[0014] One technical aspect of the present application relates to a method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, the method including: acquiring first spatial parameters of a first registered virtual camera, where a set position of the first registered virtual camera corresponds to a first two-dimensional image of the two-dimensional image dataset; and adjusting second spatial parameters of a first unregistered virtual camera with the first spatial parameters of the first registered virtual camera to reposition the first unregistered virtual camera, thereby generating an updated reconstructed image, where a set position of the first unregistered virtual camera does not correspond to the first two-dimensional image of the two-dimensional image dataset.

[0015] One technical aspect of the present application relates to a navigation system for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, the navigation system including: a memory for storing a plurality of instructions; and a processor for executing the following steps: acquiring first spatial parameters of a first registered virtual camera by acquiring the plurality of instructions using the memory, wherein a set position of the first registered virtual camera corresponds to a first two-dimensional image of the two-dimensional image dataset; and adjusting second spatial parameters of a first unregistered virtual camera with the first spatial parameters of the first registered virtual camera to reposition the first unregistered virtual camera to generate an updated reconstructed image, wherein the set position of the first unregistered virtual camera does not correspond to the first two-dimensional image of the two-dimensional image dataset. Effect of the Invention

[0016] Therefore, based on the technical contents of the present application, the method and navigation system in the embodiments of the present application provide a re-registration procedure when the normal 2D and 3D registration procedure fails. Since the re-registration procedure is a continuation of the obtained result, the 2D and 3D registration procedure can continue to complete the registration procedure based on the existing result, so that the method and navigation system of the present application can complete the registration procedure more quickly than re-running the entire flow.

[0017] After referring to the following embodiments, those skilled in the art can easily understand the basic spirit and other inventive objects of the present application, as well as the technical means and embodiments adopted by the present application. [Brief description of the drawings]

[0018] To make the above and other objects, features, advantages and embodiments of the present application more clearly and comprehensibly, reference is made to the accompanying drawings as follows: [Figure 1] 1 is a schematic diagram of a navigation system according to an embodiment of the present application; [Diagram 2] FIG. 2 is a schematic diagram of a computing device, a database, and an optical tracking device of the navigation system shown in FIG. 1 according to one embodiment of the present application. [Diagram 3] 4 is a flow chart of a method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest according to an embodiment of the present application; [Figure 4] FIG. 2 is a schematic diagram of the contents of a portion of a region of interest according to an embodiment of the present application; [Diagram 5] FIG. 2 is a schematic diagram of the contents of a portion of a region of interest according to an embodiment of the present application; [Figure 6] FIG. 2 is a schematic diagram of the contents of a portion of a region of interest according to an embodiment of the present application; [Figure 7] FIG. 2 is a schematic diagram of the contents of a portion of a region of interest according to an embodiment of the present application; [Figure 8] FIG. 2 is a schematic diagram illustrating the operation of the C-arm device shown in FIG. 1 according to one embodiment of the present application. [Figure 9]FIG. 2 is a schematic diagram illustrating the operation of the C-arm device shown in FIG. 1 according to one embodiment of the present application. [Figure 10] 4 is a flow chart of a method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest according to an embodiment of the present application; [Figure 11] 4 is a flow chart of a method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest according to an embodiment of the present application; [Figure 12] 4 is a flow chart of a method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest according to an embodiment of the present application; [Figure 13] 4 is a flow chart of a method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest according to an embodiment of the present application; [Figure 14] FIG. 2 is a schematic diagram of a virtual camera simulated by a computing device of the navigation system shown in FIG. 1 according to one embodiment of the present application. [Figure 15] 4 is a flow chart of a method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest according to an embodiment of the present application; [Figure 16] 1 is a flow chart of a method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest according to an embodiment of the present application. In accordance with normal working practices, the various features and elements in the drawings are not drawn to scale, and the drawing manner is used to best present specific features and elements relevant to the present application. It should be noted that the same or similar element symbols refer to similar elements / components in different drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] In order to make the description of the present disclosure more detailed and complete, the following provides an illustrative description of the embodiments and specific examples of the present application. However, this is not the only way to implement or operate the specific examples of the present application. The embodiments include features of multiple specific examples and method steps and sequences for constructing and operating these specific examples. However, other specific examples can be used to achieve the same or equivalent functions and sequences of steps.

[0020] Unless otherwise defined herein, scientific and technical terms used herein have the same meaning as understood and commonly used by those skilled in the art. Also, unless conflicting with the context, singular nouns used herein cover the plural of that noun. The use of plural nouns also covers the singular of that noun.

[0021] 1 is a schematic diagram of a navigation system 1000 having a computing device 1100, a database 1200, and an optical tracking device 1400 according to an embodiment of the present application. The navigation system 1000 can be used to navigate an instrument 9500 during a surgical operation of a target 9000, but the present application is not limited thereto. In some embodiments, the navigation system 1000 can navigate any kind of instrument during any type of operation according to actual needs. For example, the navigation system 1000 can be used to navigate screws, graspers, clamps, surgical scissors, cutters, needle drivers, retractors, distractors, dilators, suction tips and tubes, sealing devices, irrigation and injection needles, powered devices, scopes and probes, carriers and appliers, ultrasonic tissue disruptors, cryotomes and cutting laser guides, and measurement devices. The navigation system 1000 can similarly be used to navigate any type of implant, including orthopedic implants, spinal implants, cardiovascular implants, neurovascular implants, soft tissue implants, or other implants that can be implanted within a target 9000 (e.g., a patient).

[0022] The computing device 1100 may be, but is not limited to, a computer, a desktop computer, a laptop computer, a tablet computer, or other devices capable of performing computing functions. The database 1200 may be, but is not limited to, a data storage device, a computer, a server, a cloud storage device, or other devices capable of storing data. The database 1200 is used to store a pre-acquired three-dimensional (3D) image dataset of the target 9000. The three-dimensional image dataset may be acquired by imaging the target 9000 using computed tomography (CT).

[0023] The imaging device 1300 may be established and executed by the navigation system 1000, and may be, but is not limited to, a C-arm mobile fluoroscopy machine or a mobile X-ray image intensifier. The imaging device 1300 includes an X-ray emitter 1310 and an X-ray receiver 1320. The former emits X-rays to pass through the torso of the target 9000, and the latter receives the X-rays and converts them into a digital two-dimensional (2D) image displayed on the display of the imaging device 1300. Based on the plane along which the image is acquired along the imaging device 1300, the two-dimensional image may be an anterior / posterior (AP) two-dimensional image and a lateral view (LA) two-dimensional image of the target 9000. The target 9000 may be, but is not limited to, a patient.

[0024] During the image acquisition process, the calibrator 1510 having the calibration marks is used to calibrate the imaging device 1300 and track the position of the imaging device 1300. The calibrator 1510 is located in or on the X-ray receiving device 1320, and dynamic reference frames 1520A, 1520B are placed at or adjacent to the regions of interest of the X-ray receiving device 1320 and the target 9000. The calibrator 1510 is located on the path from the X-ray emitting device 1310 to the X-ray receiving device 1320, and is not transparent or semi-transparent to X-rays. Therefore, after the X-rays enter the target 9000 and the calibrator 1510, they are partially absorbed by certain tissues and the calibration marks in the calibrator 1510 before being received by the X-ray receiving device 1320. In this way, the two-dimensional image can display the region of interest of the target 9000 and the calibration marks of the calibrator 1510. The computing device 1100 calculates the relationship between the X-ray emitting device 1310 and the X-ray receiving device 1320 based on the pattern of the calibration marks displayed in the two-dimensional image. Both the AP two-dimensional image and the LA two-dimensional image of the vertebrae of interest of the target 9000 are included in the two-dimensional image data set shown in the embodiment of the present application. However, in other embodiments, only the AP two-dimensional image or the LA two-dimensional image may be included in the two-dimensional image data set. Also, the two-dimensional image data set can be transferred to the computing device 1100 as an electronic file for subsequent use.

[0025] The three-dimensional image data set in the database 1200, the two-dimensional image data set generated by the imaging device 1300, the optical tracking device 1400, the calibrator 1510, the dynamic reference coordinates 1520A, 1520B, and the target 9000 each have coordinates in a different coordinate system. The navigation system 1000 of the present application can establish a relationship between these coordinate systems so that the surgeon can navigate the instrument 9500 using the navigation system 1000 during a surgical operation. The relationship between the above coordinates will be described later. The calibrator 1510 and the dynamic reference coordinates 1520A are installed on the X-ray receiving device 1320, and the positions of the three are considered to be substantially the same, so the relationship between the calibrator coordinate system and the imaging device coordinate system is established. The optical tracker 1400 then tracks the dynamic reference coordinate 1520A including the reflector to introduce the calibrator 1510, the dynamic reference coordinate 1520A, and the X-ray receiving device 1320 into the tracker coordinate system, so that the position of the X-ray receiving device 1320 in the tracker coordinate system can be obtained. In some embodiments, the optical tracker 1400 can track the calibrator 1530 including the reflector to introduce the calibrator 1530 into the tracker coordinate system, so that the position of the calibrator 1530 in the tracker coordinate system can be obtained. The relationship between the X-ray emitting device 1310 and the X-ray receiving device 1320 can be obtained as described above, and the position of the X-ray emitting device 1310 in the tracker coordinate system is obtained. It should be noted that the present application is not limited to the embodiment shown in FIG. 1, which is only used to illustratively illustrate one of the implementation forms of the present application.

[0026] It should be noted that the registration procedure of the navigation system 1000 is roughly composed of three procedures: an initialization calibration procedure, a two-dimensional and three-dimensional registration procedure, and a re-calibration procedure. In some embodiments, details for the program installed in the navigation system 1000 to execute the above three procedures will be described later. It should be noted that in other embodiments, the program can execute the initialization calibration procedure, the two-dimensional and three-dimensional registration procedure, and the re-registration procedure alone or in combination with the above procedures according to the current situation or the required functions. Other details will be described later.

[0027] Initialize Alignment

[0028] Prior to using the navigation system 1000, an initialization alignment procedure is performed on the navigation system 1000 to further improve the efficiency and accuracy of the navigation system 1000, as will be described in further detail below.

[0029] 2 is a schematic diagram of the computing device 1100, database 1200, and optical tracking device 1400 of the navigation system 1000 shown in FIG 1 according to one embodiment of the present application. As shown, the computing device 1100 is electrically connected to the database 1200 and the imaging device 1300.

[0030] The computing device 1100 includes a memory 1110, a processor 1120, and an I / O interface 1130. The processor 1120 is electrically connected to the memory 1110 and the I / O interface 1130. The memory 1110 is used to store instructions, and the processor 1120 performs steps of a method 2000 for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest shown in FIG.

[0031] In some embodiments, the memory 1110 may include, but is not limited to, at least one of a flash memory, a hard disk drive (HDD), a solid-state drive (SSD), a dynamic random access memory (DRAM), and a static random access memory (SRAM), or a combination of the above elements. In some embodiments, as a non-transitory computer-readable storage medium, the memory 1110 may be used to store computer-readable instructions executable by the processor 1120.

[0032] In some embodiments, the processor 1120 may include a single processor or a multi-microprocessor integrated device, such as, but not limited to, a central processing unit (CPU), a graphics processing unit (GPU), or an application-specific integrated circuit (ASIC).

[0033] See also Figures 2 and 3. In operation, the processor 1120 executes step 2100 to adjust a first virtual camera based on the two-dimensional image data set and distance parameters calculated corresponding to the area of ​​interest.

[0034] For example, the processor 1120 of the computing device 1100 may directly acquire an X-ray image by the imaging device 1300, or may acquire an X-ray image previously stored in the database 1200. As described above, the two-dimensional data set includes at least an AP two-dimensional image and an LA two-dimensional image, both of which include the calibrator 1510 of FIG. 1. The calibrator 1510 can be used to identify the position of the imaging device 1300 in the above-mentioned calculation relationship. Upon acquiring the position of the imaging device 1300, the processor 1120 can estimate the distance between the X-ray emitter 1310 and the site of interest of the target vertebral segment in the target 9000. The above-mentioned estimated distance is then converted into a distance parameter by the processor 1120 of the computing device 1100.

[0035] As described above, the embodiment of the present application has been outlined with respect to the initialization alignment 2000. In brief, the initialization alignment 2000 may include three steps 2100, 2200, and 2300.

[0036] In a first step, the surgeon acquires a first or AP image and a second or LA image by the imaging device 1300. These images can be used to help refine and navigate the system, as well as to initialize the orientation. The imaging device 1300 includes a calibrator 1510, so that the position of the X-ray emitter 1310 (e.g., the patient's AP direction and the patient's LA direction) is known during the image acquisition procedure relative to the patient coordinate system. This information is combined with information on how to position the patient 9000 during the three-dimensional acquisition set by the user. After the set orientation is given, a digitally reconstructed radiographic image (DRR) can be created with the three-dimensional data set acquired by computed tomography (CT). The digitally reconstructed radiographic image (DRR) is also a digital image, so it can be used to interpret how and where the virtual camera generates these digital images. In other words, the AP and LA digitally reconstructed radiographs (DRR) are expected to be generated by simulating two virtual cameras taking images, for example, placing the two virtual cameras at specific points along the two AP and LA planes of the patient 9000, respectively, in the direction toward the vertebrae of interest of the patient 9000. The digitally reconstructed radiographs (DRR) correspond to the actual AP and LA radiographs. The digitally reconstructed radiographs (DRR) and the actual radiographs are displayed to the surgeon, who identifies the same points in all images. In this step, the same points or positions in the CT image data and the dynamic reference coordinates 1520A or patient space coordinates are provided. Once the position and orientation of the patient 9000 in the three-dimensional data set and the dynamic reference coordinates are known, the method of the present application continues to perform the subsequent refinement steps. After the refinement step, all related coordinate systems can be connected and known by calculation.

[0037] In the initialization step, the surgeon is required to obtain AP and LA x-rays or x-rays along any two planes with the imaging device 1300. In spine surgery, the surgeon is instructed to point out the center of the vertebral body of interest. Combining all the above information, a good orientation and position assessment is obtained, and this assessment is used to generate a digitally reconstructed radiograph (DRR). Such a digitally reconstructed radiograph (DRR) has a high degree of correspondence with the actual x-ray and may be used to trigger the 2D-3D registration procedure. The digitally reconstructed radiograph (DRR) was calculated by combining 3D data acquired by computed tomography (CT) and data acquired by a local target acquired by a C-arm.

[0038] The processor 1120 of the computing device 1100 can access the three-dimensional image dataset stored in the database 1200, and an AP virtual camera or an LA virtual camera corresponding to the three-dimensional image dataset can be created by the processor 1120. Each virtual camera has its own spatial parameters to mark the position and orientation of the site of interest in its corresponding three-dimensional image. As with a real camera, the position and orientation of the camera relative to the target object determines the image content of the capture. Thus, changing the spatial parameters of the virtual camera generates different reconstructed images in the three-dimensional image dataset. In this embodiment, the spatial parameters include at least one distance parameter indicating the distance from the target site of interest of the target 9000 to the virtual camera.

[0039] Initial registration

[0040] In the initial registration process, the software includes a matching algorithm for reprocessing the initial 2D and 3D registration. The software design is used to minimize the difference between the digitally reconstructed radiograph (DRR) and the actual radiograph by adjusting the initial position and orientation, and improve the whole registration flow. In the initial registration process, a similarity or cost measurement method can be adopted to identify the degree of image matching. The inverted initial algorithm maximizes the similarity between the digitally reconstructed radiograph (DRR) and the actual radiograph by simultaneously adjusting the initial position and orientation parameters. The calculation of the similarity or cost can be selected from known similarity and cost calculation methods, such as normalized mutual data, mutual data, gradient difference algorithms, surface contour algorithms, pattern intensity algorithms, sum of squared differences, normalized cross-correlation, local normalized correlation, and correlation ratio. By performing this procedure, the entire 2D and 3D registration can be made efficient and accurate.

[0041] The processor 1120 of the computing device 1100 can then adjust the AP virtual camera or the LA virtual camera based on the distance parameter.

[0042] In some embodiments, the distance parameter is calculated based on the actual distance between two planar marks in an image of the two-dimensional image dataset, for example, a surgeon can calculate the distance parameter by manually marking two markers on a single AP or LA two-dimensional image of interest.

[0043] In the embodiment of Fig. 4, the target 9000 may be a patient with spinal cord injury, and a surgeon may manually mark two marks V0 and V1 at the center of two target vertebrae of the target 9000 through the I / O interface 1130 of Fig. 2, but the present application is not limited thereto. In this manner, the distance between the marks V0 and V1 may be calculated by the processor 1120 of the computing device 1100 through image recognition techniques. In some embodiments, the I / O interface 1130 may be electrically connected to any kind of input device, such as a mouse, a keyboard, or a touch panel.

[0044] 5, this is an AP Digital Reconstructed Radiographic Image (DRR) generated by the processor 1120 using input data. The input data includes position or orientation information of the imaging device 1300 relative to the target 9000 when taking the X-ray image.

[0045] In some embodiments, the virtual camera is a modular coding in a 2D and 3D registration program that is used to parameter-driven position and parameter-driven orientation or perspective to compute known digital reconstruction radiography (DRR) algorithms by the processor 1120 to generate that 2D image for a particular 3D volume.

[0046] Using this orientation estimation method, a digitally reconstructed radiograph (DRR) generated based on the computed tomography (CT) is established (FIG. 5), which substantially corresponds to the actual AP radiograph of FIG. 4. By using the patient orientation information from the three-dimensional image data and the patient orientation information at the position of the image device 1300, the above information is combined with a well-known digitally reconstructed radiograph (DRR) algorithm to generate the digitally reconstructed radiograph (DRR). In other words, the content of the radiograph is known, and the left and right and AP directions or any other direction of the computed tomography (CT) data are both known, so that when looking into the three-dimensional volume data in the direction of the radiograph, an accurate digitally reconstructed radiograph (DRR) can be obtained. The digitally reconstructed radiograph (DRR) is essentially a two-dimensional image generated for a three-dimensional volume, whose content is the image result displayed by looking inside the three-dimensional volume at a correct orientation or angle, so that a virtual line can be created for each volume pixel, and each line is used to generate a new volume pixel value to generate the digitally reconstructed radiograph (DRR).

[0047] However, initial digitally reconstructed radiographs (DRR) do not always match the AP and LA x-ray images exactly, which is why a registration procedure is performed in the 2D and 3D registration procedures.

[0048] The surgeon can manually mark the other two marks V0' and V1' on the two target vertebral bodies of the AP digitally reconstructed radiographic image (DRR) through the I / O interface 1130 of FIG. 2. In this way, the distance between the two marks V0' and V1' of the AP digitally reconstructed radiographic image (DRR) of FIG. 5 can be calculated by the processor 1120. Then, if the distance between the two marks V0 and V1 of the AP two-dimensional image of FIG. 4 is 100 pixels, the processor 1120 can adjust the distance parameter of the AP virtual camera to regenerate an adjusted AP digitally reconstructed radiographic image (DRR), in which the distance between the two marks V0' and V1' is also 100 pixels. Therefore, the size of the target vertebral body in the AP digitally reconstructed radiographic image (DRR) of FIG. 5 is adjusted to be similar to the size of the target vertebral body in the AP two-dimensional image of FIG. 4. As described above, in the present application, the alignment procedure can be performed very quickly since it is only necessary to focus on the corresponding distances of the marks in the AP two-dimensional image of FIG. 4 and the AP digitally reconstructed radiographic image (DRR) of FIG. 5, and then adjust the distances to match.

[0049] It should be noted that in the embodiment of Fig. 6, the surgeon can manually mark marks V0 and V1 on two target vertebrae of the target 9000 through the I / O interface 1130 of Fig. 2. In this manner, the distance between marks V0 and V1 can be calculated by the processor 1120 of the computing device 1100 through image recognition techniques.

[0050] As shown in FIG. 7, this is an LA digitally reconstructed radiographic image (DRR) generated by the processor 1120, which is generated in response to the information of the position or defined position of the image device 1300 relative to the target 9000 in the X-ray imaging procedure. The surgeon can manually mark two marks V0' and V1' on the two target vertebrae of the target through the I / O interface 1130 of FIG. 2. In this way, the distance between the marks V0' and V1' can be calculated by the processor 1120 of the computing device 1100 through image recognition technology. After that, if the distance between the two marks V0 and V1 in the LA two-dimensional image of FIG. 6 is 100 pixels, the processor 1120 can adjust the distance parameter of the LA virtual camera to regenerate an adjusted LA digitally reconstructed radiographic image (DRR), in which the distance between the two marks V0' and V1' is also 100 pixels. Therefore, the size of the target vertebrae in FIG. 7 is adjusted to be similar to the size of the target vertebrae in the LA two-dimensional image of FIG. 6. As described above, the present application focuses only on the correlation distance of the marks in the LA two-dimensional image of FIG. 6 and the LA digitally reconstructed radiographic image (DRR) of FIG. 7, and then adjusts the distances to match to perform alignment, so that the present alignment procedure can be performed very quickly.

[0051] In some embodiments, the distance parameter is calculated based on the distance between the estimated position of the emitter and the estimated position of the region of interest. For example, as described above, the computing device 1100 can calculate the estimated position of the X-ray emitter 1310 by using the calibrator 1510 in the X-ray two-dimensional image generated by the imaging device 1300 of FIG. 1 as a reference. Note that the estimated position of the target 9000 can be calculated by the computing device 1100 based on the displayed X-ray image of the calibrator 1510 and the estimated position of the X-ray emitter 1310. After that, by obtaining the estimated position of the X-ray emitter 1310 and the estimated position of the target 9000, the distance between the estimated position of the X-ray emitter 1310 and the estimated position of the target 9000 can be calculated by the computing device 1100 as well.

[0052] In some embodiments, the estimated position of the emitter can be calculated based on the two-dimensional image data set, and the estimated position of the region of interest is located at the closest position of the first virtual line and the second virtual line. For example, the computing device 1100 in FIG. 2 can obtain an X-ray image by the imaging device 1300. As described above, the X-ray image of the imaging device 1300 may include the calibrator 1510 in FIG. 1. Therefore, the computing device 1100 can calculate the estimated position of the X-ray emitter 1310 based on the X-ray image including the calibrator 1510. See FIG. 8. In the figure, a first virtual line VL1 is generated between the estimated position of the X-ray emitter 1310 and the calibrator 1510 in the X-ray receiving device 1320. See FIG. 9. In the figure, a second virtual line VL2 is generated between the estimated position of the X-ray emitter 1310 and the calibrator 1510 in the X-ray receiving device 1320. When the first virtual line VL1 in Figure 8 and the second virtual line VL2 in Figure 9 are located in the same coordinate system, the estimated position of the target 9000 can be determined to be located at the position closest to the first virtual line VL1 and the second virtual line VL2.

[0053] In some embodiments, the two-dimensional image dataset includes a first two-dimensional image and a second two-dimensional image, and a first virtual line is generated between an estimated position of the launcher and a reflective center point of the at least two reflectors when the first two-dimensional image dataset is captured, and a second virtual line is generated between the estimated position of the launcher and a reflective center point of the at least two reflectors when the second two-dimensional image dataset is captured.

[0054] For example, the X-ray image of the image device 1300 in FIG. 1 includes an AP two-dimensional image and an LA two-dimensional image of the target 9000. See FIG. 8. A first virtual line VL1 is generated between the estimated position of the X-ray emitter 1310 and the center point 1521 of the reflectors 1523, 1525, 1527, which corresponds to the dynamic reference coordinate 1520A in FIG. 1 and can be acquired by the optical tracking device 1400. See FIG. 9. A second virtual line VL2 is generated between the estimated position of the X-ray emitter 1310 and the center point 1521 of the reflectors 1523, 1525, 1527, which corresponds to the dynamic reference coordinate 1520A in FIG. 1 and can be acquired by the optical tracking device 1400. It should be clarified that the center point 1521 may be the origin of the calibrator coordinates established by the reflectors 1523 , 1525 , 1527 , but the location of said center point 1521 is not limited to the physical center of the reflectors 1523 , 1525 , 1527 .

[0055] See Figures 2 and 3. In operation, the processor 1120 executes step 2200 to rotate a first virtual camera based on an angular difference between a first vector and a second vector, the first vector being calculated from two spatial marks in the three-dimensional image dataset, and the second vector being calculated from two first planar marks in the two-dimensional image dataset.

[0056] For example, the first vector is calculated by the AP virtual camera from two spatial marks in the three-dimensional image based on a suitable DDR algorithm, see FIG. 4. The second vector is calculated by two planar marks V0 and V1 in the AP two-dimensional image. After obtaining the first and second vectors, the processor 1120 can calculate the angle difference between the first and second vectors. Then, the processor 1120 can rotate the AP virtual camera based on the angle difference. Therefore, the orientation or perspective orientation of the AP virtual camera is adjusted to be similar or similar to the direction or perspective orientation of the X-ray emitter 1310 of the image device 1300 along the AP plane in FIG. 4. The above steps can be performed by the computing device 1100 to adjust or modify the orientation or perspective orientation parameters of the digitally reconstructed radiographic image (DRR) algorithm, and generate an adjusted or modified digitally reconstructed radiographic image (DRR) that is more similar or similar to the AP two-dimensional X-ray image. As described above, the vertebral bodies in the AP two-dimensional image and the AP digitally reconstructed radiograph (DRR) can be adjusted similarly, and the frontal viewing angles associated with the LA two-dimensional image and the LA digitally reconstructed radiograph (DRR) can be adjusted to be similar.

[0057] Also, the first vector is calculated and generated by the A virtual camera from two spatial marks in the three-dimensional image based on a suitable DDR algorithm. See FIG. 6. The second vector is calculated by two planar marks V0 and V1 in the LA two-dimensional image. After obtaining the first and second vectors, the processor 1120 can calculate the angle difference between the first and second vectors. Then, the processor 1120 can rotate the LA virtual camera based on the angle difference. Therefore, the orientation or perspective orientation of the LA virtual camera is adjusted to be similar or similar to the orientation or perspective orientation of the X-ray emitter 1310 of the image device 1300 along the LA plane in FIG. 6. The above steps can be performed by the computing device 1100 to adjust or modify the orientation or perspective orientation parameters of the digitally reconstructed radiographic image (DRR) algorithm, and generate an adjusted or modified digitally reconstructed radiographic image (DRR) that is more similar or similar to the two-dimensional X-ray image. As described above, the vertebral bodies in the LA two-dimensional image and the LA digitally reconstructed radiograph (DRR) can be adjusted similarly, and the frontal viewing angles associated with the AP two-dimensional image and the AP digitally reconstructed radiograph (DRR) can be adjusted to be similar.

[0058] In some embodiments, the two-dimensional image dataset includes a first two-dimensional image, the first vector is calculated from two spatial marks in the three-dimensional image dataset by a first virtual camera, and the second vector is calculated from two first planar marks in the first two-dimensional image data. For example, the two-dimensional image dataset includes the AP two-dimensional image in FIG. 4. The first vector is calculated from two spatial marks in the three-dimensional image dataset by an AP virtual camera. The second vector is calculated from two planar marks V0 and V1 of the AP two-dimensional image in FIG. 4.

[0059] In some embodiments, the two-dimensional image dataset includes a second two-dimensional image, and the second vector is calculated from two second planar marks in the second two-dimensional image data. For example, the two-dimensional image dataset includes the LA two-dimensional image in FIG. 6. The first vector is generated by calculating from two spatial marks in the three-dimensional image dataset by the LA virtual camera. The second vector is calculated from two planar marks V0 and V1 of the LA two-dimensional image in FIG. 6.

[0060] See also Figures 2 and 3. In operation, the processor 1120 executes step 2300 to rotate a first virtual camera based on an angle, the angle corresponding to an angle difference between a reconstructed image having a maximum similarity parameter and the two-dimensional image dataset, the reconstructed image with the maximum similarity parameter having a maximum similarity parameter among a plurality of similarity parameters calculated for a plurality of reconstructed images in the three-dimensional image dataset, the plurality of reconstructed images including a reconstructed image generated by the first virtual camera and another reconstructed image generated by another virtual camera, and the other reconstructed image and the reconstructed image generated by the first virtual camera have different angles or pixels, and after the alignment and rotation, performs a registration operation of the two-dimensional image dataset and the three-dimensional image dataset of the navigation system.

[0061] For example, first, several LA digitally reconstructed radiographic images (DRRs) are generated in the coordinate system of the three-dimensional image dataset by adjusting a preset position and / or orientation of the LA virtual camera, which may be determined by a surgeon based on a corresponding volume of interest in the three-dimensional dataset.

[0062] The preset position and / or orientation of the LA virtual camera is set as the center, and then the LA virtual camera is rotated by a roll angle range of -20 degrees to 20 degrees and / or moved by a position range of -15 pixels to 15 pixels to generate a plurality of digitally reconstructed radiographic images (DRR) at different positions, respectively. Since different LA digitally reconstructed radiographic images (DRR) have different contents, a plurality of similarity values ​​can be calculated based on the plurality of LA digitally reconstructed radiographic images (DRR) and the preset LA two-dimensional image, and the preset LA two-dimensional image can be selected by the surgeon based on the region of interest of the corresponding volume of interest. Then, a maximum similarity value can be obtained from the plurality of similarity values. In order to achieve a good registration between the two-dimensional image and the digitally reconstructed radiographic image (DRR), the current position and / or orientation of the LA virtual camera is adjusted based on the pixel and the tilt angle corresponding to the maximum similarity value. In other words, the vertebrae in the LA digitally reconstructed radiographic image (DRR) are adjusted to be similar to the vertebrae in the LA two-dimensional image. The AP virtual camera for each vertebral body of the target 9000 can be adjusted based on the LA virtual camera that corresponds to the vertebral body of the target 9000.

[0063] In addition, in the coordinate system of the three-dimensional image data set, several AP digitally reconstructed radiographic images (DRR) are generated by a preset position and / or orientation of the AP virtual camera, which may be determined by the surgeon based on a corresponding volume of interest in the three-dimensional data set.

[0064] The preset position and / or orientation of the AP virtual camera is set as the center, and then rotated by an azimuth angle range of -15 degrees to 15 degrees and / or moved by a position range of -15 pixels to 15 pixels to generate a plurality of digitally reconstructed radiographic images (DRR) at different positions. Since different AP digitally reconstructed radiographic images (DRR) have different contents, a plurality of similarity values ​​can be calculated based on the plurality of AP digitally reconstructed radiographic images (DRR) and the preset AP two-dimensional image, and the preset AP two-dimensional image can be selected by the surgeon based on the region of interest of the corresponding volume of interest. Then, a maximum similarity value can be obtained from the plurality of similarity values. In order to achieve a good registration between the two-dimensional image and the digitally reconstructed radiographic image (DRR), the current position and / or orientation of the AP virtual camera is adjusted based on the pixel and the azimuth angle corresponding to the maximum similarity value. In other words, the vertebrae in the AP digitally reconstructed radiographic image (DRR) are adjusted to be similar to the vertebrae in the AP two-dimensional image. The LA virtual camera for each vertebral body of the target 9000 can be adjusted based on the AP virtual camera corresponding to the vertebral body of the target 9000.

[0065] In some embodiments, the adjusted first virtual camera is generated after rotating the first virtual camera, and the processor 1120 is adapted to perform a step of rotating the adjusted first virtual camera based on an angle, the angle corresponding to an angle difference between the reconstructed image having the adjusted maximum similarity parameter and the two-dimensional image dataset, the reconstructed image having the adjusted maximum similarity parameter is the one having a maximum similarity parameter among the plurality of similarity parameters calculated for the adjusted plurality of reconstructed images, the adjusted plurality of reconstructed images include a reconstructed image generated by the adjusted first virtual camera and other reconstructed images generated by other virtual cameras, and the other reconstructed images have different angles from the reconstructed image generated by the adjusted first virtual camera.

[0066] For example, the adjusted AP virtual camera or the adjusted LA virtual camera is generated after rotating the AP virtual camera or the LA virtual camera. The adjusted LA virtual camera is generated by moving the roll angles range of -20 degrees to 20 degrees and / or the position range of -15 pixels to 15 pixels, and a plurality of digitally reconstructed radiographic images (DRR) are generated at different positions. Since different LA digitally reconstructed radiographic images (DRR) have different contents, a plurality of similarity values ​​can be calculated based on the plurality of adjusted LA digitally reconstructed radiographic images (DRR) and a preset LA two-dimensional image, and the preset LA two-dimensional image can be selected by the surgeon based on the region of interest of the corresponding volume of interest. Then, a maximum similarity value can be obtained from the plurality of similarity values. In order to achieve a good alignment between the two-dimensional image and the digitally reconstructed radiographic image (DRR), the current position and / or orientation of the LA virtual camera is adjusted based on the maximum similarity value corresponding to the pixel and the roll angle. In other words, the vertebrae in the LA digitally reconstructed radiographic image (DRR) are adjusted to be similar to the vertebrae in the LA two-dimensional image.

[0067] In addition, the adjusted AP virtual camera is moved by an azimuth angle range of -15 degrees to 15 degrees and / or a position range of -15 pixels to 15 pixels to generate multiple digitally reconstructed radiographs (DRR) at different positions. Since different AP digitally reconstructed radiographs (DRR) have different contents, multiple similarity values ​​can be calculated based on the multiple adjusted AP digitally reconstructed radiographs (DRR) and the AP two-dimensional image, and the AP two-dimensional image calculation can be selected by the surgeon based on the region of interest of the corresponding volume of interest. Then, a maximum similarity value can be obtained from the multiple similarity values. In order to achieve good alignment between the two-dimensional image and the digitally reconstructed radiograph (DRR), the current position and / or orientation of the AP virtual camera is adjusted based on the pixel and the tilt angle corresponding to the maximum similarity value. In other words, the vertebrae in the AP digitally reconstructed radiograph (DRR) are adjusted to be similar to the vertebrae in the AP two-dimensional image.

[0068] It should be clarified that the present application is not limited to the structure and operation of the embodiment shown in FIGS. 2 to 9, which are only used to exemplify one of the implementation forms of the present application.

[0069] In some embodiments, the processor 1120 is used to perform a step of adjusting a first virtual camera of the plurality of virtual cameras based on a matrix corresponding to the two-dimensional image dataset, and the plurality of virtual cameras correspond to the three-dimensional image dataset. For example, both the AP two-dimensional image and the LA two-dimensional image of the image device 1300 in FIG. 1 include the calibrator 1510. The processor 1120 can calculate a matrix of a coordinate system for transforming the coordinate system of the AP two-dimensional image to the LA two-dimensional image according to the known parameters of the calibrator 1510. Thus, a matrix corresponding to the relationship between the AP two-dimensional image and the LA two-dimensional image can be obtained. Then, the processor 1120 can adjust the AP virtual camera or the LA virtual camera based on the matrix, so that the relationship between the AP virtual camera and the LA virtual camera is similar to the relationship between the AP two-dimensional image and the LA two-dimensional image.

[0070] 2D and 3D Registration

[0071] Once performed as the initial registration in the first procedure, the navigation system 1000 continues to establish a relationship between the three-dimensional image dataset and the X-ray two-dimensional image of the two-dimensional image dataset, the three-dimensional image dataset being stored in the database 1200, and the X-ray two-dimensional image of the two-dimensional image dataset being displayed on the display of the imaging device 1300 or stored in the database 1200. The above relationship establishment method will be described later.

[0072] Figure 10 is a flow chart of a method for registering at least one two-dimensional image of a two-dimensional image dataset of a region of interest and a three-dimensional image dataset of the same region of interest according to one embodiment of the present application, see also Figures 2 and 10. In operation, the processor 1120 executes step 3100 to generate a first reconstructed image from the three-dimensional image dataset based on the first spatial parameters.

[0073] For example, the processor 1120 of the computing device 1100 can access the database 1200 and simulate a virtual camera corresponding to the 3D image dataset based on the first spatial parameters, and generate a first digitally reconstructed radiographic image (DRR). In this embodiment, the virtual camera is pre-registered (to facilitate 2D and 3D registration by initial registration and / or other adjustment schemes), but in other embodiments of the present application, the virtual camera may not be pre-registered by initial registration and / or other adjustment schemes.

[0074] See also Figures 2 and 10. In operation, the processor 1120 executes step 3200 to calculate a criterion similarity measure based on the first reconstructed image and at least one two-dimensional image data set. For example, the processor 1120 of the computing device 1100 may calculate a criterion similarity measure based on the first digitally reconstructed radiographic image (DRR) and the two-dimensional image data.

[0075] See also Figures 2 and 10. In operation, the processor 1120 executes step 3300 to generate a second reconstructed image from the three-dimensional image dataset based on the second spatial parameters. For example, the processor 1120 of the computing device 1100 can access the database 1200 and simulate a virtual camera corresponding to the three-dimensional image dataset based on the second spatial parameters, and further generate a second Digitally Reconstructed Radiographic Image (DRR).

[0076] See also Figures 2 and 10. In operation, the processor 1120 executes step 3400 to calculate a comparative similarity value based on the second reconstructed image and the at least one two-dimensional image data set. For example, the processor 1120 of the computing device 1100 may calculate a comparative similarity value based on the second digitally reconstructed radiographic image (DRR) and the two-dimensional image data.

[0077] See also Figures 2 and 10. In operation, the processor 1120 performs step 3500 to compare the comparison similarity value to a reference similarity value. For example, the processor 1120 of the computing device 1100 may compare the comparison similarity value to a reference similarity value.

[0078] Please refer to FIG. 2 and FIG. 10 together. In operation, the processor 1120 executes step 3600 to register the two-dimensional image dataset and the three-dimensional image dataset when the comparison similarity value is equal to or less than the reference similarity value, and utilizes the registered two-dimensional image dataset and the three-dimensional image dataset for computer-assisted surgical navigation. For example, when the comparison similarity value is equal to or less than the reference similarity value, it indicates that the registration between the second digitally reconstructed radiograph (DRR) and the two-dimensional image data is completed. Therefore, the processor 1120 of the computing device 1100 can register the two-dimensional image dataset of the imaging device 1300 and the three-dimensional image dataset stored in the database 1200. Thereafter, since the two-dimensional image dataset of the imaging device 1300 and the three-dimensional image dataset stored in the database 1200 have been registered, the method 3000 in FIG. 10 of the present application can utilize the registered two-dimensional image dataset and the three-dimensional image dataset for computer-assisted surgical navigation.

[0079] It should be noted that the present application is not limited to the operation of the embodiment shown in Fig. 10, which is only used to exemplarily illustrate one of the implementation forms of the present application. It should be noted that the method 3000 in Fig. 10 can be executed by the processor 1120 to register a two-dimensional image dataset and a three-dimensional image dataset, where the two-dimensional image dataset includes an AP two-dimensional image and an LA two-dimensional image, and the AP digitally reconstructed radiographic image (DRR) and the LA digitally reconstructed radiographic image (DRR) are from the three-dimensional image dataset.

[0080] In some embodiments, generating the first or second reconstructed image from the three-dimensional image dataset based on the first or second reconstructed image includes positioning a virtual camera based on a three-dimensional target, the three-dimensional target being constructed based on the first or second reconstructed image, and capturing the first or second reconstructed image of the three-dimensional target by the virtual camera.

[0081] For example, the three-dimensional image data set may be volume pixel values ​​acquired after computed tomography (CT) of the target 9000, which includes the region of interest. The volume pixel values ​​may be represented by volume rendering, after which the three-dimensional target (also called a three-dimensional model) of the region of interest may be simultaneously displayed on a display and stored in the database 1200. The processor 1120 may access the three-dimensional model in the database 1200, and based on the spatial parameters of the given virtual camera, determine the position and orientation of the virtual camera of the corresponding three-dimensional model. Once the virtual camera is set, the processor 1100 may acquire one or more two-dimensional images as a first digitally reconstructed radiographic image (DRR) or a second digitally reconstructed radiographic image (DRR) according to a digitally reconstructed radiographic image (DRR) algorithm in the art.

[0082] In some embodiments, the virtual camera is a modularized functional instruction and includes one of an algorithm and an equation. In some embodiments, the virtual camera can be simulated but is not limited to radiographic volume rendering.

[0083] In the initial alignment, the virtual camera is adjusted to a preset initial spatial parameter with the position and / or orientation of the virtual camera, which further improves the efficiency of the navigation system 1000 and enhances the accuracy of the navigation system 1000. However, since the efficiency is still lacking for medical purposes, after the initial alignment, the adjusted virtual camera needs to be further optimized in the two-dimensional and three-dimensional registration procedure to be more accurate. In some embodiments, each of the first spatial parameter and the second spatial parameter is used to define the position and / or orientation of the virtual camera, and the position and / or orientation is defined to view the three-dimensional volume relative to the virtual camera, or the virtual camera is defined to view the three-dimensional volume. As described above, the adjusted virtual camera needs to be further optimized, and the optimization method of the adjusted virtual camera is to adjust the spatial parameter with the position and / or orientation of the adjusted virtual camera. For example, each of the first spatial parameter and the second spatial parameter is used to define the position and / or orientation of the virtual camera, and the position and / or orientation is defined to view the three-dimensional volume relative to the virtual camera, or the virtual camera is defined to view the three-dimensional volume.

[0084] In some embodiments, each of the first and second spatial parameters includes one of a position, an orientation, and a parameter including a position and an orientation, for example, the first spatial parameter may be a parameter including a position, an orientation, or a position and an orientation, and the second spatial parameter may be a parameter including a position, an orientation, and a parameter including a position and an orientation.

[0085] In some embodiments, each comparison similarity measure and the reference similarity measure are calculated by local normalized correlation (LNC), sum of squared differences (SSD), normalized cross-correlation (NCC), or correlation ratio (CR).

[0086] In some embodiments, the three-dimensional image dataset (e.g., the three-dimensional image dataset stored in database 1200 in FIG. 1 ) is generated by one of a magnetic resonance imaging (MRI) device, an iso-centric C-arm fluoroscopic imaging device, an O-arm device, a bi-plane fluoroscopy device, a computed tomography (CT) device, a multi-slice computed tomography (MSCT) device, a high frequency ultrasound (HIFU) device, an optical coherence tomography (OCT) device, an intra-vascular ultrasound (IVUS) device, a 3D or 4D ultrasound device, and an intraoperative computed tomography (CT) device.

[0087] In some embodiments, the two-dimensional image dataset is generated by one of a C-arm fluoroscopic imaging device, a magnetic resonance imaging (MRI) device, an iso-centric C-arm fluoroscopic imaging device, an O-arm device, a bi-plane fluoroscopy device, a computed tomography (CT) device, a multi-slice computed tomography (MSCT) device, a high frequency ultrasound (HIFU) device, an optical coherence tomography (OCT) device, an intra-vascular ultrasound (IVUS) device, a two-dimensional, three-dimensional or four-dimensional ultrasound device, and an intraoperative computed tomography (CT) device.

[0088] In some embodiments, the first spatial parameter is generated based on a spatial relationship between at least one marker in the two-dimensional image capture device and the region of interest. As described above, the virtual camera is adjusted to the initial spatial parameter in the preset initial alignment to further improve the efficiency and accuracy of the navigation system 1000, and the adjusted spatial parameter after the initial alignment is then the first spatial parameter in the two-dimensional and three-dimensional registration. For example, the first spatial parameter is generated from the spatial relationship between the calibrator 1510 and the target 9000 in the X-ray receiving device 1320 in FIG. 1.

[0089] In some embodiments, the first spatial parameter is the spatial parameter with the greatest similarity among the multiple similarities generated in the previous comparison step. For example, the comparison step may be performed multiple times, and each comparison step generates a similarity. The first spatial parameter is obtained by finding the parameter with the greatest similarity among the multiple similarities.

[0090] In some embodiments, at the start of the 2D and 3D registration, the adjusted virtual camera has a first spatial parameter, the first spatial parameter having a first distance and / or a first orientation. After the start of the 2D and 3D registration, the adjusted virtual camera moves from the first spatial parameter to a second spatial parameter, the second spatial parameter having a second distance and / or a second orientation. Thus, the second spatial parameter differs from the first spatial parameter in distance and / or orientation of the corresponding virtual camera relative to the 3D image dataset (e.g., the 3D image dataset is stored in the database 1200 in FIG. 1 ). More specifically, the adjusted virtual camera in this embodiment is a modularized functional instruction, including an algorithm, an equation, and one or more parameters, so that the processor 1100 can use the parameters and thereafter simulate the adjusted virtual camera at a specific position in the coordinate system of the 3D image dataset.

[0091] In some embodiments, the processor 1120 is further used to perform the steps of generating a third reconstructed image from the three-dimensional image dataset based on the third spatial parameters if the comparison similarity number is greater than the reference similarity number, calculating a second comparison similarity number based on the third reconstructed image and the at least one two-dimensional image dataset, comparing the second comparison similarity number with the reference similarity number, and registering the two-dimensional image dataset and the three-dimensional image dataset if the second comparison similarity number is less than or equal to the reference similarity number.

[0092] For example, if the comparison similarity number is greater than the baseline similarity in the previous comparison, the processor 1120 of the computing device 1100 can access the database 1200 and simulate an adjusted virtual camera corresponding to the three-dimensional image dataset in a third position, and further generate a third digitally reconstructed radiographic image (DRR).

[0093] Additionally, the processor 1120 of the computing device 1100 may calculate a second comparison similarity value based on a third AP and / or LA digitally reconstructed radiographic image (DRR) and the corresponding AP and / or LA two-dimensional image of the two-dimensional image dataset.

[0094] Additionally, the processor 1120 of the computing device 1100 may compare the second comparison similarity value with a reference similarity value.

[0095] Also, if the second comparison similarity value is equal to or less than the reference similarity value, it indicates that the registration of the third AP and / or LA digitally reconstructed radiographic image (DRR) with the corresponding AP and / or LA two-dimensional image of the two-dimensional image dataset is completed. Thereafter, the two-dimensional image dataset acquired in real time by the imaging device 1300 can be considered to be registered with the previously acquired three-dimensional image dataset pre-stored in the database 1200.

[0096] FIG. 11 is a flow chart of a method 4000 for registering a two-dimensional image dataset of a region of interest and a three-dimensional image dataset of a region of interest according to an embodiment of the present application. Please refer to FIG. 2 and FIG. 11 together. In operation, the processor 1120 executes step 4100 to simulate moving the adjusted virtual camera from an origin or a previous spatial position in the coordinate system of the three-dimensional image dataset to a first spatial position, and the processor 1120 executes step 4150 to generate a first digitally reconstructed radiographic image (DRR) (also called a first reconstructed image) of the first spatial position corresponding to the first virtual camera. The processor 1120 then executes step 4200 to calculate a first similarity value based on the first digitally reconstructed radiographic image (DRR) and the two-dimensional image, the two-dimensional image generated by the image device 1300 may be an AP or LA two-dimensional image, and the processor 1120 executes step 4250 to move the adjusted virtual camera back to the initial spatial position.

[0097] Then, the processor 1120 executes step 4300 to determine whether the adjusted virtual camera has moved from the initial spatial position to all spatial positions. For example, the adjusted virtual camera can move from the initial spatial position to a first spatial position (e.g., from (0,0) to (1,0) in the Cartesian coordinate system), and then move from the initial spatial position to a second spatial position, etc. (e.g., from (0,0) to (-1,0) in the Cartesian coordinate system). The movement amount of the adjusted virtual camera may be preset according to actual needs. If it is determined that the adjusted virtual camera has not moved to all spatial positions, the method 4000 returns to execute step 4100 to further move the adjusted virtual camera to another spatial position (e.g., the second spatial position). The processor 1120 then performs steps 4150, 4200 to generate another Digitally Reconstructed Radiograph (DRR) (e.g., a second Digitally Reconstructed Radiograph (DRR)) and to calculate another similarity measure (e.g., a second similarity measure). The processor 1120 then performs step 4250 to move the adjusted virtual camera back to the initial spatial position.

[0098] If it is determined that the adjusted virtual camera has been moved to all spatial positions, the method 4000 continues to execute step 4350. The processor 1120 then executes step 4350 to determine whether the similarity value corresponding to the moved virtual camera is high. If the similarity value corresponding to the moved virtual camera is determined to be high, it indicates that the digitally reconstructed radiographic image (DRR) generated by the adjusted virtual camera at the next position is more similar to the two-dimensional image than the digitally reconstructed radiographic image (DRR) generated by the adjusted virtual camera at the initial or previous spatial position. Therefore, the processor 1120 then executes step 4400 to adjust the adjusted virtual camera from the initial or previous spatial position to the next spatial position. The initial spatial position in this embodiment represents a position where the adjusted virtual camera is located each time the processor 1120 initially positions the adjusted virtual camera, and the previous spatial position represents a position where the adjusted virtual camera is repositioned after adjustment, the adjustment being generated by one or a combination of the above-mentioned alignment (e.g., initial alignment) or registration steps.

[0099] After performing step 4400, method 4000 returns to perform step 4100. Steps 4100, 4150, 4200, 4250, 4300, and 4350 are then performed by processor 1120. See step 4350. If it is determined that the similarity value corresponding to the adjusted virtual camera at the next position is less than or equal to the similarity value of any previous position, method 4000 continues with execution of step 4450.

[0100] After performing step 4450, processor 1120 performs step 4500 to reduce the adjustment amount. For example, if the previous adjustment amount was to move the adjusted virtual camera by 1 mm from its previous position in the coordinate system of the displayed three-dimensional target of the three-dimensional image dataset, processor 1120 performs step 4500 to reduce the adjustment amount to move by 0.5 mm. Processor 1120 then performs step 4550 to determine whether the adjustment amount is less than a preset spatial value, for example 0.75 mm. If it is determined that the adjustment amount is equal to or greater than the preset spatial value, method 4000 returns to perform step 4100.

[0101] If it is determined that the adjustment amount is smaller than the preset spatial value, it indicates that the alignment between the digitally reconstructed radiographic image (DRR) and the two-dimensional image is almost complete, and therefore, it can be defined that the registration between the two-dimensional image data set of the imaging device 1300 and the three-dimensional image data set pre-stored in the database 1200 is complete.

[0102] In some embodiments, if the difference between the first spatial parameter and the second spatial parameter is equal to or less than a preset spatial value, the two-dimensional image dataset and the three-dimensional image dataset are registered. For example, in step 4550 in FIG. 11, the difference between the first spatial position and the second spatial position is the adjustment amount in step 4550. If the adjustment amount between the first spatial position and the second spatial position is equal to or less than a preset spatial value, it indicates that the registration between the digitally reconstructed radiographic image (DRR) and the two-dimensional image data is completed. For example, the current spatial value may be 0.01 mm. If the adjustment amount is equal to or less than 0.01 mm, it indicates that the registration between the digitally reconstructed radiographic image (DRR) and the two-dimensional image data is completed. Therefore, the processor 1120 of the computing device 1100 can register the two-dimensional image dataset of the imaging device 1300 and the three-dimensional image dataset stored in the database 1200.

[0103] It should be clarified that the present application is not limited to the operation of the embodiment shown in FIG. 11, which is only used to illustratively show one of the implementation forms of the present application.

[0104] Figure 12 is a flow chart of a method 5000 for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest according to one embodiment of the present application (see Figures 2 and 12 in conjunction). In operation, the processor 1120 executes step 5100 to perform an initial alignment procedure for the navigation platform 1000 and further refine the accuracy of the navigation platform 1000.

[0105] The processor 1120 executes step 5200 to perform an XY alignment procedure to pre-set the XY position of the calibrated virtual camera, which corresponds to the three-dimensional image dataset pre-stored in the database 1200 of the navigation platform 1000.

[0106] The processor 1120 executes step 5300 to perform an initial thumbnail alignment for a calibrated virtual camera, which corresponds to a three-dimensional image dataset pre-stored in the database 1200 of the navigation platform 1000.

[0107] The processor 1120 executes step 5400 to perform original view registration for the calibrated virtual camera, which corresponds to the three-dimensional image dataset pre-stored in the database 1200 of the navigation platform 1000 .

[0108] It should be explained that the flow charts of XY coordinate alignment in step 5200, thumbnail alignment in step 5300 and initial alignment in step 5400 are all shown in Fig. 11. The difference is that XY coordinate alignment focuses on image alignment in X and Y coordinates, thumbnail alignment achieves the purpose of quick alignment using preview images or reduced images, and initial alignment further improves the alignment situation after thumbnail alignment (e.g., preview alignment).

[0109] The processor 1120 executes step 5500 to determine whether realignment is necessary. If an error or alignment failure is determined by the processor 1120 or the surgeon, a realignment procedure needs to be performed. For example, if after multiple calculation cycles the difference is much larger than a pre-set value, the method 5000 continues to execute step 5600 to perform realignment. On the other hand, if the alignment is nearly complete and there is no need to perform a realignment procedure, the method 5000 continues to execute step 5700. In this situation, the entire alignment procedure in FIG. 12 has already been performed.

[0110] In some embodiments, before the calibrated virtual camera generates the first and second reconstructed images, the navigation system 1000 performs an initial registration based on the two-dimensional image using a low-resolution image. The use of a low-resolution format is to reduce interference or noise between two similar images, thus contributing to the initial registration. In step 5300 of the method 5000, the digitally reconstructed radiographic image (DRR) used for thumbnail registration is the low-resolution image.

[0111] Realign

[0112] It should be clarified that the present application is not limited to the operation of the embodiment shown in FIG. 12, which is only used to illustratively show one of the implementation forms of the present application.

[0113] The re-registration flow of step 5600 in the method 5000 will be described in detail as in the following FIG. 13. FIG. 13 is a flow chart of a method 6000 for re-registration according to an embodiment of the present application. In clinical practice, the registration procedure of the two-dimensional image and the three-dimensional image may fail in the registration procedure. Therefore, the registration of the AP or LA two-dimensional image with the digitally reconstructed radiographic image (DRR) may not be successful, and the digitally reconstructed radiographic image (DRR) is generated based on the three-dimensional image data set. However, the registration failure occurs when only one of the two-dimensional images is not registered with the corresponding digitally reconstructed radiographic image (DRR). When any kind of failure occurs, all known navigation systems are set to stop performing the registration procedure. The impact on the person skilled in the art or the surgeon is very large, and they need to go back to the first step and re-perform the registration procedure once. Thus, the whole registration procedure is significantly delayed. In this embodiment, the navigation system 1000 can solve this problem by performing the re-registration flow.

[0114] Please refer to FIG. 2 and FIG. 13. If the registration between the LA digitally reconstructed radiographic image (DRR) (also called the LA three-dimensional reconstruction image) and the LA C-arm X-ray image (also called the two-dimensional image) is not completed, re-registration is required, and the processor 1120 can obtain the spatial parameters of the AP virtual camera, which correspond to the AP digitally reconstructed radiographic image (DRR) successfully registered with the AP C-arm X-ray image in the previous step. As described above, the AP virtual camera is hereinafter regarded as a registered virtual camera, and the LA virtual camera is hereinafter regarded as an unregistered virtual camera. However, the present application is not limited to the above embodiment, and the AP virtual camera can be set as an unregistered virtual camera and the LA virtual camera as a registered virtual camera according to actual needs.

[0115] To facilitate understanding of the method 6000 shown in FIG. 13, please refer to FIG. 14, which is a schematic diagram of a virtual camera VC according to an embodiment of the present application. It should be noted that the virtual camera VC in FIG. 14 is used to explain the concept of the present application. The spatial parameters of the virtual camera VC may include, but are not limited to, multiple position and / or orientation information, such as vectors. Both the registered virtual camera and the unregistered virtual camera can be interpreted in FIG. 14. For example, the spatial parameters of each of the registered virtual camera and the unregistered virtual camera may include Z-axis and Y-axis vectors and a focus FP, which can define a coordinate system of a three-dimensional target generated from a three-dimensional image dataset. It should be noted that the focus FP may not be included in the spatial parameters, but may be obtained from the database 1200 by the processor 1120. In the recognition procedure, the registered virtual camera is subsequently marked as VC1, and its axis direction and focus are marked as Z1, Y1, and FP1. Note that the unregistered virtual camera is subsequently marked as VC2, and its axis direction and focus are marked as Z2, Y2, and FP2.

[0116] See Figures 2, 13 and 14. In operation, the processor 1120 executes step 6100 to obtain a first vector from the spatial parameters of the registered virtual camera in the coordinate system of the three-dimensional image dataset. For example, the processor 1120 can obtain a Z1 axis from the spatial parameters of the registered virtual camera VC1 in the coordinate system of the three-dimensional image dataset pre-stored in the database 1200.

[0117] The processor 1120 executes step 6200 to transform the first vector of the registered virtual camera by at least one transformation matrix to obtain the first transformation vector of the unregistered virtual camera in the coordinate system of the three-dimensional image dataset. For example, the processor 1120 can transform the Z1 axis of the registered virtual camera VC1 by a set or pre-stored transformation matrix to obtain the Z2 axis of the unregistered virtual camera VC2 in the coordinate system of the three-dimensional image dataset, where the set or pre-stored transformation matrix is ​​established by the processor 1120 based on the spatial relationship of a particular element in the navigation system 1000.

[0118] The processor executes step 6300 to obtain a focal point of an unregistered virtual camera in the coordinate system of the three-dimensional image dataset, where the unregistered virtual camera is on a reference point of the two-dimensional image dataset, and the unregistered two-dimensional image is an X-ray image of the unregistered LA. For example, the processor 1120 can obtain a focal point FP2 of the unregistered virtual camera VC2 at a center point of the calibrator of the two-dimensional image in the coordinate system of the three-dimensional image dataset based on a previously performed, unsuccessful registration.

[0119] The processor executes step 6400 to reposition the unregistered virtual camera based on the first transformation vector and the focal point of the unregistered virtual camera, and generate an updated reconstructed image based on the repositioning of the unregistered virtual camera. For example, the unregistered virtual camera VC2 can be simulated to be moved from its initial position to a calculated position calculated by the processor 1120, where the calculated position was calculated based on the Z2 axis of the unregistered virtual camera VC2 and the focal point FP2 of the unregistered virtual camera VC2.

[0120] In some embodiments, the first vector of the registered virtual camera is from the position of the registered virtual camera to the focal point of the registered virtual camera. For example, as shown in FIG 14, the Z1 axis of the registered virtual camera VC1 is from the position of the registered virtual camera VC1 to the focal point FP1 of the registered virtual camera VC1.

[0121] In some embodiments, the first vector is defined as the Z axis (e.g., the Z1 axis in FIG. 14) of a registered virtual camera (e.g., the registered virtual camera VC1 in FIG. 14) in the coordinate system of the three-dimensional image dataset.

[0122] In some embodiments, the processor 1120 is used to execute the steps of obtaining a second vector from the spatial parameters of the registered virtual camera in the coordinate system of the three-dimensional image dataset, and obtaining a second transformation vector from the spatial parameters of the unregistered virtual camera based on the second vector of the registered virtual camera in the coordinate system of the three-dimensional image dataset, by obtaining a plurality of instructions via the memory 1110.

[0123] For example, the processor 1120 obtains a Y axis from the spatial parameters of the registered virtual camera VC1 in the coordinate system of the three-dimensional image dataset. The processor 1120 obtains a Y2 axis of the unregistered virtual camera VC2 based on the Y1 axis of the spatial parameters of the registered virtual camera VC1 in the coordinate system of the three-dimensional image dataset.

[0124] In some embodiments, the second vector is from a center point (e.g., point FP in FIG. 14) to the top of a reconstructed image (e.g., the top in FIG. 14), the reconstructed image being acquired by a registered virtual camera parallel to the reconstructed image (e.g., registered virtual camera VC1 in FIG. 14).

[0125] In some embodiments, in the coordinate system of the three-dimensional image dataset, the second vector is defined as the Y axis (e.g., the Y1 axis in FIG. 14) of a spatial parameter of a registered virtual camera (e.g., the registered virtual camera VC1 in FIG. 14).

[0126] It should be clarified that the present application is not limited to the structure and operation of the embodiment shown in FIG. 13 and FIG. 14, which are only used to exemplify one of the implementation forms of the present application.

[0127] In some embodiments, the two-dimensional image dataset includes first and second two-dimensional images, and the at least one transformation matrix includes a first matrix for transforming a coordinate system of the first two-dimensional image to a coordinate system of the second two-dimensional image, for example, the two-dimensional image dataset includes an AP two-dimensional image and an LA two-dimensional image, and the transformation matrix includes a first matrix of a coordinate system for transforming a coordinate system of the AP two-dimensional image to a coordinate system of the LA two-dimensional image.

[0128] In some embodiments, in the coordinate system of the three-dimensional image dataset, the reference point is located at the center point of the calibrator module in the two-dimensional image dataset. For example, the reference point is located at the center point (also called the origin) of the calibrator module, and the calibrator module is displayed on an X-ray two-dimensional image of the unregistered AP or LA, and the X-ray two-dimensional image of the unregistered AP or LA is included in the two-dimensional image dataset. The reference point is then simulated to be positioned in position and orientation in the coordinate system of the three-dimensional image dataset for further provision to a program of the navigation system 1000.

[0129] In some embodiments, the at least one transformation matrix includes a second matrix and a third matrix, the second matrix is ​​used to transform the coordinate system of the fiducial mark into the coordinate system of the three-dimensional image data set, and the third matrix is ​​used to transform the coordinate system of the fiducial mark into the coordinate system of the tracker module. For example, the at least one transformation matrix includes a second matrix and a third matrix. The second matrix is ​​used to transform the coordinate system of the calibrator 1510, 1530 and the dynamic reference coordinate 1520A in FIG. 1 into the coordinate system of the three-dimensional image data set stored in the database 1200 in FIG. 1. The third matrix is ​​used to transform the coordinate system of the calibrator 1510, 1530 and the dynamic reference coordinate 1520A in FIG. 1 into the coordinate system of the tracker 1400 in FIG. 1. Transformation matrices and transformation of coordinate systems are well known techniques. For more details, see L. Dorst, etc., Geometric Algebra For Computer Science, published by Morgan Kaufmann Publishers and MN Oosterom, etc., Navigation of a robot-integrated fluorescence laparoscope in preoperative SPECT / CT and intraoperative freehand SPECT imaging data: A phantom study, August 2016, Journal of Biomedical Optics 21(8):086008, the entire contents of which are incorporated herein by reference.

[0130] The re-registration of step 5600 of method 5000 in Fig. 12 is described in subsequent Fig. 15. Fig. 15 is a flow chart of a re-registration method 7000 according to one embodiment of the present application. Please refer to Fig. 2 and Fig. 15. If an unregistered virtual camera corresponding to vertebral body V1 in Fig. 5 or Fig. 7 needs to be re-registered, the processor 1120 can find a registered virtual camera corresponding to vertebral body V0 or V2 in Fig. 5 or Fig. 7.

[0131] See Figures 2 and 15. In operation, the processor 1120 executes step 7100 to obtain first spatial parameters of a first registered virtual camera, where the setting position of the first registered virtual camera is positioned corresponding to a first two-dimensional image of the two-dimensional image dataset. For example, the processor 1120 can obtain first spatial parameters corresponding to a registered virtual camera for a vertebra V0 or V2 in Figure 5 or Figure 7. The setting position of the registered virtual camera is positioned corresponding to a two-dimensional image for a vertebra V0 or V2 in Figure 4 or Figure 6.

[0132] See Figures 2 and 15. In operation, the processor 1120 executes step 7200 to generate an updated reconstructed image by adjusting the second spatial parameters of the first unregistered virtual camera with the first spatial parameters of the first registered virtual camera and repositioning the first unregistered virtual camera, where the position of the first unregistered virtual camera does not correspond to the first two-dimensional image of the two-dimensional image dataset. For example, the processor 1120 can adjust the second spatial parameters of the unregistered virtual camera with the first spatial parameters of the registered virtual camera. The first spatial parameters of the registered virtual camera correspond to the vertebral body V0 or V2 shown in Figure 5 or Figure 7, and the second spatial parameters of the unregistered virtual camera correspond to the vertebral body V1 shown in Figure 5 or Figure 7. The position of the unregistered virtual camera does not correspond to the two-dimensional image of the vertebral body V1 shown in Figure 4 or Figure 6.

[0133] In some embodiments, a first portion of the region of interest is included in the first two-dimensional image, a second portion of the region of interest is included in the second two-dimensional image, and the first and second portions of the region of interest are adjacent to each other. For example, the target 9000 is a patient with a spinal disease and needs to perform surgery to stabilize three vertebral segments. All three vertebral segments are considered as regions of interest and are depicted in Figures 4 and 6. Based on the success or failure of the registration of the three-dimensional image data sets, the vertebral segments can be classified into two portions.

[0134] The first portion includes vertebral body V0 or V2, and the second portion includes vertebral body V1. More specifically, the first portion of the area of ​​interest in FIG. 1 is included in the first two-dimensional image corresponding to vertebral body V0 or V2. The first two-dimensional image in FIG. 4 is an AP x-ray image, and the first two-dimensional image in FIG. 6 is an LA x-ray image. The second portion of the area of ​​interest in FIG. 1 is included in the first two-dimensional image corresponding to vertebral body V1. It is the first two-dimensional image depicted in FIG. 4 and FIG. 6. As shown in FIG. 4 and FIG. 6, the vertebral body V0 or V2 included in the first portion of the target 9000 is adjacent to the vertebral body V1 included in the second portion of the target 9000 corresponding to the vertebral body V1.

[0135] In some embodiments, a first portion of the region of interest is defined based on a first marker in a first two-dimensional image of the two-dimensional image data set, and a second portion of the region of interest is defined based on a second marker in the first two-dimensional image of the two-dimensional image data set. For example, the first portion of the region of interest of the target 9000 in Fig. 1 can be defined by a program of the navigation system 1000 to automatically recognize an image edge of a vertebral body, which is marked with V0 or V2 in the two-dimensional images in Fig. 4 and Fig. 6. Similar to the manner of the first portion, the second portion of the region of interest of the target 9000 in Fig. 1 is defined by a program of the navigation system 1000 to automatically recognize an image edge of a vertebral body, which is marked with V1 in the two-dimensional images in Fig. 4 and Fig. 6.

[0136] In some embodiments, the first spatial parameters of the first registered virtual camera include position and / or orientation data, the position and / or orientation being defined to view the three-dimensional volume relative to the first registered virtual camera, or the first registered virtual camera relative to the three-dimensional volume. For example, the first spatial parameters of the registered virtual camera include position and / or orientation data, the position and / or orientation being defined to view the three-dimensional volume relative to the registered virtual camera, or the registered virtual camera relative to the three-dimensional volume. The first spatial parameters of the registered virtual camera correspond to a vertebral body V0 or V2 in FIG. 5 or FIG. 7, and the three-dimensional volume is generated from the three-dimensional image dataset.

[0137] In some embodiments, the second spatial parameters of the first unregistered virtual camera include position and / or orientation data, where the position and / or orientation is defined relative to the first unregistered virtual camera with respect to the three-dimensional volume, or the first unregistered virtual camera is defined to view the three-dimensional volume. For example, the second spatial parameters of the unregistered virtual camera include position and / or orientation data, where the position and / or orientation is defined relative to the first unregistered virtual camera with respect to the three-dimensional volume, or the unregistered virtual camera is defined to view the three-dimensional volume. The second spatial parameters of the unregistered virtual camera correspond to vertebral body V1 in FIG. 5 or FIG. 7, and the three-dimensional volume is generated from the three-dimensional image dataset.

[0138] In some embodiments, the position of the first registered virtual camera is set based on a comparison result between the plurality of similarity values, the comparison result between the plurality of similarity values ​​being calculated based on a first two-dimensional image of the plurality of different reconstructed images and the two-dimensional image data, and the plurality of different reconstructed images are obtained from the three-dimensional image data set. For example, the position of the registered virtual camera is set based on a comparison result between the plurality of similarity values, the comparison result between the plurality of similarity values ​​being calculated based on a first two-dimensional image of the plurality of different reconstructed images and the two-dimensional image data, and the plurality of different reconstructed images are obtained from the three-dimensional image data set.

[0139] In some embodiments, each similarity measure is calculated by local normalized correlation (LNC), sum of squared differences (SSD), normalized cross-correlation (NCC), or correlation ratio (CR).

[0140] In some embodiments, the processor 1120 is further used to perform the steps of defining a second spatial parameter of the first unregistered virtual camera as an N spatial parameter; determining whether a setting position of the first registered virtual camera corresponds to a first two-dimensional image of the two-dimensional image dataset, where the first registered virtual camera has NM spatial parameters, N and M are integers and M is less than N; and if the setting position of the first registered virtual camera corresponds to the first two-dimensional image of the two-dimensional image dataset, defining the NM spatial parameter as the first spatial parameter of the first registered virtual camera.

[0141] For example, the processor 1120 of the computing device 1100 can define the second spatial parameter of the unregistered virtual camera as the N spatial parameter. The processor 1120 of the computing device 1100 can determine whether the setting position of the registered virtual camera corresponds to the two-dimensional image, the two-dimensional image is related to the vertebra V0 or V2 in FIG. 4 or FIG. 6, and the registered virtual camera has NM spatial parameters, N and M are integers, and M is smaller than N. If the setting position of the registered virtual camera corresponds to the two-dimensional image, the processor 1120 of the computing device 1100 defines the NM spatial parameter as the first spatial parameter of the registered virtual camera, the two-dimensional image is related to the vertebra V0 or V2 in FIG. 4 or FIG. 6. In particular, the N spatial parameter corresponds to the unregistered virtual camera. The navigation system 1000 can find the NM spatial parameter corresponding to the registered virtual camera, and the NM spatial parameter is used for the unregistered virtual camera to contribute to the subsequent procedure.

[0142] In some embodiments, the processor 1120 is further used to perform the steps of defining a second spatial parameter of the first unregistered virtual camera as an N spatial parameter; determining whether the setting position of the first registered virtual camera corresponds to a first two-dimensional image of the two-dimensional image dataset, where the first registered virtual camera has N+M spatial parameters, N and M are integers and M is less than N; and if the setting position of the first registered virtual camera corresponds to the first two-dimensional image of the two-dimensional image dataset, defining the N+M spatial parameter as the first spatial parameter of the first registered virtual camera.

[0143] For example, the processor 1120 of the computing device 1100 can define the second spatial parameter of the unregistered virtual camera as N spatial parameters. The processor 1120 of the computing device 1100 determines whether the setting position of the registered virtual camera corresponds to a two-dimensional image, the two-dimensional image is related to the vertebral body V0 or V2 in FIG. 4 or FIG. 6, and the registered virtual camera has N+M spatial parameters, N and M are integers, and M is smaller than N. If the setting position of the registered virtual camera corresponds to the two-dimensional image, the processor 1120 of the computing device 1100 can define the N+M spatial parameters as the first spatial parameters of the registered virtual camera. The two-dimensional image is related to the vertebral body V0 or V2 in FIG. 4 or FIG. 6.

[0144] It should be clarified that the present application is not limited to the operation of the embodiment shown in FIG. 15, which is only used to illustratively show one of the implementation forms of the present application.

[0145] As mentioned above, the realignment in step 5600 of method 5000 in Fig. 12 includes, but is not limited to, the two realignment procedures in Fig. 13 and Fig. 15. To facilitate understanding of the realignment procedures in Fig. 13 and Fig. 15, please refer to Fig. 16, which is a flow chart of a method 8000 for realignment according to one embodiment of the present application.

[0146] Please refer to FIG. 2 and FIG. 16 together. In operation, the processor 1120 executes step 8200 to determine whether registration of one of the AP digitally reconstructed radiographic image (DRR) acquired by the AP virtual camera and the LA digitally reconstructed radiographic image (DRR) acquired by the LA virtual camera is completed. If it is determined that one of the digitally reconstructed radiographic image (DRR) acquired by the AP virtual camera or the LA virtual camera is not registered, the method 8000 continues to execute step 8300. In particular, if the AP virtual camera is not registered, the processor 1120 executes step 8300 to reset the unregistered AP virtual camera based on the registered LA virtual camera. The method 6000 in FIG. 13 is similar to step 8300 in the method 8000. If the digitally reconstructed radiographic image (DRR) acquired by the AP virtual camera or the LA virtual camera is not registered, the method 6000 finds and uses the registered virtual camera to reset the unregistered virtual camera. After that, the processor 1120 executes the step 8700 to perform original view registration using the re-set virtual camera. Note that the processor 1120 executes the step 8400 to adjust the ROI region to avoid interference.

[0147] After performing step 8200, if it is determined that neither of the digitally reconstructed radiographic images (DRR) acquired by the AP virtual camera and the LA virtual camera are registered, the method 8000 continues to perform step 8500. In particular, if neither of the AP virtual camera and the LA virtual camera are registered, it indicates that the registration of both of the AP and LA digitally reconstructed radiographic images (DRR) corresponding to the first vertebral body has failed. The processor 1120 performs step 8500 to find another AP virtual camera and another LA virtual camera corresponding to the second vertebra, and the digitally reconstructed radiographic images (DRR) of the virtual cameras are successfully registered with the three-dimensional image dataset. Thereafter, the processor 1120 can reset the unregistered AP virtual camera and the unregistered LA virtual camera corresponding to the first vertebra based on the spatial parameters or data of the registered AP virtual camera and the registered LA virtual camera corresponding to the second vertebra. The method 7000 in FIG. 15 is similar to step 8500 in the method 8000. If the digitally reconstructed radiographic images (DRR) acquired by the AP virtual camera and the LA virtual camera corresponding to the first vertebra both fail to be registered, the method 7000 finds and uses the spatial parameters or data of the registered AP virtual camera and the registered LA virtual camera corresponding to the second vertebra to reset the unregistered AP virtual camera and the unregistered LA virtual camera corresponding to the first vertebra. Then, the processor 1120 executes the step 8700 to perform original view registration using the reset virtual cameras. Note that the processor 1120 executes the step 8600 to adjust the region of interest to avoid interference.

[0148] As can be seen from the above embodiments of the present application, the present application has the following advantages: The method and navigation system for registering a two-dimensional image dataset of a region of interest and a three-dimensional image dataset of a region of interest shown in the embodiments of the present application can pre-store the three-dimensional image dataset of the region of interest in a database, and then only two X-ray images (two-dimensional images) of the patient (region of interest) during surgery can be taken to establish the relationship between the two-dimensional image dataset and the three-dimensional image dataset. The method and navigation system of the present application can then provide accurate navigation during surgery through the pre-stored three-dimensional image dataset. The method and navigation system of the present application can reduce the radiation exposure of the patient (region of interest) by more than 98% because only two X-ray images (two-dimensional images) of the region of interest are required. As described above, the method and navigation system of the present application can perform two-dimensional and three-dimensional registration in a more accurate and more efficient manner.

[0149] In the above embodiments, specific examples of the present application are disclosed, but are not intended to limit the present application. Those skilled in the art may make various changes and modifications without departing from the principles and spirit of the present application, and therefore the scope of protection of the present application should be based on that defined in the attached patent scope.

Claims

1. 1. A method for registering a two-dimensional image dataset of a region of interest and a three-dimensional image dataset of the region of interest, comprising the steps of: generating a first reconstructed image from the three-dimensional image dataset based on first spatial parameters; calculating a reference similarity measure based on the first reconstructed image and the two-dimensional image data set; generating a second reconstructed image from the three-dimensional image dataset based on second spatial parameters; calculating a comparative similarity measure based on the second reconstructed image and the two-dimensional image data set; comparing the comparison similarity value with the reference similarity value; if the comparison similarity score is less than or equal to the reference similarity score, registering the two-dimensional image dataset and the three-dimensional image dataset, and using the registered two-dimensional image dataset and the three-dimensional image dataset for computer-assisted surgical navigation; 16. A method for registering a two-dimensional image dataset of a region of interest and a three-dimensional image dataset of the region of interest, comprising:

2. generating a third reconstructed image from the three-dimensional image data set based on third spatial parameters if the comparison similarity measure is greater than the reference similarity measure; calculating a second comparative similarity measure based on the third reconstructed image and the two-dimensional image data set; comparing the second comparison similarity value with the reference similarity value; registering the two-dimensional image data set and the three-dimensional image data set if the second comparison similarity measure is less than or equal to the reference similarity measure; The method of claim 1 further comprising:

3. generating the first reconstructed image or the second reconstructed image from the three-dimensional image data set based on the first spatial parameter or the second spatial parameter, determining a position and orientation of a virtual camera within a coordinate system of the three-dimensional image dataset; using the virtual camera corresponding to the position and orientation to target the three-dimensional image data set and generate a two-dimensional image as the first reconstruction image or the second reconstruction image; 2. The method of claim 1, comprising:

4. The method of claim 3 , wherein the virtual camera is a modularized functional instruction, comprising one of an algorithm and an equation.

5. 4. The method of claim 3, wherein each of the first and second spatial parameters is used to define a position and / or an orientation of the virtual camera, the position and / or the orientation being defined to view a three-dimensional volume relative to the virtual camera, or to view the virtual camera relative to the three-dimensional volume.

6. The method of claim 1 , wherein each of the first spatial parameter and the second spatial parameter comprises one of a position, an orientation, and a parameter that includes the position and the orientation.

7. 2. The method of claim 1, wherein each of the comparison similarity values ​​and the reference similarity value is calculated by a local normalized correlation (LNC), a sum of squared differences (SSD), a normalized cross-correlation (NCC), or a correlation ratio (CR).

8. The three-dimensional image data set may be acquired from a magnetic resonance imaging (MRI) device, an isocentric C-arm fluoroscopic imaging device, an O-arm device, a bi-plane fluoroscopy device, a computed tomography (CT) device, a multi-slice computed tomography (MSCT) device, a high frequency ultrasound (HIFU) device, an optical coherence tomography (OCT) device, or a combination of the above.

2. The method of claim 1, wherein the image is generated by one of an optical coherence tomography (OCT) device, an intravascular ultrasound (IVUS) device, a 3D or 4D ultrasound device, and an intraoperative computed tomography (CT) device.

9. The two-dimensional image data set may be obtained from a C-arm fluoroscopic imaging device, a magnetic resonance imaging (MRI) device, an isocentric C-arm fluoroscopic imaging device, an O-arm device, a bi-plane fluoroscopy device, a computed tomography (CT) device, a multi-slice computed tomography (MSCT) device, a high frequency ultrasound (HFUS) device, or a combination of the above.

2. The method of claim 1, wherein the image is generated by one of a high-frequency ultrasound (HIFU) device, an optical coherence tomography (OCT) device, an intra-vascular ultrasound (IVUS) device, a two-dimensional, three-dimensional or four-dimensional ultrasound device, and an intraoperative computed tomography (CT) device.

10. The method of claim 1 , wherein the first spatial parameter is generated based on a spatial relationship between at least one marker in a two-dimensional image capture device and the area of ​​interest.

11. The method of claim 1 , wherein the first spatial parameter is the spatial parameter having the greatest similarity among a plurality of similarities generated in the previous comparison step.

12. The method of claim 1 , wherein the second spatial parameters differ from the first spatial parameters in distance and / or orientation of the corresponding virtual camera relative to the 3D image dataset.

13. 2. The method of claim 1, further comprising the step of registering the two-dimensional image dataset and the three-dimensional image dataset if a difference between the first spatial parameter and the second spatial parameter is less than or equal to a preset spatial value.

14. The method of claim 1 , wherein the first reconstructed image and the second reconstructed image are low resolution images.

15. adjusting a first virtual camera based on distance parameters calculated corresponding to a relationship between the two-dimensional image data set and the area of ​​interest; rotating the first virtual camera based on an angular difference between a first vector calculated from two spatial marks in the three-dimensional image data set and a second vector calculated from two first planar marks in the two-dimensional image data set; rotating the first virtual camera based on an angle, the angle corresponding to an angular difference between a reconstructed image having a maximum similarity parameter and the two-dimensional image data set, the reconstructed image having a maximum similarity parameter among a plurality of similarity parameters calculated for a plurality of reconstructed images in the three-dimensional image data set, the plurality of reconstructed images including a reconstructed image generated by the first virtual camera and another reconstructed image generated by another virtual camera, the other reconstructed image and the reconstructed image generated by the first virtual camera having different angles or pixels; The method of claim 1 further comprising:

16. defining a first vector of a registered virtual camera in a coordinate system of the three-dimensional image dataset; transforming the first vector of the registered virtual camera by at least one transformation matrix to obtain a first transformation vector of an unregistered virtual camera located in the coordinate system of the three-dimensional image dataset; defining a focal point of the unregistered virtual camera on a reference point of the two-dimensional image dataset in the coordinate system of the three-dimensional image dataset; repositioning the unregistered virtual camera based on the first transformation vector and the focal point of the unregistered virtual camera; The method of claim 1 further comprising:

17. obtaining first spatial parameters of a first registered virtual camera positioned corresponding to a first two-dimensional image of the two-dimensional image dataset; adjusting a second spatial parameter of a first unregistered virtual camera with the first spatial parameter of the first registered virtual camera, the first unregistered virtual camera being unable to position itself relative to a second two-dimensional image of the two-dimensional image dataset; The method of claim 1 further comprising:

18. 1. A navigation system for registering a two-dimensional image dataset of a region of interest and a three-dimensional image dataset of said region of interest, comprising: a memory for storing a plurality of instructions; retrieving the instructions by the memory; generating a first reconstructed image from the three-dimensional image dataset based on first spatial parameters; calculating a reference similarity measure based on the first reconstructed image and the two-dimensional image data set; generating a second reconstructed image from the three-dimensional image dataset based on second spatial parameters; calculating a comparative similarity measure based on the second reconstructed image and the two-dimensional image data set; comparing the comparison similarity value with the reference similarity value; if the comparison similarity score is less than or equal to the reference similarity score, registering the two-dimensional image dataset and the three-dimensional image dataset, and using the registered two-dimensional image dataset and the three-dimensional image dataset for computer-assisted surgical navigation; Includes a navigation system.

19. The processor retrieves the instructions from the memory, generating a third reconstructed image from the three-dimensional image data set based on third spatial parameters if the comparison similarity measure is greater than the reference similarity measure; calculating a second comparative similarity measure based on the third reconstructed image and the two-dimensional image data set; comparing the second comparison similarity value with the reference similarity value; and if the second comparison similarity measure is less than or equal to the reference similarity measure, registering the two-dimensional image data set and the three-dimensional image data set.

20. The step of generating the first reconstructed image or the second reconstructed image from the three-dimensional image dataset based on the first spatial parameter or the second spatial parameter, which is executed by the processor, comprises: The navigation system of claim 18, further comprising a step of positioning a virtual camera based on a three-dimensional target, the three-dimensional target being constructed based on the first reconstructed image or the second reconstructed image, and capturing the first reconstructed image or the second reconstructed image of the three-dimensional target by the virtual camera.

21. 21. The navigation system of claim 20, wherein the virtual camera is a modularized functional instruction, comprising one of an algorithm and an equation.

22. 21. The navigation system of claim 20, wherein each of the first and second spatial parameters is used to define a position and / or orientation of the virtual camera, the position and / or the orientation being defined to view the three-dimensional volume relative to the virtual camera, or to view the virtual camera relative to the three-dimensional volume.

23. 20. The navigation system of claim 18, wherein each of the first and second spatial parameters includes one of a position, an orientation, and a parameter that includes the position and the orientation.

24. 20. The navigation system of claim 18, wherein each of the comparison similarity values ​​and the reference similarity value is calculated by a local normalized correlation (LNC), a sum of squared differences (SSD), a normalized cross-correlation (NCC) or a correlation ratio (CR).

25. The three-dimensional image data set may be acquired from a magnetic resonance imaging (MRI) device, an isocentric C-arm fluoroscopic imaging device, an O-arm device, a bi-plane fluoroscopy device, a computed tomography (CT) device, a multi-slice computed tomography (MSCT) device, a high frequency ultrasound (HIFU) device, an optical coherence tomography (OCT) device, or a combination of the above.

20. The navigation system of claim 18, wherein the navigation system is generated by one of an optical computed tomography (OCT) device, an intravascular ultrasound (IVUS) device, a 3D or 4D ultrasound device, and an intraoperative computed tomography (CT) device.

26. The two-dimensional image data set may be obtained from a C-arm fluoroscopic imaging device, a magnetic resonance imaging (MRI) device, an isocentric C-arm fluoroscopic imaging device, an O-arm device, a bi-plane fluoroscopy device, a computed tomography (CT) device, a multi-slice computed tomography (MSCT) device, a high frequency ultrasound (HFUS) device, or a combination of the above.

20. The navigation system of claim 18, wherein the image is generated by one of a high-frequency ultrasound (HIFU) device, an optical coherence tomography (OCT) device, an intra-vascular ultrasound (IVUS) device, a two-dimensional, three-dimensional or four-dimensional ultrasound device, and an intraoperative computed tomography (CT) device.

27. 20. The navigation system of claim 18, wherein the first spatial parameter is generated based on a spatial relationship between at least one marker in a two-dimensional image capture device and the area of ​​interest.

28. 20. The navigation system of claim 18, wherein the first spatial parameter is the spatial parameter having the greatest similarity among a plurality of similarities generated in the previous comparison step.

29. 20. The navigation system of claim 18, wherein the second spatial parameter differs from the first spatial parameter in distance and / or orientation of the corresponding virtual camera relative to the 3D image data set.

30. The processor retrieves the instructions from the memory, 20. The navigation system of claim 18, adapted to perform a step of registering the two-dimensional image dataset and the three-dimensional image dataset when a difference between the first spatial parameter and the second spatial parameter is less than or equal to a preset spatial value.

31. The navigation system of claim 18 , wherein the first reconstructed image and the second reconstructed image are low resolution images.

32. The processor retrieves the instructions from the memory, adjusting a first virtual camera based on distance parameters calculated corresponding to a relationship between the two-dimensional image data set and the area of ​​interest; rotating the first virtual camera based on an angular difference between a first vector calculated from two spatial marks in the three-dimensional image data set and a second vector calculated from two first planar marks in the two-dimensional image data set; 20. The navigation system of claim 18, further comprising: a step of rotating the first virtual camera based on an angle, the angle corresponding to an angular difference between a reconstructed image having a maximum similarity parameter and the two-dimensional image dataset, the reconstructed image having a maximum similarity parameter being one of a plurality of similarity parameters calculated for a plurality of reconstructed images in the three-dimensional image dataset, the plurality of reconstructed images including a reconstructed image generated by the first virtual camera and another reconstructed image generated by another virtual camera, and the other reconstructed images and the reconstructed image generated by the first virtual camera having different angles or pixels.

33. The processor retrieves the instructions from the memory, defining a first vector of a registered virtual camera in a coordinate system of the three-dimensional image dataset; transforming the first vector of the registered virtual camera by at least one transformation matrix to obtain a first transformation vector of an unregistered virtual camera located in the coordinate system of the three-dimensional image dataset; defining a focal point of the unregistered virtual camera on a reference point of the two-dimensional image dataset in the coordinate system of the three-dimensional image dataset; and repositioning the unregistered virtual camera based on the first transformation vector and the focal point of the unregistered virtual camera.

34. The processor retrieves the instructions from the memory, obtaining first spatial parameters of a first registered virtual camera positioned corresponding to a first two-dimensional image of the two-dimensional image dataset; 20. The navigation system of claim 18, adapted to perform a step of adjusting a second spatial parameter of a first unregistered virtual camera with the first spatial parameter of the first registered virtual camera, the first unregistered virtual camera being unable to be positioned relative to a second two-dimensional image of the two-dimensional image dataset.

35. 1. A method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, comprising the steps of: calibrating a first virtual camera based on distance parameters calculated based on the two-dimensional image data set and the region of interest; rotating the first virtual camera based on an angular difference between a first vector calculated from two spatial marks in the three-dimensional image data set and a second vector calculated from two first planar marks in the two-dimensional image data set; rotating the first virtual camera based on an angle, the angle corresponding to an angle difference between a reconstructed image having a maximum similarity parameter and the two-dimensional image dataset, the reconstructed image with the maximum similarity parameter having a maximum similarity parameter among a plurality of similarity parameters calculated for a plurality of reconstructed images in the three-dimensional image dataset, the plurality of reconstructed images including a reconstructed image generated by the first virtual camera and another reconstructed image generated by another virtual camera, the other reconstructed image and the reconstructed image generated by the first virtual camera having different angles or pixels; and after the alignment and rotation, registering the two-dimensional image dataset and the three-dimensional image dataset using the method in a navigation system; The method includes:

36. The distance parameter is calculated based on the two first planar markers in the two-dimensional image data set, the method comprising:

36. The method of claim 35, further comprising adjusting the reconstructed image based on the distance parameter, the reconstructed image being generated by the first virtual camera.

37. 36. The method of claim 35, wherein the distance parameter is calculated based on a distance between an estimated location of a launcher and an estimated location of the site of interest.

38. 38. The method of claim 37, wherein the estimated position of the launcher is calculated based on the two-dimensional image data set, and the estimated position of the site of interest is located at a closest point between a first virtual line and a second virtual line.

39. 39. The method of claim 38, wherein the two-dimensional image dataset includes a first two-dimensional image and a second two-dimensional image, and when capturing the first two-dimensional image dataset, the first virtual line is generated between the estimated position of the launcher and a center point of at least two reflectors, and when capturing the second two-dimensional image dataset, the second virtual line is generated between the estimated position of the launcher and the center point of the at least two reflectors.

40. 36. The method of claim 35, wherein the two-dimensional image dataset includes a first two-dimensional image, the first vector is calculated from the two spatial markers in the three-dimensional image dataset, the two spatial markers being generated by the first virtual camera, and the second vector is calculated from the two first planar markers in the first two-dimensional image data.

41. 41. The method of claim 40, wherein the two-dimensional image data set includes a second two-dimensional image, and the second vector is calculated from two second planar markers in the second two-dimensional image data.

42. A calibrated first virtual camera is generated after rotation of the first virtual camera, and the method further comprises:

36. The method of claim 35, further comprising: rotating the adjusted first virtual camera based on the angle, the angle corresponding to an angular difference between a reconstructed image having an adjusted maximum similarity parameter and the two-dimensional image dataset, the reconstructed image with the adjusted maximum similarity parameter having a maximum similarity parameter among a plurality of similarity parameters calculated for the adjusted plurality of reconstructed images, the adjusted plurality of reconstructed images including a reconstructed image generated by the adjusted first virtual camera and another reconstructed image generated by another virtual camera, and the other reconstructed image and the reconstructed image generated by the adjusted first virtual camera have different angles.

43. The method of claim 35, wherein the adjusted maximum similarity parameter reconstructed image has a maximum similarity parameter among a plurality of similarity parameters calculated for a plurality of adjusted reconstructed images, the plurality of adjusted reconstructed images including a reconstructed image generated by the adjusted first virtual camera and another reconstructed image generated by another virtual camera, and the other reconstructed image and the reconstructed image generated by the adjusted first virtual camera have different angles or pixels.

44. 36. The method of claim 35, further comprising adjusting the first virtual camera of a plurality of virtual cameras based on a matrix corresponding to the two-dimensional image dataset, the plurality of virtual cameras corresponding to the three-dimensional image dataset.

45. 1. A navigation system for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, comprising: a memory for storing a plurality of instructions; retrieving the instructions by the memory; calibrating a first virtual camera based on distance parameters calculated based on the two-dimensional image data set and the region of interest; rotating the first virtual camera based on an angular difference between a first vector calculated from two spatial marks in the three-dimensional image data set and a second vector calculated from two first planar marks in the two-dimensional image data set; and a step of rotating the first virtual camera based on an angle, the angle corresponding to an angular difference between a reconstructed image having a maximum similarity parameter and the two-dimensional image dataset, the reconstructed image with the maximum similarity parameter having a maximum similarity parameter among a plurality of similarity parameters calculated for a plurality of reconstructed images in the three-dimensional image dataset, the plurality of reconstructed images including a reconstructed image generated by the first virtual camera and another reconstructed image generated by another virtual camera, the other reconstructed image and the reconstructed image generated by the first virtual camera having different angles or pixels. After alignment and rotation, registering the two-dimensional image dataset and the three-dimensional image dataset using the method in a navigation system.

46. The distance parameter is calculated based on the two first planar markers in the two-dimensional image dataset, and the processor further comprises:

46. ​​A navigation system according to claim 45, adapted to perform the step of adjusting the reconstructed image based on the distance parameter, the reconstructed image being generated by the first virtual camera.

47. 46. ​​The navigation system of claim 45, wherein the distance parameter is calculated based on a distance between an estimated location of a launcher and an estimated location of the site of interest.

48. 48. The navigation system of claim 47, wherein the estimated position of the launcher is calculated based on the two-dimensional image data set, and the estimated position of the area of ​​interest is located at a closest point between a first virtual line and a second virtual line.

49. 49. The navigation system of claim 48, wherein the two-dimensional image dataset includes a first two-dimensional image and a second two-dimensional image, and when capturing the first two-dimensional image dataset, the first virtual line is generated between the estimated position of the launcher and a center point of at least two reflectors, and when capturing the second two-dimensional image dataset, the second virtual line is generated between the estimated position of the launcher and the center point of the at least two reflectors.

50. The navigation system of claim 451, wherein the two-dimensional image dataset includes a first two-dimensional image, the first vector is calculated from the two spatial markers in the three-dimensional image dataset, the two spatial markers being generated by the first virtual camera, and the second vector is calculated from the two first planar markers in the first two-dimensional image data.

51. 51. The navigation system of claim 50, wherein the two-dimensional image data set includes a second two-dimensional image, and the second vector is calculated from two second planar markers in the second two-dimensional image data.

52. A calibrated first virtual camera is generated after rotation of the first virtual camera, and the method further comprises:

46. ​​The navigation system of claim 45, further comprising a step of rotating the adjusted first virtual camera based on the angle, the angle corresponding to an angular difference between a reconstructed image having an adjusted maximum similarity parameter and the two-dimensional image dataset, the reconstructed image with the adjusted maximum similarity parameter having a maximum similarity parameter among a plurality of similarity parameters calculated for the adjusted plurality of reconstructed images, the adjusted plurality of reconstructed images including a reconstructed image generated by the adjusted first virtual camera and another reconstructed image generated by another virtual camera, and the other reconstructed image and the reconstructed image generated by the adjusted first virtual camera have different angles.

53. The navigation system of claim 45, wherein the adjusted maximum similarity parameter reconstructed image has a maximum similarity parameter among a plurality of similarity parameters calculated for a plurality of adjusted reconstructed images, the adjusted plurality of reconstructed images including a reconstructed image generated by the adjusted first virtual camera and another reconstructed image generated by another virtual camera, and the other reconstructed image and the reconstructed image generated by the adjusted first virtual camera have different angles or pixels.

54. The processor further comprises:

46. ​​The navigation system of claim 45, adapted to perform a step of adjusting the first virtual camera of a plurality of virtual cameras based on a matrix corresponding to the two-dimensional image dataset, the plurality of virtual cameras corresponding to the three-dimensional image dataset.

55. 1. A method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, comprising the steps of: defining a first vector of a registered virtual camera in a coordinate system of the three-dimensional image dataset; transforming the first vector of the registered virtual camera by at least one transformation matrix to obtain a first transformation vector of an unregistered virtual camera in the coordinate system of the three-dimensional image dataset; defining, in the coordinate system of the three-dimensional image dataset, a focal point of the unregistered virtual camera located at a reference point of the two-dimensional image dataset; generating an updated reconstructed image after repositioning the unregistered virtual camera by repositioning the unregistered virtual camera based on the first transformation vector and the focal point of the unregistered virtual camera; The method includes:

56. 56. The method of claim 55, wherein the first vector is from a position of the registered virtual camera to the focal point of the registered virtual camera.

57. 56. The method of claim 55, wherein the first vector is defined as the Z axis of the registered virtual camera in the coordinate system of the three-dimensional image dataset.

58. The method comprises: defining a second vector of the registered virtual camera in the coordinate system of the 3D image dataset; obtaining a second transformation vector of the unregistered virtual camera based on the second vector of the registered virtual camera in the coordinate system of the three-dimensional image data set; 56. The method of claim 55, comprising:

59. 58. The method of claim 57, wherein the second vector is directed from a center point to a top edge of the reconstructed image, the reconstructed image being acquired with the registered virtual camera parallel to the reconstructed image.

60. 59. The method of claim 58, wherein the second vector is defined as the Y axis of the registered virtual camera in the coordinate system of the three-dimensional image dataset.

61. 56. The method of claim 55, wherein the two-dimensional image dataset comprises a first two-dimensional image and a second two-dimensional image, and the at least one transformation matrix comprises a first matrix, the first matrix being used to transform the coordinate system of the first two-dimensional image to the coordinate system of the second two-dimensional image.

62. 56. The method of claim 55, wherein in the coordinate system of the three-dimensional image dataset, the reference point is located at a center point of a calibration module in the two-dimensional image dataset.

63. 63. The method of claim 62, wherein the at least one transformation matrix includes a second matrix and a third matrix, the second matrix being used to transform the coordinate system of the fiducial marker to the coordinate system of the three-dimensional image dataset, and the third matrix being used to transform the coordinate system of the fiducial marker to the coordinate system of a tracker module.

64. generating a first reconstructed image by the repositioned unregistered virtual camera corresponding to the three-dimensional image dataset based on first spatial parameters; calculating a reference similarity measure based on the first reconstructed image and the two-dimensional image data set; generating a second reconstructed image with the repositioned unregistered virtual camera corresponding to the three-dimensional image dataset based on second spatial parameters; calculating a comparative similarity measure based on the second reconstructed image and the two-dimensional image data set; comparing the comparison similarity value with the reference similarity value; 56. The method of claim 55, further comprising the step of: registering the two-dimensional image data set and the three-dimensional image data set if the comparison similarity measure is less than or equal to the reference similarity measure.

65. 1. A navigation system for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, comprising: a memory for storing a plurality of instructions; retrieving the instructions by the memory; defining a first vector of a registered virtual camera in a coordinate system of the three-dimensional image dataset; transforming the first vector of the registered virtual camera by at least one transformation matrix to obtain a first transformation vector of an unregistered virtual camera in the coordinate system of the three-dimensional image dataset; defining, in the coordinate system of the three-dimensional image dataset, a focal point of the unregistered virtual camera located at a reference point of the two-dimensional image dataset; and generating an updated reconstructed image after repositioning the unregistered virtual camera by repositioning the unregistered virtual camera based on the first transformation vector and the focal point of the unregistered virtual camera.

66. 66. The navigation system of claim 65, wherein the first vector is from a position of the registered virtual camera to the focal point of the registered virtual camera.

67. 66. The navigation system of claim 65, wherein the first vector is defined as the Z axis of the registered virtual camera in the coordinate system of the three-dimensional image data set.

68. The processor retrieves the instructions from the memory, defining a second vector of the registered virtual camera in the coordinate system of the 3D image dataset; and obtaining a second transformation vector of the unregistered virtual camera based on the second vector of the registered virtual camera in the coordinate system of the three-dimensional image dataset.

69. 68. The navigation system of claim 67, wherein the second vector is directed from a center point to an upper edge of the reconstructed image, the reconstructed image being acquired with the registered virtual camera parallel to the reconstructed image.

70. 69. The navigation system of claim 68, wherein the second vector is defined as the Y axis of the registered virtual camera in the coordinate system of the three-dimensional image data set.

71. 66. The navigation system of claim 65, wherein the two-dimensional image data set includes a first two-dimensional image and a second two-dimensional image, and the at least one transformation matrix includes a first matrix, the first matrix being used to transform the coordinate system of the first two-dimensional image to the coordinate system of the second two-dimensional image.

72. 66. The navigation system of claim 65, wherein in the coordinate system of the three-dimensional image dataset, the reference point is located at a center point of a calibration module in the two-dimensional image dataset.

73. 73. The navigation system of claim 72, wherein the at least one transformation matrix includes a second matrix and a third matrix, the second matrix being used to transform the coordinate system of the fiducial marker to the coordinate system of the three-dimensional image dataset, and the third matrix being used to transform the coordinate system of the fiducial marker to the coordinate system of a tracking device module.

74. The processor retrieves the instructions from the memory, generating a first reconstructed image by the repositioned unregistered virtual camera corresponding to the three-dimensional image dataset based on first spatial parameters; calculating a reference similarity measure based on the first reconstructed image and the two-dimensional image data set; generating a second reconstructed image with the repositioned unregistered virtual camera corresponding to the three-dimensional image dataset based on second spatial parameters; calculating a comparative similarity measure based on the second reconstructed image and the two-dimensional image data set; comparing the comparison similarity value with the reference similarity value; and if the comparison similarity score is less than or equal to the reference similarity score, registering the two-dimensional image data set and the three-dimensional image data set.

75. 1. A method for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, comprising the steps of: obtaining first spatial parameters of a first registered virtual camera, the setting position of the first registered virtual camera corresponding to a first two-dimensional image of the two-dimensional image data set; generating an updated reconstructed image by adjusting a second spatial parameter of a first unregistered virtual camera with the first spatial parameter of the first registered virtual camera to reposition the first unregistered virtual camera, where the repositioned position of the first unregistered virtual camera does not correspond to the first two-dimensional image of the two-dimensional image dataset; The method includes:

76. 76. The method of claim 75, wherein a first portion of the area of ​​interest is included in the first two-dimensional image, a second portion of the area of ​​interest is included in the first two-dimensional image, and the first and second portions of the area of ​​interest are adjacent to one another.

77. 77. The method of claim 76, further comprising defining the first portion of the area of ​​interest based on a first marker in the first two-dimensional image of the two-dimensional image dataset, and defining the second portion of the area of ​​interest based on a second marker in the first two-dimensional image of the two-dimensional image dataset.

78. 76. The method of claim 75, wherein the first spatial parameters of the first registered virtual camera include position and / or orientation data, the position and / or the orientation being defined to view a three-dimensional volume relative to the first registered virtual camera, or to view the first registered virtual camera relative to the three-dimensional volume.

79. 79. The method of claim 78, wherein the second spatial parameters of the first unregistered virtual camera include position and / or orientation data, the position and / or the orientation being defined to view a three-dimensional volume relative to the first unregistered virtual camera, or to view the first unregistered virtual camera relative to the three-dimensional volume.

80. 76. The method of claim 75, wherein the position of the first registered virtual camera is set based on a comparison result between a plurality of similarity values, the comparison result between the plurality of similarity values ​​being calculated based on a plurality of different reconstructed images and the first two-dimensional image of the two-dimensional image data, and the plurality of different reconstructed images are obtained from the three-dimensional image dataset.

81. 81. The method of claim 80, wherein each of the trivial similarity values ​​is calculated by a local normalized correlation (LNC), a sum of squared difference (SSD), a normalized cross-correlation (NCC), or a correlation ratio (CR).

82. The step of obtaining the first spatial parameters of the first registered virtual camera, the set position of which corresponds to the first two-dimensional image of the two-dimensional image dataset, comprises: defining the second spatial parameter of the first unregistered virtual camera as an N spatial parameter; determining whether a setup position of the first registered virtual camera corresponds to the first two-dimensional image of the two-dimensional image dataset, the first registered virtual camera having N-M spatial parameters, N and M being integers and M being less than N; defining the N-M spatial parameters as the first spatial parameters of the first registered virtual camera when a set position of the first registered virtual camera corresponds to the first two-dimensional image of the two-dimensional image dataset; 76. The method of claim 75, comprising:

83. The step of obtaining the first spatial parameters of the first registered virtual camera, the set position of which corresponds to the first two-dimensional image of the two-dimensional image dataset, comprises: defining the second spatial parameter of the first unregistered virtual camera as an N spatial parameter; determining whether a set position of the first registered virtual camera corresponds to the first two-dimensional image of the two-dimensional image dataset, the first registered virtual camera having N+M spatial parameters, N and M being integers and M being less than N; defining the N+M spatial parameters as the first spatial parameters of the first registered virtual camera when a setting position of the first registered virtual camera corresponds to the first two-dimensional image of the two-dimensional image dataset; 76. The method of claim 75, comprising:

84. generating a first reconstructed image with the calibrated first unregistered virtual camera, the first unregistered virtual camera being calibrated to correspond to the three-dimensional image dataset based on initial spatial parameters; calculating a reference similarity measure based on the first reconstructed image and the two-dimensional image data set; generating a second reconstructed image with the first unregistered virtual camera, the first unregistered virtual camera being calibrated to correspond to the three-dimensional image dataset based on the calibrated spatial parameters; calculating a comparative similarity measure based on the second reconstructed image and the two-dimensional image data set; comparing the comparison similarity value with the reference similarity value; 76. The method of claim 75, further comprising the step of: registering the two-dimensional image data set and the three-dimensional image data set if the comparison similarity measure is less than or equal to the reference similarity measure.

85. 1. A navigation system for registering a two-dimensional image dataset and a three-dimensional image dataset of a region of interest, comprising: a memory for storing a plurality of instructions; retrieving the instructions by the memory; obtaining first spatial parameters of a first registered virtual camera, the setting position of the first registered virtual camera corresponding to a first two-dimensional image of the two-dimensional image data set; a processor for performing a step of adjusting a second spatial parameter of a first unregistered virtual camera with the first spatial parameter of the first registered virtual camera to reposition the first unregistered virtual camera to generate an updated reconstructed image, where the repositioned position of the first unregistered virtual camera does not correspond to the first two-dimensional image of the two-dimensional image dataset; Includes a navigation system.

86. 86. The navigation system of claim 85, wherein a first portion of the area of ​​interest is included in the first two-dimensional image, a second portion of the area of ​​interest is included in the first two-dimensional image, and the first and second portions of the area of ​​interest are adjacent to one another.

87. 87. The navigation system of claim 86, wherein the first portion of the area of ​​interest is defined based on a first marker in the first two-dimensional image of the two-dimensional image dataset, and the second portion of the area of ​​interest is defined based on a second marker in the first two-dimensional image of the two-dimensional image dataset.

88. 86. The navigation system of claim 85, wherein the first spatial parameters of the first registered virtual camera include position and / or orientation data, the position and / or the orientation being defined to view a three-dimensional volume relative to the first registered virtual camera, or to view the first registered virtual camera relative to the three-dimensional volume.

89. 90. The navigation system of claim 88, wherein the second spatial parameters of the first unregistered virtual camera include position and / or orientation data, the position and / or the orientation being defined to view a three-dimensional volume relative to the first unregistered virtual camera, or to view the first unregistered virtual camera relative to the three-dimensional volume.

90. The navigation system of claim 85, wherein the position of the first registered virtual camera is set based on a comparison result between a plurality of similarity values, the comparison result between the plurality of similarity values ​​being calculated based on a plurality of different reconstructed images and the first two-dimensional image of the two-dimensional image data, and the plurality of different reconstructed images are obtained from the three-dimensional image data set.

91. 907. The navigation system of claim 906, wherein each of the trivial similarity values ​​is calculated by local normalized correlation (LNC), sum of squared difference (SSD), normalized cross-correlation (NCC) or correlation ratio (CR).

92. The step of obtaining the first spatial parameters of the first registered virtual camera, the setting position of which corresponds to the first two-dimensional image of the two-dimensional image dataset, is performed by the processor, the step comprising: defining the second spatial parameter of the first unregistered virtual camera as an N spatial parameter; determining whether a setup position of the first registered virtual camera corresponds to the first two-dimensional image of the two-dimensional image dataset, the first registered virtual camera having N-M spatial parameters, N and M being integers and M being less than N; defining the N-M spatial parameters as the first spatial parameters of the first registered virtual camera when a set position of the first registered virtual camera corresponds to the first two-dimensional image of the two-dimensional image dataset; 86. The navigation system of claim 85, comprising:

93. The step of obtaining the first spatial parameters of the first registered virtual camera, the setting position of which corresponds to the first two-dimensional image of the two-dimensional image dataset, is performed by the processor, the step comprising: defining the second spatial parameter of the first unregistered virtual camera as an N spatial parameter; determining whether a set position of the first registered virtual camera corresponds to the first two-dimensional image of the two-dimensional image dataset, the first registered virtual camera having N+M spatial parameters, N and M being integers and M being less than N; defining the N+M spatial parameters as the first spatial parameters of the first registered virtual camera when a setting position of the first registered virtual camera corresponds to the first two-dimensional image of the two-dimensional image dataset; 86. The navigation system of claim 85, comprising:

94. The processor retrieves the instructions from the memory, generating a first reconstructed image with the calibrated first unregistered virtual camera, the first unregistered virtual camera being calibrated to correspond to the three-dimensional image dataset based on initial spatial parameters; calculating a reference similarity measure based on the first reconstructed image and the two-dimensional image data set; generating a second reconstructed image with the first unregistered virtual camera, the first unregistered virtual camera being calibrated to correspond to the three-dimensional image dataset based on the calibrated spatial parameters; calculating a comparative similarity measure based on the second reconstructed image and the two-dimensional image data set; comparing the comparison similarity value with the reference similarity value; and if the comparison similarity score is less than or equal to the reference similarity score, registering the two-dimensional image data set and the three-dimensional image data set.

Citation Information

Patent Citations

  • A device for determining the position of a first object within a second object.

    JP2010510822A

  • Image alignment device, method and program

    JP2017164075A

  • Data processing device and data processing method

    JP2018068982A

  • Intraoperative 3D visualization with reduced radiation exposure

    JP2019500185A