Computerized tomography (CT) image correction using position and direction (P&D) tracking-assisted optical visualization

By using medical probes and sensor systems in surgical procedures, combined with wavefront sampling and passive stereo vision technology, the problem of difficulty in providing real three-dimensional observation in the existing technology is solved, efficient and low-cost three-dimensional imaging is achieved, and surgical surgery is improved operational accuracy and efficiency.

CN112867427BActive Publication Date: 2024-11-08BIOSENSE WEBSTER (ISRAEL) LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980065108.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-09-30
Filing Date
2019-10-01
Publication Date
2024-11-08
Estimated Expiration
2039-10-01

AI Technical Summary

Technical Problem

In surgical procedures, the prior art is difficult to provide real three-dimensional observations, resulting in surgeons increasing the time and possibility of errors in positioning tools during operation, and conventional 3D endoscopic devices are highly complex and cost-effective, and are not applicable in all procedures.

Method used

Using a system including medical probes, orientation and orientation sensors, cameras and processors, the system can be used to receive three-dimensional image signals and register with the sensor coordinate system in the image coordinate system, identify voxels in the cavity and update density values, and provide three-dimensional image display. Wavefront sampling and passive stereo vision technology generate 3D image data.

Benefits of technology

It realizes efficient three-dimensional imaging in surgical procedures, reduces equipment complexity and cost, is suitable for a variety of surgical procedures, and improves operating accuracy and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112867427B_ABST
    Figure CN112867427B_ABST
Patent Text Reader

Abstract

The present invention discloses a system (11) comprising a medical probe (36) for insertion into a cavity of an organ, the medical probe comprising a position and orientation sensor (60) and a camera (45), both of which operate in a sensor coordinate system (62). The system also comprises a processor (44) configured to: receive a three-dimensional image of the cavity including open space and tissue from an imaging system (21) operating in an image coordinate system (28); receive a signal indicating the position and corresponding orientation of the medical probe in the cavity from the medical probe; receive a corresponding visualization position in the cavity from the camera; align the image coordinate system with the sensor coordinate system so as to identify one or more voxels in the image at the visualization position, and when the identified voxel has a density value in the received image that does not correspond to the open space, update the density value of the identified voxel to correspond to the open space.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 741,403, filed on October 4, 2018, the disclosure of which is incorporated herein by reference. Technical Field

[0003] The present invention relates generally to medical imaging, and in particular to updating outdated medical images during the performance of an invasive medical procedure. Background Art

[0004] During a medical procedure, a real-time representation of an anatomical structure detected by a medical device (such as a probe tracked by an orientation tracking system) may be superimposed on a pre-acquired medical image to improve the anatomical structure representation. For example, U.S. Patent Application Publication 2018 / 0146883 describes methods, apparatus, and computer program products that include receiving a three-dimensional image of a body cavity including open space and body tissue from an imaging system operating in an image coordinate system, and receiving a signal indicating the position of a distal tip of the medical probe in a sensor coordinate system from a medical probe having a position sensor and inserted into the body cavity. The image coordinate system is aligned with the sensor coordinate system to identify one or more voxels in the three-dimensional image at the indicated position, and when the identified voxel has a density value in the received three-dimensional image that does not correspond to the open space, the density value of the identified voxel is updated to correspond to the open space.

[0005] As another example, U.S. Patent Application Publication No. 2014 / 0147027 describes an imaging correction system including a tracked ultrasound imaging probe configured to generate imaging volumes of a region of interest from different orientations. An image compensation module is configured to process image signals from an ultrasound imaging device associated with the probe, and compare one or more ultrasound image volumes with a reference such as a CT image, to determine distortion in ultrasound imaging using the image compensation module and generate a corrected ultrasound image for display based on compensation for the distortion.

[0006] Image guided surgery (IGS) such as the one described above is a technique in which a computer is used to obtain a real-time correlation of the position of an instrument that has been inserted into a patient's body with a set of images obtained before surgery (e.g., CT or MRI scans, 3D maps, etc.), so that the computer system can overlay the current position of the instrument on the images obtained before surgery. An example of an electromagnetic IGS navigation system that can be used in IGS surgery is the EM-1000 provided by Biosense-Webster, Inc. (Irvine, California). 3 systems.

[0007] In some IGS procedures, a digital tomographic scan (e.g., CT or MRI, 3D mapping, etc.) of the surgical area is obtained before the surgical procedure. The digital tomographic data is then converted into a digital mapping map using a specially programmed computer. During the surgical procedure, a special instrument with an orientation sensor (e.g., an electromagnetic coil that emits an electromagnetic field and / or responds to an externally generated electromagnetic field) is used to perform the procedure, and the sensor sends data to a computer indicating the current position of each surgical instrument. The computer associates the data received from the sensor with a digital map generated by the preoperative tomographic scan. The tomographic image is displayed on a video monitor together with an indicator (e.g., a crosshair or illuminated point, etc.), thereby showing the real-time position of each surgical instrument relative to the anatomical structure shown in the scanned image. Even if the surgeon cannot directly visually view the instrument itself at its current position in the body, the surgeon is able to understand the exact position of each sensor-equipped instrument by viewing the video monitor.

[0008] While the IGS navigation system provides useful views and information during surgical procedures, surgeons may also desire real-time photographs and videos of the anatomical structures being operated on. In such cases, the endoscope can be deployed to the surgical site with the aid of IGS navigation to acquire images of the anatomical structures; and can also be paired or otherwise deployed with other surgical instruments such as cutting instruments, ablation instruments, dilation catheters, etc. The photographic images and videos acquired in this manner may be more useful than the IGS navigation images, which may only provide generic and simulated images. Alternatively, the photographic images and videos acquired in this manner may provide a useful supplement to the IGS navigation images.

[0009] Compared to the surgeon's direct visual observation, the images collected by conventional endoscopes can also be limited to a certain extent because the images collected using conventional endoscopes can be limited to two-dimensional (2D) representations of anatomical structures. Operations such as cutting, ablation, dilation, etc. can be performed using a combination of 2D endoscopy and IGS navigation, neither of which can provide a sense of depth that can be used for true three-dimensional (3D) observation. Manipulating and operating surgical tools without the aid of 3D observation can increase the time spent positioning the tools; and can increase the possibility of errors. Although there are some 3D endoscopic cameras, they have high complexity and high cost; and can have rigid and non-flexible parts that make it difficult or impossible to use in some procedures. Specific implementations with both flexible and 3D features may require expensive and fragile fiber optic components to transmit the collected images from the distal camera to the proximal observer. Therefore, conventional options for 3D endoscopy may be expensive, have limited available features, and are not suitable for large-scale use and disposability.

[0010] While several systems and methods have been developed and used in surgical procedures, it is believed that the inventors have not previously developed or used the invention described in the appended claims. Summary of the invention

[0011] Embodiments of the present invention provide a system comprising a medical probe, a position and orientation sensor, a camera, and a processor. The medical probe is configured to be inserted into a cavity of an organ of a patient. The position and orientation sensor is located in the medical probe and operates in a sensor coordinate system. The camera is located in a distal edge of the medical probe and operates in a sensor coordinate system. The processor is configured to: (a) receive a three-dimensional image of the cavity including an open space and organ tissue from an imaging system operating in an image coordinate system; (b) receive a signal indicating the position and corresponding direction of the distal edge of the medical probe in the cavity from the medical probe; (c) receive a corresponding visualization position in the cavity from the camera of the probe; (d) align the image coordinate system with the sensor coordinate system so as to identify one or more voxels in the three-dimensional image at the visualization position; and (e) when the identified voxel has a density value that does not correspond to the open space in the received three-dimensional image, update the density value of the identified voxel to correspond to the open space.

[0012] In some embodiments, the imaging system comprises a computed tomography scanner.

[0013] In some embodiments, the position and direction sensor comprises a magnetic field sensor.

[0014] In one embodiment, the processor is configured to form a correspondence between the density values ​​and visual effects, wherein a given visual effect corresponds to a given density value indicating the open space. In another embodiment, the visual effect is selected from the group consisting of color, shading, and pattern.

[0015] In some embodiments, the processor is configured to present the three-dimensional image on a display using the visual effect. In other embodiments, the given visual effect comprises a first given visual effect, and wherein before updating the density value, the processor is configured to present the three-dimensional image by presenting the one or more identified voxels using a second given visual effect different from the first given visual effect.

[0016] In one embodiment, when updating the density value, the processor is configured to render the three-dimensional image by rendering the one or more identified voxels using the first given visual effect. In another embodiment, the processor is configured to render the three-dimensional image using a multi-viewing Figure 3The angular partitioning model extracts the distance of a location from the camera from the visual signal.

[0017] According to an embodiment of the present invention, a method is also provided, the method comprising receiving a three-dimensional image of a cavity of an organ of a patient including open space and organ tissue from an imaging system operating in an image coordinate system. A signal indicating the position and corresponding direction of a distal edge of the medical probe within the cavity is received from a medical probe having an orientation and direction sensor and a camera, wherein the probe is operated in a sensor coordinate system and inserted into the cavity. The image coordinate system is aligned with the sensor coordinate system so as to identify one or more voxels in the three-dimensional image at the visualization location. When the identified voxel has a density value that does not correspond to the open space in the received three-dimensional image, the density value of the identified voxel is updated to correspond to the open space.

[0018] According to an embodiment of the present invention, there is also provided a computer software product operating in conjunction with a probe, the probe being configured for insertion into a cavity of an organ of a patient and comprising a position and orientation sensor operating in a sensor coordinate system and a camera located in a distal edge of the medical probe operating in the sensor coordinate system, and the product comprising a non-transitory computer-readable medium having program instructions stored therein, the instructions, when read by a computer, causing the computer to: (a) receive a three-dimensional image of the cavity comprising an open space and organ tissue from an imaging system operating in an image coordinate system; (b) receive a signal from the medical probe indicating the position and corresponding orientation of the distal edge of the medical probe within the cavity; (c) receive a corresponding visualization position of the wall of the cavity; (d) align the image coordinate system with the sensor coordinate system so as to identify one or more voxels in the three-dimensional image at the visualization position; and (e) when the identified voxel has a density value in the received three-dimensional image that does not correspond to the open space, update the density value of the identified voxel to correspond to the open space.

[0019] According to an embodiment of the present invention, there is also provided a three-dimensional (3D) imaging system comprising an endoscope and a processor. The endoscope comprises: (i) a shaft having a distal end, the shaft being adapted to be inserted into a patient and positioned at a surgical site of the patient; (ii) an orientation sensor, the orientation sensor being proximal to the distal end and being configured to generate an orientation signal set based on the position of the endoscope during use; and (iii) an imaging module, the imaging module being positioned at the distal end and being operable to acquire an image data set of the surgical site, wherein the image data set comprises one or more two-dimensional (2D) images. The processor is communicatively coupled to the endoscope and configured to: (i) receive the image dataset and the orientation signal set from the endoscope; (ii) determine a perspective dataset based on the orientation signal set, wherein the perspective dataset indicates the position of the endoscope during acquisition of each of the one or more 2D images; (iii) perform image depth analysis to determine a 3D feature set for each of the one or more 2D images, wherein the 3D feature set includes pixel depth; (iv) create a 3D image dataset based on the one or more 2D images and the 3D feature set; and (v) associate the perspective dataset with the 3D image dataset, wherein the image depth analysis includes a technique selected from the group consisting of: (i) a wavefront sampling technique performed on a single image of the one or more 2D images; and (ii) a passive stereoscopic vision technique performed on a set of two images of the one or more 2D images, wherein each image in the set of two images is associated with the same perspective data.

[0020] In some embodiments, the imaging module includes: (i) a single lens; (ii) an aperture plate positioned between a first side of the single lens and the surgical site, the aperture plate including one or more apertures offset from the optical axis of the single lens; and (iii) an image pane positioned on a second side of the single lens to receive reflected light from the surgical site via the one or more apertures and the single lens, wherein the image pane is configured to generate the image dataset based on the reflected light.

[0021] In some embodiments, (i) the one or more apertures include at least two apertures positioned on the aperture plate and offset from the optical axis of the single lens, and (ii) the aperture plate has a fixed position and orientation relative to the lens.

[0022] In one embodiment, (i) the one or more apertures include a single aperture positioned on the aperture plate offset from the optical axis of the single lens, and (ii) the aperture plate is operable to rotate about its circular axis relative to the lens during image acquisition.

[0023] In another embodiment, the processor is configured to, when performing the image depth analysis: (i) identify two or more unfocused images of the surgical site within the image dataset; (ii) determine a spatial relationship between the two or more unfocused images of the surgical site; and (iii) determine the pixel depth in the image dataset based on the spatial relationship between the two or more unfocused images.

[0024] In some embodiments, the imaging module includes two or more cameras, and wherein each of the two or more cameras: (i) is statically positioned relative to every other camera of the two or more cameras; and (ii) is oriented to have a parallel optical axis with every other camera of the two or more cameras.

[0025] In some embodiments, the processor is further configured to, when performing the image depth analysis, (i) identify a point in a first image in the image dataset, wherein the point includes a portion of the surgical site that is present in both the first image acquired by a first camera among the two or more cameras and a second image acquired by a second camera among the two or more cameras; (ii) identify the point in the second image; (iii) determine a displacement of the point from the first image to the second image; and (iv) determine the pixel depth of the point based on the displacement.

[0026] In one embodiment, the processor is also configured to, when identifying the point in the second image: (i) determine the epipolar line of the first image and the second image based on the static position of the first camera relative to the second camera; and (ii) search for the point in the second image along the epipolar line while excluding portions of the second image that do not fall on the epipolar line.

[0027] In another embodiment, the processor is further configured to: (i) associate the 3D image dataset and the perspective dataset with a coordinate system of an image-guided surgery system; and (ii) display the 3D image dataset during an image-guided surgery navigation procedure based on the association with the coordinate system.

[0028] In one embodiment, (i) the orientation sensor is configured to generate the orientation signal set based on the position and orientation of the endoscope during use, (ii) the perspective dataset indicates the position and orientation of the endoscope during acquisition of the image dataset, and (iii) the processor is further configured to provide the 3D image dataset and the perspective dataset to an image-guided surgical navigation system.

[0029] In another embodiment, the processor is further configured to: (i) receive input from a user defining a viewing angle relative to the surgical site; (ii) determine a first portion of the 3D image dataset showing the surgical site from the viewing angle based on identifying the viewing angle within the viewing angle dataset; and (iii) display the first portion of the 3D image dataset on a display.

[0030] In another embodiment, the processor is further configured to: (i) receive an indirect 3D scan of the surgical site and a scanning perspective dataset associated with the indirect 3D scan; (ii) determine a second portion of the indirect 3D scan showing the surgical site from the perspective based on identifying the perspective within the scanning perspective dataset; and (iii) simultaneously display the first portion of the 3D image dataset and the second portion of the indirect 3D scan on the display.

[0031] In another embodiment, (i) the indirect 3D scan of the surgical site includes image data acquired pre-operatively, and (ii) the 3D image dataset includes image data acquired post-operatively.

[0032] In one embodiment, (i) the indirect 3D scan of the surgical site includes image data acquired before surgery, (ii) the 3D image dataset includes image data acquired before surgery, and (iii) the processor is further configured to: (A) receive scan adjustment input from a user; and (B) reconfigure the association between the indirect 3D scan of the surgical site and the scan view dataset based on the scan adjustment input.

[0033] According to an embodiment of the present invention, there is also provided a method for three-dimensional (3D) imaging, the method comprising deploying a distal end of an endoscope to a surgical site of a patient, the distal end comprising: (i) an imaging module operable to acquire image data of the surgical site, wherein the acquired image data comprises one or more two-dimensional (2D) images; and (ii) an orientation sensor proximate to the distal end and configured to generate an orientation signal based on the position of the endoscope. An image data set from the imaging module and an orientation signal set from the orientation sensor are received. A view data set is determined based on the orientation signal set, wherein the view data set indicates the position of the endoscope during acquisition of each of the one or more 2D images. Image depth analysis is performed to determine a 3D feature set for each of the one or more 2D images, wherein the 3D feature set comprises pixel depth. A 3D image dataset is created based on the one or more 2D images and the 3D feature set, and the perspective dataset is associated with the 3D image dataset, wherein the image depth analysis includes a technique selected from the group consisting of: (i) a wavefront sampling technique performed on a single image of the one or more 2D images; and (ii) a passive stereoscopic vision technique performed on a set of two images of the one or more 2D images, wherein each image in the set of two images is associated with the same perspective data.

[0034] Another embodiment of the present invention provides an image-guided surgery (IGS) navigation system, which includes a processor, a memory and a display, and the processor is configured to: (a) receive an image dataset generated by a tracking endoscope, the image dataset including one or more two-dimensional (2D) images; (b) receive a perspective dataset generated by the tracking endoscope, wherein the perspective dataset indicates the position of the tracking endoscope during the acquisition of the image dataset; (c) perform image depth analysis to determine a 3D feature set of the image dataset, wherein the 3D feature set includes pixel depths in the one or more 2D images; (d) create a 3D image dataset based on the one or more 2D images and the 3D feature set; (e) associate the perspective dataset with the 3D image dataset; and (f) cause the display to display the 3D image dataset from the selected perspective based on the perspective dataset including the selected perspective.

[0035] The present invention will be more fully understood through the following detailed description of embodiments of the present invention in conjunction with the accompanying drawings, in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1A and Figure 1B(generally referred to as FIG. 1 ) is a schematic illustration of a medical system configured to correct outdated computed tomography (CT) images during performance of an invasive medical procedure in accordance with an embodiment of the present invention;

[0037] Figure 2 A flowchart schematically illustrating a method for correcting an outdated CT image according to an embodiment of the present invention;

[0038] Figure 3 A schematic illustration showing an image slice of a three-dimensional CT image according to an embodiment of the present invention;

[0039] Figure 4 A schematic detailed view showing a distal end of a probe inserted into a sinus cavity according to an embodiment of the present invention;

[0040] Figure 5 To illustrate the process before updating the CT image according to an embodiment of the present invention, Figure 3 Schematic illustration of the region of interest of a CT image slice;

[0041] Figure 6 To illustrate the embodiment of the present invention after updating the CT image Figure 5 Schematic illustration of the region of interest of a CT image slice;

[0042] Figure 7 shows a schematic diagram of an exemplary surgical navigation system in use on a patient seated in an exemplary medical procedure chair in accordance with another embodiment of the present invention;

[0043] Figure 8 It shows that the embodiment according to the present invention can be used with Figure 7 A perspective view of an exemplary instrument for use with a surgical navigation system;

[0044] Fig. 9A A method for producing a single imaging device according to an embodiment of the present invention is shown. Figure 8 A perspective view of an exemplary distal tip of an instrument of

[0045] Fig. 9B The embodiment according to the present invention is shown Fig. 9A A front view of the distal end of

[0046] Fig. 10A The embodiment according to the present invention is shown Fig. 9A a perspective view of the distal tip of the device with the irrigation diverter removed to illustrate the irrigation channel;

[0047] Fig. 10B The embodiment according to the present invention is shown Fig. 9AA front view of the distal tip of the device with the irrigation diverter removed to illustrate the irrigation channel;

[0048] Fig.11 The invention is shown for use in Fig. 9A A schematic diagram of an exemplary static sampling aperture configuration of an imaging device;

[0049] Fig.12 Shown for Fig. 9A A schematic diagram of an exemplary active sampling aperture configuration of an imaging device;

[0050] Fig.13 shows a method that can be performed to utilize Fig. 9A A set of exemplary steps for generating a three-dimensional image model by an imaging device;

[0051] Fig.14A A device having a set of imaging devices is shown. Figure 8 A perspective view of an exemplary alternative distal tip of an instrument of;

[0052] Fig. 14B The embodiment according to the present invention is shown Fig.14A A front view of the distal end of

[0053] Fig.15A The embodiment according to the present invention is shown Fig.14A A side diagrammatic view of an exemplary set of overlapping fields of view of a set of imaging devices;

[0054] Fig. 15B The embodiment according to the present invention is shown Fig.14A A set of top view solution diagrams of overlapping fields of view;

[0055] Fig.16 shows a method that can be performed to utilize Fig.14A A set of exemplary steps for generating a three-dimensional image model by a set of imaging devices;

[0056] Fig.17 shows a diagram illustrating the relationship between disparity and depth in stereoscopic imaging;

[0057] Fig.18 shows a diagram illustrating epipolar lines in stereoscopic imaging according to an embodiment of the present invention; and

[0058] Fig.19 shows that can be performed to utilize Figure 7 A flow chart of a set of steps for a surgical navigation system to generate and use a three-dimensional image model.

[0059] The drawings are not intended to be limiting in any way, and it is contemplated that various embodiments of the invention may be performed in a variety of other ways, including those not necessarily shown in the drawings. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several aspects of the invention and, together with the description, serve to explain the principles of the invention; however, it should be understood that the invention is not limited to the precise arrangements shown. DETAILED DESCRIPTION

[0060] Overview

[0061] According to one aspect of the present invention, intraoperative medical imaging systems such as computed tomography (CT) imaging systems are modalities that are not typically available in clinics performing ear, nose and throat (ENT) procedures. Therefore, ENT physicians performing the procedures typically rely on previously acquired medical images, such as CT images. The CT images can be used to generate real-time guidance (i.e., "maps") that enable the physician performing the ENT medical procedure to navigate a medical probe within a volume of interest within a cavity in the patient's head.

[0062] However, in some cases, for example, during or after surgical ENT procedures such as shaving of obstructing cartilage or bone at the cavity wall, the CT images no longer reflect the true anatomy of the cavity.

[0063] Embodiments of the present invention described below provide systems and methods for updating outdated medical images, such as outdated CT images of a lumen of a patient's organ, which CT images are also referred to herein as three-dimensional images.

[0064] In some embodiments, during the update procedure, a camera located at the distal end of the medical probe and operating in the probe coordinate system visualizes the position within the cavity including open space and organ tissue. The processor estimates the position using the visualization from different positions and directions provided by the camera. The processor derives different positions and directions based on signals received from a position and direction (P&D) sensor mounted at the distal end of the medical probe.

[0065] Use Multiview Figure 3 The processor reconstructs the visual position by reconstructing the distance of the fail position from the camera. Figure 3 The triangulation model can be used, for example, with the name "triangulateMultiview". Function that returns the position of a 3D world point that corresponds to a point matched on multiple images taken with a camera calibrated with P&D.

[0066] In order for the CT image to reflect the most recently visualized (i.e., true) anatomy of the cavity, the processor registers the coordinate system associated with the CT imaging system with the coordinate system associated with the medical probe (e.g., with the sensor coordinate system). The registration enables the processor to identify one or more voxels in the three-dimensional image at the indicated location and when the identified voxels have a density value in the received three-dimensional image that does not correspond to a newly formed open space (e.g., formed by an ENT shaving procedure) (as visualized by the camera and as viewed by the processor using a multi-view image). Figure 3 When the open space is estimated by the angular partitioning model, the density values ​​of the identified voxels are updated to correspond to the new open space.

[0067] Medical imaging systems typically use different visual effects to present open space and body tissue. For example, a CT system may present open space in black, hard body tissue in white, and soft body tissue in gray (various shades). Since a camera located at the distal edge of the medical probe may optically image wall tissue only through open space, a system implementing an embodiment of the present invention may correct any optical imaging position of the cavity that is not presented as open space in an updated three-dimensional (3D) CT image. In addition, if the camera has entered a position that was not previously open space, embodiments of the present invention may also present the entered position as open space in an updated 3D image.

[0068] Other aspects of the disclosed invention provide imaging techniques, such as wavefront sampling and passive stereoscopic vision, for generating 3D image data and 3D models from single-camera and multi-camera 2D imaging systems. Such techniques are used to generate 3D functional imaging systems with small size and low complexity compared to 3D imaging systems with independent dynamic focus cameras. Such imaging techniques (e.g., systems) are implemented at the distal end of surgical instruments with static or flexible shafts, with reduced size and power requirements, and can be used to generate 3D imaging of surgical sites within the human body. Such 3D imaging can be used to generate synthetic image sets that can be used during IGS navigation to provide comparisons of pre-operative image sets and to provide comparisons of post-operative image sets.

[0069] In some embodiments, a 3D endoscopic imaging system is provided, the system comprising: (a) an endoscope comprising: a position sensor; and an imaging module positioned at a distal end of the endoscope and operable to acquire an image dataset of a surgical site, wherein the image dataset comprises one or more two-dimensional (2D) images; and (b) a processor communicatively coupled to the endoscope and configured to:

[0070] (i) receiving an image data set and a position signal set from an endoscope,

[0071] (ii) determining a view angle dataset based on the set of orientation signals, wherein the view angle dataset indicates a position of the endoscope during acquisition of each of the one or more 2D images,

[0072] (iii) performing image depth analysis to determine a 3D feature set for each of the one or more 2D images, wherein the 3D feature set includes pixel depth,

[0073] (iv) creating a 3D image dataset based on the one or more 2D images and the 3D feature set, and

[0074] (v) associating the viewpoint dataset with the 3D image dataset;

[0075] Wherein the image depth analysis comprises a technique selected from the group consisting of:

[0076] (i) a wavefront sampling technique performed on a single image of the one or more 2D images, and

[0077] (ii) Passive stereoscopic techniques performed on a set of two images of one or more 2D images, wherein each image in the set of two images is associated with the same viewing angle data.

[0078] Embodiments of the present invention disclose medical image (such as CT image) correction using P&D tracking-assisted visualization techniques of cavity position.Use of these embodiments may eliminate the need for repeated imaging procedures such as CT imaging procedures, which typically require large and expensive operating facilities.

[0079] CT image correction using optical visualization assisted by P&D tracking

[0080] The following description of certain examples of the present invention should not be used to limit the scope of the present invention. According to the following description shown by way of example, other examples, features, aspects, embodiments and advantages of the present invention will be apparent to those skilled in the art, and a best mode is contemplated for implementing the present invention. As will be appreciated, the present invention can have other different and obvious aspects, all of which do not depart from the present invention. Therefore, the drawings and description should be considered to be illustrative and non-restrictive in nature.

[0081] It should be understood that the terms "proximal" and "distal" as used herein are relative to a clinician holding the handpiece assembly. Thus, the end effector is distal relative to the closer handpiece assembly. It should also be understood that for convenience and clarity, spatial terms such as "top" and "bottom" are also used herein with respect to the situation where the clinician holds the handpiece assembly. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be limiting and absolute.

[0082] It should also be understood that any one or more of the teachings, expressions, versions, examples, etc. described herein may be combined with any one or more of the other teachings, expressions, versions, examples, etc. described herein. Therefore, the following teachings, expressions, versions, examples, etc. should not be considered separate from each other. With reference to the teachings herein, various suitable ways in which the teachings herein may be combined will be apparent to those skilled in the art. Such modifications and variations are intended to be included within the scope of the claims.

[0083] Figure 1A and Figure 1B (collectively referred to as FIG. 1 ) is a schematic illustration of a medical system 11 configured to correct outdated computed tomography (CT) images during the performance of an invasive medical procedure according to an embodiment of the present invention. In the example shown in FIG. 1 , the medical system 11 includes a medical imaging system including a mobile CT scanner 21, a console 34, and a medical probe 36. In the embodiments described herein, it is assumed that the medical probe 36 is used for ENT diagnostic or therapeutic treatment, such as minimally invasive catheter-based sinus surgery performed on a patient 26. Alternatively, with necessary changes, the medical probe 36 can be used for other therapeutic and / or diagnostic purposes.

[0084] like Figure 1A As shown, before an invasive medical procedure is performed on a patient 26, a CT scanner 21 generates an electrical signal including image data of an internal cavity of the patient (e.g., a nasal cavity or a paranasal sinus) and transmits the generated image data to a console 34. The computed tomography scanner 21 generates image data in an image coordinate system 28 including an X-axis 31, a Y-axis 33, and a Z-axis 35.

[0085] like Figure 1B As shown, the medical probe 36 includes a handle 38 that can be grasped and manipulated by an operator 43 to insert a distal end 41 of the medical probe into an internal cavity of the patient 26, such as a nasal cavity or a paranasal sinus.

[0086] The distal end 41 includes a camera 45 that generates image data in response to a scene viewed by the camera. The distal end 41 also includes a magnetic field sensor 60 that generates a signal providing the position and orientation of the distal end as described below. In the configuration shown in FIG. 1 , the console 34 includes a processor 44 that converts the image data received from the camera 45 into an image 46 and presents the image on a display 48 as information about a medical procedure.

[0087] Based on signals received from the medical probe 36 and other components of the medical system 20, the console 34 drives the display 48 to update the image 46, thereby presenting the current position of the distal end 41 within the patient's head and status information and guidance about the ongoing medical procedure. The processor 44 stores data representing the image 46 in the memory 50. In some embodiments, the operator 40 can manipulate the image 46 using one or more input devices 52.

[0088] The processor 44 typically includes a general-purpose computer having a suitable front end and interface circuits for receiving signals from the medical probe 36 and controlling other components of the console 34. The processor 44 can be programmed by software to perform the functions described herein. The software can be downloaded to the console 34 in electronic form, for example, via a network, or it can be provided on a non-transient tangible medium such as an optical storage medium, a magnetic storage medium, or an electronic storage medium. Alternatively, some or all of the functions of the processor 44 can be performed by dedicated or programmable digital hardware components.

[0089] In the embodiment described herein, the medical system 11 uses magnetic position sensing to determine the position and direction coordinates of the distal end 41 of the medical probe 36 in the patient 26. In order to achieve magnetic-based position and direction sensing, the console 34 includes a drive circuit 56 for driving a field generator 58 to generate a magnetic field in the detection organ of the patient 26. Typically, the field generator 58 includes a coil that is placed below the patient at a known position outside the patient 26. These coils generate a magnetic field in a predetermined working space containing an inner cavity such as a paranasal sinus. The magnetic field sensor 60 (also referred to herein as a position and direction (P&D) sensor 60) in the distal end 41 of the medical probe 36 generates an electrical signal in response to the magnetic field from the coil, thereby enabling the processor 44 to determine the position and direction of the distal end 41 in the working volume.

[0090] Magnetic orientation tracking techniques are described, for example, in U.S. Patents 5,391,199, 6,690,963, 5,443,489, 6,788,967, 5,558,091, 6,172,499, and 6,177,792, the disclosures of which are incorporated herein by reference. The signal generated by the magnetic field sensor 60 indicates the current position of the distal end 41 in a sensor coordinate system 62 defined by the orientation of the generator 58, and the system 62 is assumed to include an X-axis 64, a Y-axis 66, and a Z-axis 68. In the example shown in FIG. 1 , the X-axis 64 corresponds approximately to the X-axis 31, the Y-axis 66 corresponds approximately to the Y-axis 33, and the Z-axis 68 corresponds approximately to the Z-axis 35. Such systems and techniques may be used in conjunction with other aspects of the present invention (in conjunction with Figure 7 The systems and techniques described herein are similar or identical to those described herein.

[0091] In the embodiments described herein, the medical system 11 uses a camera 45 to image a portion of a cavity. As described above, the sensor 60 is used to derive the position of the camera 45 that is imaging the cavity and the corresponding direction that the camera is aiming. Figure 3 angular partitioning model, the processor 44 is capable of determining (i) the position of the portion of the wall of the cavity in which the distal end is disposed, and (ii) the position between imaging positions on the cavity wall and cameras (including the camera 45 itself), all of which are located in open space; and is capable of updating the CT image accordingly.

[0092] Figure 2 A flowchart schematically illustrating a method for correcting an outdated CT image, and Figure 3 , Figure 4 , Figure 5 and 6 Schematic diagram of a method according to an embodiment of the present invention. In acquisition step 80, processor 44 acquires image data from CT scanner 21, stores the image data in memory 50, and generates a three-dimensional image based on the acquired image data. Step 80 is usually performed before performing an update CT image procedure on patient 26.

[0093] Figure 3 8 is a schematic illustration of an image slice 101 of a three-dimensional computed tomography image that may be acquired in step 80. Processor 44 generates image slice 101 in response to receiving image data from CT scanner 21. In operation, processor 44 typically incorporates image slice 101 into image 46.

[0094] exist Figure 3 In the example shown, the image slice 101 includes a two-dimensional "slice" of a head 103 of the patient 26. As shown in the legend 106, the image slice 101 includes voxels 105 of the three-dimensional image corresponding to three-dimensional locations in the computed tomography image of the head 103. The processor 44 typically presents each given voxel 105 using a different visual effect corresponding to the density detected at the three-dimensional location corresponding to the given voxel. Examples of density include, but are not limited to, open space, hard organ tissue, and soft organ tissue, and examples of visual effects include, but are not limited to, color, shading (e.g., different shades of gray), and patterns (e.g., gradients, pictures, textures, lines, dots, and boxes).

[0095] As shown in legend 107, voxels 105 may be distinguished by appending letters to the identification numbers, such that the voxels include voxels 105A-105C. Figure 3In the example shown, voxel 105A indicates open space and is rendered in black, voxel 105B indicates hard organ tissue (e.g., bone) and is rendered in white, and voxel 105C indicates soft tissue (e.g., fat, muscle cartilage, and brain tissue) and is rendered in gray. Although the embodiments herein describe an image slice 101 including voxels 105 (i.e., voxels 105A-105C) having three different visual effects, it is considered within the spirit and scope of the present invention to render voxels having any number of visual effects representing any number of densities.

[0096] The image slice 101 also includes a region of interest 109. Figure 5 and Figure 6 As described in the description of , the region of interest 109 includes the location of the cavity wall 47 having an outdated CT value (ie, density value), and the processor 44 can update the outdated CT value using the embodiments described herein.

[0097] Returning to the flow chart, to initiate the update procedure, operator 43 manipulates handle 38 such that, in an insertion step 82, distal end 41 enters the cavity of an organ of patient 26. It should be appreciated that there is typically a delay between steps 80 and 82, which may be days or even weeks.

[0098] Next, in a registration step 84, the processor 44 registers the image coordinate system 28 with the sensor 60 coordinate system 62. The registration may be performed by any suitable method known in the art. When performing the registration, each three-dimensional position indicated by the signal generated by the magnetic field sensor 60 corresponds to one or more voxels in the CT image acquired in step 80.

[0099] In a receive visual signal step 86, processor 44 receives a visual signal from camera 45 indicating the position of cavity wall 47, and in an identification step 88, processor 44 uses the registration and multi-viewing Figure 3 The angular partitioning model is used to identify one or more voxels in the CT image corresponding to the indicated location.

[0100] Figure 4 FIG. 4 is a schematic detail diagram showing the distal end 41 inserted into the sinus cavity 111. As the distal end moves, the camera 45 in the distal end 41 collects images of the cavity wall 47 from different viewing angles. Based on the visual signals of the wall 47 collected by the camera 45 and as described above, the multi-view Figure 3 Angular partitioning model, processor 44 estimates location 47A of the cavity wall. It should be understood that any location on the corresponding line of sight of camera 45 to location 47A, such as location 49, and the location of camera 45 itself must be open space within the cavity.

[0101] Returning to the flowchart, in the first comparison step 90, if any of the one or more identified voxels has an outdated CT density value, i.e., a CT density value of tissue rather than a CT density value of open space (newly formed by the ENT procedure), then in a correction step 92, the processor 44 corrects the outdated voxel value to represent the voxel as (newly formed) open space. For example, if a given identified voxel has an outdated CT density value corresponding to hard tissue or soft tissue, the processor 44 may update the density value to correspond to open space. If, in step 90, none of the one or more identified voxels has an outdated CT value, then the method continues with step 94.

[0102] In an embodiment of the present invention, updating one or more updated voxels in an image slice updates the three-dimensional computed tomography image stored in the memory 50 .

[0103] After completing step 92 , control proceeds to a second comparison step 94 .

[0104] In a second comparison step 94 , if the operator 40 has completed the update procedure, ie the operator has sufficiently checked the cavity 47 or other cavities, the method ends. However, if the operator 40 has not yet completed the update procedure, the method continues with step 86 .

[0105] Figure 5 8 is a schematic illustration showing a region of interest 118 of an image slice 101 before updating a computed tomography image as acquired in step 80. Figure 5 In the example shown, blocking bone 120 includes voxel 105B (ie, compact bone) that surrounds the interior region of voxel 105A (ie, the hollow region where bone grows), and voxel 105C includes soft tissue.

[0106] Figure 6 To illustrate the process after updating the computed tomography image (i.e., after completing Figure 2 Schematic illustration of a region of interest 118 of an image slice 101 (after the steps of the flowchart of FIG. 1 ). Figure 6 As shown, the region of interest 118 includes an updated region 150. Region 150 includes an area estimated by the processor 44 based on the visualization from the camera 45 and using the multi-view triangulation model, which is a newly formed open space. Region 150 is formed, for example, by an ENT shaving process that removes Figure 5 Some of the blocking bones 120 in the blocking bones.

[0107] The display 48 generally presents a subset of the voxels 105 that the processor 44 may update during an update procedure using the embodiments described herein. As described above, the image slices 101 include two-dimensional "slices" of a three-dimensional image generated by the processor 44 in response to receiving image data from the mobile computed tomography scanner 21. As the operator moves the distal end 41 in three-dimensional space (i.e., the three-dimensional coordinate systems 28 and 62), it should be understood that there may be additional voxels 105 (i.e., not included in the two-dimensional image slice currently being presented on the display 48) whose corresponding density values ​​are updated by the processor 44 in response to the position of the cavity wall 47 detected by the camera 45.

[0108] Exemplary Image-Guided Surgery Navigation System

[0109] According to another aspect of the present invention, when performing a medical procedure within the head (H) of a patient (P), it may be desirable to obtain information about the position of an instrument within the head (H) of the patient (P), particularly when the instrument is in a position where it is difficult or impossible to obtain an endoscopic view of the working elements of the instrument within the head (H) of the patient (P).

[0110] Figure 7 An exemplary IGS navigation system (10) capable of performing ENT procedures using image guidance according to an embodiment of the present invention is shown. In addition to or in lieu of the components and operability described herein, the IGS navigation system (10) may be constructed and operated in accordance with at least some of the teachings of the following patents: U.S. Patent 7,720,521, entitled "Methods and Devices for Performing Procedures within the Ear, Nose, Throat and Paranasal Sinuses," published on May 18, 2010, the disclosure of which is incorporated herein by reference; and U.S. Patent Publication 2014 / 0364725, entitled "Systems and Methods for Performing Image Guided Procedures within the Ear, Nose, Throat and Paranasal Sinuses," published on December 11, 2014, the disclosure of which is incorporated herein by reference.

[0111] The IGS navigation system (10) of this example includes a field generator assembly (20) including a set of magnetic field generators (24) integrated into a horseshoe-shaped frame (22). The field generator (24) is operable to generate alternating magnetic fields of different frequencies around the head (H) of the patient (P). In this example, a navigation guide wire (40) is inserted into the head (H) of the patient (P). The navigation guide wire (40) may be a stand-alone device or may be positioned at other locations of an end effector or medical instrument (such as a surgical cutting instrument or expansion instrument). In this example, the frame (22) is mounted to a chair (30) in which the patient (P) sits so that the frame (22) is located near the head (H) of the patient (P). By way of example only, the chair (30) and / or field generator assembly (20) may be configured and operated in accordance with at least some of the teachings of U.S. Publication 2018 / 0310886, published on November 1, 2018, entitled “Apparatus to Secure Field Generating Device to Chair,” the disclosure of which is incorporated herein by reference.

[0112] The IGS navigation system (10) of this example also includes a processor (12) that controls the field generator (24) and other elements of the IGS navigation system (10). For example, the processor (12) is operable to drive the field generator (24) to generate an alternating electromagnetic field; and process signals from the navigation guide wire (40) to determine the position of the sensor in the navigation guide wire (40) within the head (H) of the patient (P). The processor (12) includes a processing unit (e.g., a group of electronic circuits arranged to evaluate and execute software instructions using a combinational logic circuit or other similar circuit) that communicates with one or more memories. The processor (12) of this example is installed in a console (18), which includes an operating control (14), which includes a keyboard and / or a pointing device such as a mouse or trackball. When performing a surgical procedure, the physician uses the operating control (14) to interact with the processor (12).

[0113] The navigation guide wire (40) includes a sensor (not shown) that is responsive to positioning within an alternating magnetic field generated by the field generator (24). A coupling unit (42) is secured to the proximal end of the navigation guide wire (40) and is configured to provide communication of data and other signals between the console (18) and the navigation guide wire (40). The coupling unit (42) may provide wired or wireless communication of data and other signals.

[0114] In this example, the sensor of the navigation guidewire (40) includes at least one coil located at the distal end of the navigation guidewire (40). When such a coil is positioned within the alternating electromagnetic field generated by the field generator (24), the alternating magnetic field can generate a current in the coil, and the current can be transmitted along the electrical conduit in the navigation guidewire (40) and further transmitted to the processor (12) via the connection unit (42). This phenomenon can enable the IGS navigation system (10) to determine the position of the distal end of the navigation guidewire (40) or other medical instruments (e.g., expansion instruments, surgical cutting instruments, etc.) in three-dimensional space (i.e., in the head (H) of the patient (P)). To achieve this, the processor (12) executes an algorithm to calculate the position coordinates of the distal end of the navigation guidewire (40) based on the position-related signal of the coil in the navigation guidewire (40). Although the position sensor is located in the guidewire (40) in this example, such a position sensor can be integrated into various other types of instruments, including those described in more detail below.

[0115] The processor (12) uses software stored in the memory of the processor (12) to calibrate and operate the IGS navigation system (10). Such operations include driving the field generator (24), processing data from the navigation guide wire (40), processing data from the operating control (14), and driving the display screen (16). In some embodiments, the operation may also include monitoring and enforcing one or more safety features or functions of the IGS navigation system (10). The processor (12) is also capable of operating to provide a video in real time via the display screen (16), which shows the position of the distal end of the navigation guide wire (40) relative to the video camera image of the patient's head (H), a scanned image of the patient's head (H) (e.g., CT, MRI, or other X-ray or indirect imaging methods), and / or a computer-generated three-dimensional model of the anatomical structure in and near the patient's nasal cavity. The display screen (16) can display such images simultaneously and / or superimposed on each other during a surgical procedure. Such displayed images may also include a graphical representation of an instrument, such as a navigation guidewire (40), inserted into the patient's head (H), so that the operator can view a virtual rendering of the instrument in its actual position in real time. By way of example only, the display screen (16) may provide images in accordance with at least some of the teachings of U.S. Publication 2016 / 0008083, entitled "Guidewire Navigation for Sinuplasty," published on January 14, 2016, the disclosure of which is incorporated herein by reference. In the event that the operator also uses an endoscope, an endoscopic image may also be provided on the display screen (16).

[0116] When the instrument incorporates a navigation guidewire (40), the images provided by the display screen (16) can help guide the operator in maneuvering and otherwise manipulating such an instrument within the patient's head (H). It should also be understood that the surgical instruments described below and other components of other types of surgical instruments may incorporate sensors such as the sensors of the navigation guidewire (40).

[0117] Endoscope with single-camera 3D imaging

[0118] Figure 8 An exemplary endoscope (100) that can be used with a surgical navigation system or other surgical system such as an IGS navigation system (10) according to an embodiment of the present invention is shown. The endoscope (100) includes a body (102) that can be held during use. A set of controls (103) is positioned at the distal end of the body (102). A shaft (104) extends distally from the body (102). The shaft (104) includes a flexure (108) and a distal tip (106). As will be described in more detail below, the distal tip (106) may include one or more feature structures, such as a camera, a light source, flushing and suction, and / or a channel through which a flexible tool can be deployed. The endoscope (100) also includes an orientation sensor (110) located between the flexure (108) and the distal tip (106). The position sensor (110) may include one or more coils configured to generate a position indication signal in response to an alternating electromagnetic field generated by the field generator 24, so that the navigation system (10) can track the three-dimensional position of the distal tip (106) within the head (H) of the patient (P) in real time, which is similar to the position tracking of the navigation guidewire (40) described above.

[0119] The control (103) is operable to drive the rotation of the shaft (104) about the longitudinal axis of the shaft (104). The control (103) is also operable to drive the deflection of the distal tip (106) away from the longitudinal axis of the shaft (104) at the flexure (108). By way of example only, the control circuit (103) may be configured and operated according to at least some of the teachings of U.S. Publication 2019 / 0015645, entitled "Adjustable Instrument for Dilation of Anatomical Passageway," published on January 17, 2019, the disclosure of which is incorporated herein by reference. By way of further example only, the endoscope (100) may also include one or more controls for activating a feature of the endoscope (100); or controls for advancing, retracting, and rotating a flexible tool deployed to a surgical site via the endoscope (100); or any other type of control that would be apparent to one skilled in the art in view of the teachings herein.

[0120] Fig. 9A and Fig. 9B Each shows an exemplary distal tip (300) that can be used as the distal tip (106) of the endoscope (100) according to an embodiment of the present invention. The distal tip (300) can be statically positioned at the distal end of the shaft (104); or can be positioned at the distal end of a flexible shaft or a deflectable shaft that extends from a channel of the distal tip (106) or can be advanced and withdrawn through the channel. The distal tip (300) includes an instrument channel (302) through which flexible instruments and tools can be deployed once the distal tip (300) is positioned at a desired location. By way of example only, the size of the channel (302) can be set to accommodate a dilatation catheter, a shaving instrument, an ablation instrument, and / or various other types of instruments that will be apparent to those skilled in the art with reference to the teachings herein. The distal tip 300 also includes a wavefront imaging device (304) and a set of lights (306) (e.g., LEDs, optical fibers, infrared light sources, etc.). The distal tip (300) also includes a set of irrigation diverters (308). Each irrigation diverter (308) is positioned proximally to a corresponding irrigation channel (e.g., Fig. 10A and Fig. 10B , each of which shows the distal tip (300) with one or both of the irrigation diverters (308) removed. The irrigation diverter (308) is adapted to receive and direct water or other liquid from the irrigation channel (310) to wash the wavefront imaging device (304), the lamp (306), or both.

[0121] The wavefront imaging device (304) may include a digital camera (e.g., one or more image sensors) or imaging device capable of acquiring image data and transmitting the image data to the processor (12); or another device of the IGS navigation system (10) capable of storing and processing image data to generate 3D image data. While some conventional 3D image acquisition devices may require two or more offset imaging devices that can actively and independently focus on a target, the wavefront imaging device (304) of the present example is capable of generating 3D image data from a single camera or image sensor using wavefront sampling techniques. In order to generate image data compatible with the wavefront sampling techniques, the wavefront imaging device (304) includes one or more apertures that are positioned between the target and the camera lens; and are off-axis or offset from the optical axis of the lens. When the target is imaged using the wavefront imaging device (304), a single image will be generated that will contain multiple unfocused depictions of the target that are offset from each other based on the orientation of the one or more apertures. The distance between each unfocused image can be used to calculate the depth or distance to the imaged target. The images acquired by the wavefront imaging device (304) may be provided as input to a wavefront sampling algorithm (such as Frigerio's multi-image procedure) to generate a 3D depth map and 3D model of the imaged object.

[0122] Fig.11 An example of a wavefront imaging device (304) implementing a static optical system (320) according to an embodiment of the present invention is shown. The static optical system (320) includes a lens (326) having a first static aperture (328a) and a second static aperture (328b) positioned between the lens (326) and a target (330). The apertures (328a, 328b) are diametrically opposed and offset from the optical axis of the lens (326). Two or more apertures may be used, wherein a greater number of apertures improves the accuracy of the resulting 3D image while requiring additional time for imaging of the target and processing of the image data. A smaller number of apertures may shorten the time required for imaging and processing, but at the expense of the accuracy of the resulting 3D image. The aperture size may be increased or decreased to control the depth of field of the acquired image. Reducing the distance between the aperture and the lens (326) may also produce improved wavefront sampling results.

[0123] During imaging, the lens (326) receives a first unfocused image (332a) and a second unfocused image (332b) of the target (330) through the apertures (328a, 328b) and directs the first unfocused image (332a) and the second unfocused image (332b) onto an image plane (322). The image plane (322) may be provided by a CCD, CMOS, or other image sensor. The image offset (336) between the resulting images (332a, 332b) may then be determined and used as part of a wavefront sampling technique to determine pixel depth and generate a 3D model of the target (330).

[0124] Fig.12 Another example of a wavefront imaging device (304) implemented as an active optical system (340) according to an embodiment of the present invention is shown. The active optical system (340) includes a lens (326) having a single aperture (344a) positioned on a rotatable aperture plate (342) between the lens (326) and the target (330). The aperture plate (342) can be rotated electronically and automatically during imaging to move the single aperture (344a) from its illustrated orientation to a second orientation (344b), a third orientation (344c), etc. throughout the rotation process. In this example, the aperture plate (342) rotates about a rotation axis coaxial with the optical axis of the lens (326). As with the static optical system (320), the lens (326) receives an unfocused image of the target (330) during the rotation of the aperture plate (342). Thus, a first unfocused image (346a), a second unfocused image (346b), a third unfocused image (346c), etc. are projected onto an image plane (322) at offset locations along the rotation path (348). This image data acquired by the image plane (322) can then be used with wavefront sampling techniques to determine pixel depth and generate a 3D model of the object (330).

[0125] Fig.13 An exemplary set of steps (400) is shown that may be performed to generate a three-dimensional image model using an imaging device such as imaging device 304 according to an embodiment of the present invention. The image data may be as described above in Fig.11 and Fig.12402) and is processed into 3D image data by applying a wavefront sampling algorithm (e.g., Frigerio's multi-image procedure or another similar algorithm). For example, such a process may include defining an anchor image to be used as a basis for determining the offsets of other images (box 404). The anchor image may be an arbitrarily selected image, i.e., the image that is most centrally located relative to two or more other images; or it may be selected based on other criteria. Optical flow methods may then be used to calculate pixel displacements between the anchor image and the other images (box 406). A circle may be fitted based on points across the image (box 408), and a rotation diameter (or other offset value) may be determined (box 410). The rotation diameter may then be used to calculate the pixel depth of multiple pixels of the image (box 412). The pixel depth may then be used to construct a 3D model of the imaged object or target (box 414).

[0126] The distal tip (300) and wavefront imaging device (304) can be advantageous because they provide a solution for 3D imaging of a surgical site that has relatively low complexity (e.g., compared to a 3D camera having multiple independently and actively oriented cameras); and they can be mounted on the end of an instrument, such as at the distal tip (106), or can be mounted on a flexible or deflectable shaft. The advantage of reduced complexity can be particularly realized when the wavefront imaging device (304) is implemented using a static optical system (320), which can be implemented under the premise of small physical size requirements and minimal support requirements due to low power usage and the absence of moving parts (e.g., such as aperture plates (342) or independently oriented cameras).

[0127] As described above, the endoscope (100) of this example includes an orientation sensor (110) that is operable to provide a signal indicating the real-time orientation of the distal tip (106) in three-dimensional space. When the imaging device (304) acquires images, the image data from the imaging device (304) can be stored together with the orientation data from the orientation sensor (110) so that the IGS navigation system (10) (and / or some other computer system) can associate each acquired image with its corresponding position in three-dimensional space. This will enable the IGS navigation system (10) to associate the images acquired using the imaging device (304) with one or more pre-operatively acquired images (e.g., CT or MRI scans, 3-D maps, etc.). This aspect of the invention, together with the dual-camera 3D imaging described below, can be used in conjunction with the image correction aspect of the invention described above.

[0128] Endoscope with dual-camera 3D imaging

[0129] Fig.14A and Fig. 14BAnother exemplary distal tip (500) that can be used as the distal tip (106) of the endoscope (100) according to other embodiments of the present invention is shown. The distal tip (500) can be statically positioned at the distal end of the shaft (104); or can be positioned at the distal end of a flexible shaft or a deflectable shaft that extends from or can be advanced and withdrawn through a channel of the distal tip (106). The distal tip (500) includes an instrument channel (502) having similar features and functions as the instrument channel (302); a light (506) having similar features and functions as the light (306); and a set of irrigation channels (510) having similar features and functions as the irrigation channels (310). The distal tip (500) also includes a first imaging device (504a) and a second imaging device (504b), which may include a digital camera (e.g., one or more image sensors) or other digital imaging device capable of acquiring image data at the distal tip (500) and transmitting it via a shaft or flexible shaft to a processor (12), a coupling unit (42), or another device of the IGS navigation system (10) capable of processing the acquired images. Figure 8 B, it can be seen that the imaging devices (504a, 504b) are offset from each other along the first dimension (512) and are located at the same position along the second dimension (514).

[0130] Fig.15A and Fig. 15B A diagrammatic view of a distal tip (500) for imaging a target using passive stereo vision, which is a triangulation method that uses two or more imaging devices to obtain images from two or more different viewing angles, is shown in accordance with an embodiment of the present invention. Images generated using passive stereo vision can be converted into 3D image data algorithms, such as photogrammetry algorithms that determine pixel depth based on epipolar geometry. Fig.15A , it can be seen that the first imaging device (504a) and the second imaging device (504b) are arranged on the distal tip (500) so that they have parallel optical axes. When directed to the target (520), the first imaging device (504a) will capture a first field of view (522) of the target (520), while the second imaging device (504b) will capture a second field of view (526) of the target (520).

[0131] Because the two imaging devices (504a, 504b) have parallel optical axes (e.g., are not independently focused or oriented to a convergence point), the images acquired by these devices may contain overlapping portions of the target (520). Fig.15A and Fig. 15BA first field of view (522) and a second field of view (526) are shown, as well as a shared field of view (524) in which a target (520) is to be acquired by two imaging devices. Fig. 15B , it can be seen that the target (520) is substantially centered within the shared field of view (524). However, the target (520) is positioned to the right of the first field of view (522) and to the left of the second field of view (526). This parallax, or the distance that the target (520) is offset or shifted from the first field of view (522) to the second field of view (526), ​​can be used to determine the distance between the imaging device and the target, and subsequently the depth of a pixel or portion of the captured image.

[0132] Determining and using disparity to determine depth is an example of passive triangulation. Fig.16 A set of exemplary steps (600) that may be performed to apply passive triangulation to image data in accordance with an embodiment of the present invention are shown. Image data of an object may be acquired (block 602) by two or more statically arranged cameras, such as a first imaging device (504a) and a second imaging device (504b), thereby generating two different images of the object from two different orientations. Points (e.g., one or more pixels or portions of an image) may be picked up within one image (block 604), and a matching point (e.g., a portion of an object shared between the two images, such as the object (520)) may be searched for in the second image. Fig. 9B Performing image analysis on points within a first image to find matching points within the entire second image may be both inefficient and inaccurate, thereby resulting in high processing requirements and an increased likelihood of false positive matches.

[0133] To improve the speed and accuracy of the matching, features of the epipolar lines associated with the two images may be determined (block 606). Epipolar geometry is a concept that can be applied to two images acquired by cameras with parallel optical axes in order to determine the disparity or distance between two matching pixels or points in the images without searching the entire image for a match. Fig.17 and Fig.19 A diagram showing epipolar geometry for use in stereoscopic imaging is shown according to an embodiment of the present invention.

[0134] refer to Fig.17, which shows a diagram (700) showing the relationship between disparity (e.g., the distance between two matching points in an image) and depth (e.g., the distance from a camera to an object or a pixel or point in an object), a first camera (704) and a second camera (706) are arranged so that they have parallel optical axes (e.g., line FB and line GD) passing through their respective lenses, which are arranged on the same lens plane (703). With this arrangement, object A (702) is within the field of view of each camera. The distance between the optical axes of the cameras is BC+CD, and triangles ACB and BFE have similar features but different sizes. Applying triangle geometry to Fig.17 , the displacement between two matching points or pixels in the image can be expressed as the distance between the camera optical axes multiplied by the distance from the lens plane (703) to the image sensor divided by the depth or distance to the target (702), or displacement = (BD*BF) / AC. After determining the displacement or parallax, this formula can alternatively be used to determine or solve for the distance or AC to the target (702).

[0135] To determine the displacement between points or pixels in an image without searching the entire image, we can apply Fig.18 The principle of epipolar geometry is shown. Since the cameras (704, 706) have a fixed orientation relative to each other and parallel optical axes, an epipolar line (712) extending across the first image (708) and the second image (710) can be determined along which matching points or pixels can be found. Therefore, when matching points on the first image (708) with points on the second image (710), only the portion of the second image (710) that falls on the epipolar line (712) needs to be searched, rather than the entire second image (710). By matching in this manner, the speed and accuracy with which points can be matched across the entire image can be significantly improved.

[0136] After matching the points (block 608), by applying epipolar geometry or using another matching method, the displacement or disparity between the matching points can be determined (block 610), which can then be used to determine the depth or distance between the imaging device and the object (block 612). When the depth can be determined (block 612), a 3D image dataset and a 3D model can be constructed (block 614).

[0137] As described above, the endoscope (100) of this example includes a position sensor (110) that is operable to provide a signal indicating the real-time position of the distal tip (106) in three-dimensional space. When the imaging device (504a, 504b) acquires images, the image data from the imaging device (504a, 504b) can be stored together with the position data from the position sensor (110) so that the IGS navigation system (10) (and / or some other computer system) can associate each acquired image with its corresponding position in three-dimensional space. This will enable the IGS navigation system (10) to associate the images acquired using the imaging device (504a, 504b) with one or more pre-operative images (e.g., CT or MRI scans, 3-D maps, etc.).

[0138] Methods for integrating 3D imaging and navigation

[0139] After generating 3D image data as described above, such image data can be used to provide additional features during IGS navigation. For example, Fig.19 A flow chart of a set of steps (800) that may be performed to integrate a 3D model or 3D image data with IGS navigation according to an embodiment of the present invention is shown. After image data (box 802) is acquired and a 3D model or other 3D image data set (box 804) is constructed using an appropriate imaging module (e.g., a waveform imaging device (304) of the distal tip (300), a set of cameras (504a, 504b) of the distal tip (500), such data may be used in various ways. For example, a composite image set (box 806) may be created by combining the 3D image data with other images and data-associated image sets with each other using position data such as position and orientation. This may include, for example, associating the constructed 3D model (box 804) with a CT image set acquired before surgery, a 2D image set acquired using a 2D endoscope with position tracking during a procedure, or other similar information, so that matching perspectives of the surgical site may be displayed simultaneously from multiple image sets.

[0140] As another example, a pre-operative comparison (box 808) may be created that compares a CT scan or other 3D model acquired prior to surgery to the constructed 3D model (box 804). This may be used to assist in planning or preparing for a surgical procedure, or to verify the accuracy and configuration of one or more 3D models. For example, comparison of the constructed 3D model (box 804) to the CT image set may help identify missing or incorrect image or depth data in one or both models, identify incorrectly associated position data in one or both models, or identify other errors that may be corrected by re-imaging to generate new 3D image data or by reconfiguring or recalibrating to correct position data.

[0141] As another example, a post-operative comparison (block 810) may be created that compares a post-operative CT scan, 3D model, or other image data to the constructed 3D model (block 804) to assist in assessing the success of the surgical procedure. The 3D model constructed before surgery (block 804) may be compared to the 3D model constructed after surgery (block 804); or compared to a post-operative CT scan or 2D imaging of the surgical site. Such comparative data may help a clinician assess the success or completeness of a surgical procedure by physically comparing anatomical structures from different viewing angles available in 3D imaging.

[0142] By way of further example only, 3D image data acquired or otherwise generated by distal tip (200) or distal tip (300) may be used in accordance with at least some of the teachings of the following patents: U.S. Provisional Patent Application 62 / 782,608, filed on December 20, 2018, entitled “3D Scanning of Nasal Tract with Deflectable Endoscope,” the disclosure of which is incorporated herein by reference; U.S. Patent 8,199,988, published on June 12, 2012, entitled “Method and Apparatus for Combining 3D Dental Scans with Other 3D Data Sets,” the disclosure of which is incorporated herein by reference; and / or U.S. Patent 8,821,158, published on September 2, 2014, entitled “Method and Apparatus for Matching Digital Three-Dimensional Dental Modelswith Digital Three-Dimensional Cranio-Facial CAT Scan Records,” the disclosure of which is incorporated herein by reference.

[0143] Exemplary combinations

[0144] The following examples relate to various non-exhaustive ways in which the teachings herein may be combined or applied. It should be understood that the following examples are not intended to limit the coverage of any claims that may be provided at any time in this patent application or in the subsequent submissions of this patent application. It is not intended to make a disclaimer. The following examples are provided only for illustrative purposes. It is expected that the various teachings herein may be arranged and applied in a variety of other ways. It is also contemplated that some variations may omit certain features mentioned in the following examples. Therefore, any of the aspects or features mentioned below should not be considered decisive, unless otherwise, for example, by the inventor or the successor of the inventor concerned clearly indicates so at a later date. If any claim proposed in this patent application or in the subsequent submissions related to this patent application includes additional features other than those mentioned below, these additional features should not be assumed to be added for any reason related to patentability.

[0145] Example 1

[0146] A three-dimensional (3D) imaging system comprises: an endoscope, the endoscope comprising: a shaft having a distal end, the shaft being adapted to be inserted into a patient and positioned at a surgical site of the patient; a position sensor, the position sensor being proximate to the distal end and being configured to generate a set of position signals based on the position of the endoscope during use; and an imaging module, the imaging module being positioned at the distal end and being operable to acquire an image dataset of the surgical site, wherein the image dataset comprises one or more two-dimensional (2D) images; and a processor, the processor being communicatively coupled to the endoscope and being configured to: receive the image dataset and the set of position signals from the endoscope; determine a view dataset based on the set of position signals, wherein the view dataset indicates the position of the endoscope during acquisition of the image dataset; perform image depth analysis to determine a 3D feature set of the image dataset, wherein the 3D feature set comprises pixel depths in the one or more 2D images; create a 3D image dataset based on the one or more 2D images and the 3D feature set; and associate the view dataset with the 3D image dataset.

[0147] Example 2

[0148] A 3D imaging system according to embodiment 1, wherein the imaging module comprises: a single lens; an aperture plate positioned between a first side of the single lens and the surgical site, the aperture plate comprising one or more apertures offset from the optical axis of the single lens; and an image pane positioned on a second side of the single lens to receive reflected light from the surgical site via the one or more apertures and the single lens, wherein the image pane is configured to generate the image data set based on the reflected light.

[0149] Example 3

[0150] A 3D imaging system according to embodiment 2, wherein: the one or more apertures include at least two apertures positioned on the aperture plate and offset from the optical axis of the single lens, and the aperture plate has a fixed position and orientation relative to the lens.

[0151] Example 4

[0152] A 3D imaging system according to any one or more of embodiments 2 to 3, wherein: the one or more apertures include a single aperture positioned on the aperture plate offset from the optical axis of the single lens, and the aperture plate is operable to rotate around its circular axis relative to the lens during image acquisition.

[0153] Example 5

[0154] A 3D imaging system according to any one or more of embodiments 2 to 4, wherein the processor is configured to, when performing the image depth analysis: identify two or more unfocused images of the surgical site within the image dataset; determine a spatial relationship between the two or more unfocused images of the surgical site; and determine the pixel depth in the image dataset based on the spatial relationship between the two or more unfocused images.

[0155] Example 6

[0156] A 3D imaging system according to any one or more of embodiments 1 to 5, wherein the imaging module includes two or more cameras, and wherein each of the two or more cameras is statically positioned relative to every other camera of the two or more cameras and oriented to have a parallel optical axis with every other camera of the two or more cameras.

[0157] Example 7

[0158] According to the 3D imaging system of Example 6, the processor is also configured to, when performing the image depth analysis: identify a point in a first image in the image data set, wherein the point includes a portion of the surgical site that exists in both the first image acquired by the first camera among the two or more cameras and the second image acquired by the second camera among the two or more cameras; identify the point in the second image; determine the displacement of the point from the first image to the second image; and determine the pixel depth of the point based on the displacement.

[0159] Example 8

[0160] A 3D imaging system according to Example 7, wherein the processor is further configured to, when identifying the point in the second image: determine the epipolar line of the first image and the second image based on the static position of the first camera relative to the second camera; and search for the point in the second image along the epipolar line while excluding portions of the second image that do not fall on the epipolar line.

[0161] Example 9

[0162] A 3D imaging system according to any one or more of Examples 1 to 8, wherein the processor is further configured to: associate the 3D image dataset and the viewing angle dataset with a coordinate system of an image-guided surgery system, and based on the association with the coordinate system, display the 3D image dataset during an image-guided surgery navigation procedure.

[0163] Example 10

[0164] A 3D imaging system according to any one or more of embodiments 1 to 9, wherein: the orientation sensor is configured to generate the orientation signal set based on the position and orientation of the endoscope during use, the perspective dataset indicates the position and orientation of the endoscope during acquisition of the image dataset, and the processor is further configured to provide the 3D image dataset and the perspective dataset to an image-guided surgical navigation system.

[0165] Embodiment 11

[0166] A 3D imaging system according to any one or more of embodiments 1 to 10, wherein the processor is further configured to: receive input from a user to define a viewing angle relative to the surgical site; determine a first portion of the 3D image dataset showing the surgical site from the viewing angle based on identifying the viewing angle within the viewing angle dataset; and display the first portion of the 3D image dataset on a display.

[0167] Example 12

[0168] A 3D imaging system according to embodiment 11, wherein the processor is further configured to: receive an indirect 3D scan of the surgical site and a scanning perspective dataset associated with the indirect 3D scan; determine a second portion of the indirect 3D scan showing the surgical site from the perspective based on identifying the perspective within the scanning perspective dataset; and simultaneously display the first portion of the 3D image dataset and the second portion of the indirect 3D scan on the display.

[0169] Example 13

[0170] A 3D imaging system according to Example 12, wherein: the indirect 3D scan of the surgical site includes image data acquired before the surgery, and the 3D image data set includes image data acquired after the surgery.

[0171] Embodiment 14

[0172] A 3D imaging system according to any one or more of embodiments 12 to 13, wherein: the indirect 3D scan of the surgical site includes image data acquired before surgery, the 3D image dataset includes image data acquired before surgery, and the processor is further configured to: receive scan adjustment input from a user; and reconfigure the association between the indirect 3D scan of the surgical site and the scan view dataset based on the scan adjustment input.

[0173] Embodiment 15

[0174] A method for three-dimensional (3D) imaging comprises: deploying a distal end of an endoscope to a surgical site of a patient, the distal end comprising: an imaging module, the imaging module being operable to acquire image data of the surgical site, wherein the acquired image data comprises one or more two-dimensional (2D) images; and an orientation sensor, the orientation sensor being proximate to the distal end and configured to generate an orientation signal based on the position of the endoscope; receiving an image data set from the imaging module and an orientation signal set from the orientation sensor; determining a perspective data set based on the orientation signal set, wherein the perspective data set indicates the position of the endoscope during acquisition of the image data set; performing image depth analysis to determine a 3D feature set of the image data set, wherein the 3D feature set comprises pixel depths in the one or more 2D images in the image data set; creating a 3D image data set based on the one or more 2D images in the image data set and the 3D feature set; and associating the perspective data set with the 3D image data set.

[0175] Example 16

[0176] The method according to Example 15 also includes: associating the 3D image dataset and the perspective dataset with a coordinate system of an image-guided surgery system; and displaying the 3D image dataset during an image-guided surgery navigation procedure based on the association with the coordinate system.

[0177] Embodiment 17

[0178] The method described in accordance with any one or more of embodiments 15 to 16 also includes: receiving input from a user to limit a viewing angle relative to the surgical site; determining a first portion of the 3D image dataset showing the surgical site from the viewing angle based on identifying the viewing angle within the viewing angle dataset; and displaying the first portion of the 3D image dataset on a display.

[0179] Embodiment 18

[0180] The method according to Example 17 also includes: receiving an indirect 3D scan of the surgical site and a scanning perspective dataset associated with the indirect 3D scan; determining a second portion of the indirect 3D scan showing the surgical site from the perspective based on identifying the perspective within the scanning perspective dataset; and simultaneously displaying the first portion of the 3D image dataset and the second portion of the indirect 3D scan on the display.

[0181] Embodiment 19

[0182] The method according to Example 18 also includes: receiving scan adjustment input from a user; and reconfiguring the association between the indirect 3D scan of the surgical site and the scan perspective dataset based on the scan adjustment input; wherein: the indirect 3D scan of the surgical site includes image data acquired before the operation, and the 3D image dataset includes image data acquired before the operation.

[0183] Embodiment 20

[0184] An image-guided surgery (IGS) navigation system includes a processor, a memory, and a display, wherein the processor is configured to: receive an image data set generated by a tracking endoscope, the image data set including one or more two-dimensional (2D) images; receive a perspective data set generated by the tracking endoscope, wherein the perspective data set indicates the position of the tracking endoscope during the acquisition of the image data set; perform image depth analysis to determine a 3D feature set of the image data set, wherein the 3D feature set includes pixel depths in the one or more 2D images; create a 3D image data set based on the one or more 2D images and the 3D feature set; associate the perspective data set with the 3D image data set; and cause the display to display the 3D image data set from a selected perspective based on the perspective data set including a selected perspective.

[0185] Embodiment 21

[0186] A system comprises: a medical probe configured to be inserted into a cavity of an organ of a patient; a position and orientation sensor located in the medical probe, the position and orientation sensor operating in a sensor coordinate system; a camera located in a distal edge of the medical probe, the camera operating in a sensor coordinate system; and the processor is configured to: receive a three-dimensional image of the cavity including open space and organ tissue from an imaging system operating in an image coordinate system; receive a signal indicating the position and corresponding orientation of the distal edge of the medical probe within the cavity from the medical probe; receive a corresponding visualization position within the cavity from the camera of the probe; align the image coordinate system with the sensor coordinate system to identify one or more voxels in the three-dimensional image at the visualization position; and when the identified voxel has a density value that does not correspond to the open space in the received three-dimensional image, update the density value of the identified voxel to correspond to the open space.

[0187] Embodiment 22

[0188] The system of embodiment 21, wherein the imaging system comprises a computed tomography scanner.

[0189] Embodiment 23

[0190] A system as described in embodiment 21, wherein the position and direction sensor includes a magnetic field sensor.

[0191] Embodiment 24

[0192] A system according to any one of embodiments 21 to 23, wherein the processor is configured to form a correspondence between the density values ​​and visual effects, wherein a given visual effect corresponds to a given density value indicating the open space.

[0193] Embodiment 25

[0194] A system as described in embodiment 24, wherein the visual effect is selected from the group consisting of color, shadow, and pattern.

[0195] Embodiment 26

[0196] A system as described in Example 24, wherein the processor is configured to present the three-dimensional image on a display using the visual effect.

[0197] Embodiment 27

[0198] A system according to embodiment 26, wherein the given visual effect includes a first given visual effect, and wherein before updating the density value, the processor is configured to present the three-dimensional image by presenting the one or more identified voxels using a second given visual effect different from the first given visual effect.

[0199] Embodiment 28

[0200] A system as described in Example 27, wherein when updating the density value, the processor is configured to render the three-dimensional image by rendering the one or more identified voxels using the first given visual effect.

[0201] Embodiment 29

[0202] A system according to embodiment 28, wherein the processor is configured to use a multi-view Figure 3 The angular partitioning model extracts the distance of a location from the camera from the visual signal.

[0203] Embodiment 30

[0204] A method comprising: receiving, from an imaging system operating in an image coordinate system, a three-dimensional image of a cavity of an organ of a patient including open space and organ tissue; receiving, from a medical probe having a position and orientation sensor and a camera, wherein the probe operates in a sensor coordinate system and is inserted into the cavity: a signal indicating the position and corresponding orientation of a distal edge of the medical probe within the cavity; and a corresponding visualization location within the cavity; aligning the image coordinate system with the sensor coordinate system so as to identify one or more voxels in the three-dimensional image at the visualization location; and when the identified voxel has a density value in the received three-dimensional image that does not correspond to the open space, updating the density value of the identified voxel to correspond to the open space.

[0205] Embodiment 31

[0206] A computer software product that operates in conjunction with a probe configured for insertion into a cavity of an organ of a patient and includes a position and orientation sensor operating in a sensor coordinate system and a camera located in a distal edge of the medical probe operating in the sensor coordinate system, and includes a non-transitory computer-readable medium having program instructions stored therein, which, when read by a computer, causes the computer to: receive a three-dimensional image of the cavity including open space and organ tissue from an imaging system operating in an image coordinate system; receive a signal from the medical probe indicating the position and corresponding orientation of the distal edge of the medical probe within the cavity; receive a corresponding visualization position of a wall of the cavity; align the image coordinate system with the sensor coordinate system so as to identify one or more voxels in the three-dimensional image at the visualization position; and when the identified voxel has a density value in the received three-dimensional image that does not correspond to the open space, update the density value of the identified voxel to correspond to the open space.

[0207] Miscellaneous

[0208] Although the embodiments herein describe a processor that uses magnetic P&D sensing and optical visualization by a camera to correct outdated CT images, it is considered to be within the spirit and scope of the present invention to correct other types of medical images using other types of position sensing and visualization. For example, ultrasound visualization can be used with an ultrasound transducer instead of a camera. In addition, although the embodiments described herein are primarily directed to ENT procedures, the methods and systems described herein can also be used in other applications, such as other cavities of organs of the body.

[0209] It should be understood that any one or more of the teachings, expressions, embodiments, examples, etc. described herein may be combined with any one or more of the other teachings, expressions, embodiments, examples, etc. described herein. Therefore, the above teachings, expressions, embodiments, examples, etc. should not be considered isolated from each other. With reference to the teachings herein, various suitable ways in which the teachings herein may be combined will be apparent to those skilled in the art. Such modifications and variations are intended to be included within the scope of the claims.

[0210] It should be understood that any patent, patent publication, or other public material, whether in whole or in part, allegedly incorporated herein by reference is incorporated herein only to the extent that the incorporated material does not conflict with existing definitions, statements, or other public materials described in the present disclosure. Therefore, and to the extent necessary, the disclosure explicitly listed herein supersedes any conflicting material incorporated herein by reference. Any material or portion thereof allegedly incorporated herein but conflicting with existing definitions, statements, or other public materials listed herein will be incorporated only to the extent that there is no conflict between the incorporated material and the existing public materials.

[0211] The types of devices disclosed herein can be designed to be discarded after one use, or they can be designed for multiple uses. In either case or both, these types can be repaired for reuse after at least one use. Repair can include any combination of the following steps: disassembling the device, then cleaning or replacing specific parts and subsequently reassembling. Specifically, the type of the device can be disassembled, and any number of specific components or parts of the device can be selectively replaced or removed in any combination. When cleaning and / or replacing specific parts, the type of the device can be reassembled in a repair facility for subsequent use, or reassembled by a surgical team before a surgical operation is about to be performed. Those skilled in the art will appreciate that the repair of the device can be disassembled, cleaned / replaced, and reassembled using a variety of techniques. The use of such techniques and the resulting repair device are all within the scope of this application.

[0212] By way of example only, the patterns described herein may be processed prior to a surgical procedure. First, new or used instruments may be obtained and cleaned as needed. Then, the instruments may be sterilized. In one sterilization technique, the instruments are placed in a closed and sealed container (such as a plastic or TYVEK bag). The container and instruments may then be placed in a radiation field that can penetrate the container, such as gamma radiation, x-rays, or high-energy electrons. Radiation may kill bacteria on the instruments and in the container. The sterilized instruments may then be stored in a sterile container. The sealed container may keep the instruments sterile until the container is opened in a surgical facility. Any other technique known in the art may also be used to sterilize the device, including but not limited to beta radiation or gamma radiation, ethylene oxide, or steam.

[0213] While various versions of the invention have been shown and described, further improvements to the methods and systems described herein may be achieved by appropriate modifications by those skilled in the art without departing from the scope of the invention. Several such possible modifications have been mentioned, and other modifications will be apparent to those skilled in the art. For example, the examples, versions, geometries, materials, dimensions, ratios, steps, etc. discussed above are exemplary and not required. Therefore, the scope of the invention should be considered in light of the following claims and should be understood not to be limited to the details of the structure and operation shown and described in the specification and drawings.

Claims

1. A system, include: a medical probe configured to be inserted into a cavity of an organ of a patient; a position and orientation sensor located in the medical probe, the position and orientation sensor operating in a sensor coordinate system; a camera located in a distal edge of the medical probe, the camera operating in a sensor coordinate system; as well as A processor, the processor being configured to: receiving a three-dimensional image of the cavity including open space and organ tissue from an imaging system operating in an image coordinate system; receiving a signal from the medical probe indicative of the position and corresponding orientation of the distal edge of the medical probe within the cavity; receiving a corresponding visualized location within the cavity from the camera of the probe; registering the image coordinate system with the sensor coordinate system to identify one or more voxels in the three-dimensional image at the visualization location; as well as When the identified voxel has a density value that does not correspond to the open space in the received three-dimensional image, the density value of the identified voxel is updated to correspond to the open space.

2. The system of claim 1, wherein the imaging system comprises a computed tomography scanner.

3. The system of claim 1, wherein the position and direction sensor comprises a magnetic field sensor.

4. The system of any one of claims 1-3, wherein the processor is configured to form a correspondence between the density values ​​and visual effects, wherein a given visual effect corresponds to a given density value indicating the open space.

5. The system of claim 4, wherein the visual effect is selected from the group consisting of color, shading, and pattern.

6. The system of claim 4, wherein the processor is configured to present the three-dimensional image on a display using the visual effect.

7. The system of claim 6, wherein the given visual effect comprises a first given visual effect, and wherein, before updating the density value, the processor is configured to render the three-dimensional image by rendering the identified one or more voxels using a second given visual effect different from the first given visual effect. 8 . The system of claim 7 , wherein when updating the density value, the processor is configured to render the three-dimensional image by rendering the identified one or more voxels using the first given visual effect.

9. The system of claim 4, wherein the processor is configured to extract the distance of a location from the camera from the visual signal using a multi-view triangulation model.

10. A computer software product operating in conjunction with a probe configured for insertion into a cavity of an organ of a patient and comprising a position and orientation sensor operating in a sensor coordinate system and a camera located in a distal edge of the medical probe operating in the sensor coordinate system, and comprising a non-transitory computer readable medium having program instructions stored therein, which when read by a computer causes the computer to: receiving a three-dimensional image of the cavity including open space and organ tissue from an imaging system operating in an image coordinate system; receiving a signal from the medical probe indicative of the position and corresponding orientation of the distal edge of the medical probe within the cavity; receiving a corresponding visualized location of a wall of the cavity; registering the image coordinate system with the sensor coordinate system to identify one or more voxels in the three-dimensional image at the visualization location; as well as When the identified voxel has a density value that does not correspond to the open space in the received three-dimensional image, the density value of the identified voxel is updated to correspond to the open space.