Using augmented reality to aid navigation during medical procedures

Through augmented reality display technology, combined with camera and position sensors, the synthetic image display tools and anatomical structure are generated, solving the problem of invisibility of tools in surgical procedures and improving the accuracy and precision of the surgery.

CN109998678BActive Publication Date: 2024-11-01BIOSENSE WEBSTER (ISRAEL) LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201811599640.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-12-21
Filing Date
2018-12-26
Publication Date
2024-11-01
Estimated Expiration
2038-12-26

AI Technical Summary

Technical Problem

Due to the opaque anatomy during surgery, the tool cannot be accurately seen during the operation, which affects the accuracy and precision of the surgery.

Method used

Using augmented reality display technology, the camera and position sensor tracks the position of the camera and camera, combined with 3D models and real-time images, generates synthetic images to display the position and anatomy of the tool on the display, ensuring that surgeons observe patients in real time on a single display device Image and tool operation.

Benefits of technology

It improves the accuracy and precision of the surgery, enhances the visibility of the surgeons of tools and anatomy, reduces the difficulty of operation due to tissue occlusion, and improves the effectiveness of minimally invasive surgery.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN109998678B_ABST
    Figure CN109998678B_ABST
Patent Text Reader

Abstract

The present invention is entitled "Using Augmented Reality to Assist Navigation During Medical Procedures." Physicians performing invasive procedures require accuracy and precision when working with surgical tools. Surgical procedures are increasingly becoming minimally invasive, with physicians operating with cameras to view the surgical site and guiding their tools through an eyepiece or video display. Ideally, the physician should be able to perform invasive procedures while being able to observe both a real-time image of the patient and additional data that is critical to medical decisions regarding the manipulation of the surgical tools and the next surgical step. The augmented reality navigation system of the present disclosure provides visibility of tool position for invasive procedures by using position sensors included on the camera and / or on the tools used during the procedure. The position tracking system determines and monitors the position of the tool and camera based on characteristics of the signals detected by the sensors and displays an informative overlay on the image obtained using the camera.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Surgical procedures often encounter problems with anatomical visibility. Visual aids exist to help visualize anatomical structures in the body. However, the effectiveness of such visual aids is limited because anatomical structures are opaque and therefore hinder visibility of other anatomical structures. The nature of surgical procedures involves inserting one or more tools into the human body, which often results in an inability to see exactly what is happening with these tools. Improved techniques for obtaining visibility during procedures are continually being developed. Summary of the invention

[0002] The present invention provides a method for providing an augmented reality display of an object. The method includes obtaining an image from a camera. The method also includes detecting a position of the camera. The method also includes identifying a field of view of the camera based on the position of the camera. The method also includes generating one or more stacked features corresponding to the field of view. The method also includes synthesizing the one or more stacked features with the image from the camera to form a composite image. The method also includes displaying the composite image on a display.

[0003] The present invention provides a system for providing an augmented reality display of an object. The system includes a workstation and a camera. The workstation is configured to obtain an image from the camera, detect the position of the camera, and identify the field of view of the camera based on the position of the camera. The workstation is also configured to generate one or more stacked features corresponding to the field of view. The workstation is also configured to synthesize the one or more stacked features with the image from the camera to form a composite image, and display the composite image on a display.

[0004] The present invention also provides a system for providing an augmented reality display of an object. The system includes a workstation, a display, one or more tools, a camera, an endoscope, and an endoscopic camera. The workstation is configured to obtain an image from the camera and detect the position of the camera. The workstation is also configured to identify the field of view of the camera based on the position of the camera. The workstation is also configured to generate one or more stacked features corresponding to the field of view of the camera, and the one or more stacked features corresponding to the field of view of the camera are associated with the anatomical structure of the object or are associated with one or more tools. The workstation is also configured to synthesize the one or more stacked features corresponding to the field of view of the camera with the image from the camera to form a composite image. The workstation is also configured to display the composite image on the display. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] A more detailed understanding may be obtained from the following description given by way of example in conjunction with the accompanying drawings, in which:

[0006] Figure 1 is a schematic diagram of a position tracking and navigation system according to an embodiment of the present invention;

[0007] Figure 2 shows portions of an exemplary position tracking and navigation system including tools related to otolaryngology;

[0008] Figure 3 A 3D model of a head is shown for illustrating the context in which stacked features exist and / or for illustrating the manner in which at least some of the stacked features are generated according to an example;

[0009] FIG. 4A to FIG. 4D An exemplary endoscopic image is shown;

[0010] FIG. 5A to FIG. 5B An exemplary image is shown that includes a laminated feature type referred to herein as an "anatomical feature outline"; and

[0011] Figure 6 is a flow chart of a method of displaying an augmented reality image according to an example. DETAILED DESCRIPTION

[0012] Physicians who perform invasive procedures require accuracy and precision when working with surgical tools. Surgical procedures are increasingly becoming minimally invasive, with physicians operating with cameras to view the surgical site and guiding their tools through eyepieces or video displays. Ideally, physicians should be able to perform invasive procedures while being able to observe both a real-time image of the patient on a single display device and additional data that is critical to medical decisions regarding the manipulation of the surgical tools and the next surgical step.

[0013] The augmented reality navigation system of the present disclosure provides tool position visibility for invasive procedures by using position sensors included on a camera and / or on a tool used during the procedure. A position tracking system determines and monitors the position of the tool and camera based on characteristics of signals detected by the sensor (such as the amplitude or frequency of an electromagnetic sensor). A registration process relates a 3D model of the patient (e.g., generated via a medical scan such as a computed tomography ("CT") scan) to the actual position of the patient. Thus, the position of the tool and camera is tracked relative to the position of the patient's anatomical structure. This relative position tracking allows information related to the procedure to be synthesized with an image generated by the camera. For example, since the position of the camera relative to the tool is known, the position of the tool can be displayed within the image even when the tool is not visible to a human operator due to being obscured by the patient's tissue. This display is achieved by a standard three-dimensional ("3D") rendering technique that renders an image of the position of an object relative to the camera. More specifically, through the functionality of the tracking system, the position and orientation of the camera and tool are known relative to the position of the patient. Therefore, the position of the tool is known relative to the field of view of the camera. Standard 3D rendering techniques display images of 3D objects based on the position of the object relative to the camera. Such techniques may be used to reproduce an image corresponding to a tool (e.g., a graphical representation of the tool), and additional synthesis techniques may be used to display such a reproduced image on an image of the patient received from a camera. Additional information such as visualization of the tool, reproduction of anatomical structures, including a three-dimensional structure of anatomical structures such as arteries or vascular networks, display of neural networks, or the boundaries of a tumor to be resected identified before surgery or other information may additionally or alternatively be shown in such a synthesized image.

[0014] One prior art related to the present disclosure is the CARTO TM System. CARTO TM Aspects of the system and other related technologies can be found in U.S. Patents 5,391,199, 6,690,963, 6,484,118, 6,239,724, 6,618,612 and 6,332,089, in PCT Patent Publication WO 96 / 05768, and in U.S. Patent Application Publications 2002 / 0065455A1, 2003 / 0120150A1 and 2004 / 0068178A1, the disclosures of which are all incorporated herein by reference.

[0015] Figure 1 is a schematic diagram of a position tracking and navigation system 100 according to an embodiment of the present invention. Figure 1As shown, the system includes one or more medical tools 10 (e.g., catheters, guidewires, endoscopes, etc.), a camera 44, a console 38 including at least a workstation 12, the workstation including at least a processor 14 and one or more display devices 16 (such as monitors or virtual reality displays), a catheter hub 18, a positioning pad 30, a position alignment tool 40, and a positioning pad driver ("LP driver") 20.

[0016] The workstation 12 is configured to communicate with the LP driver 20 via a link 50 so that the LP driver 20 communicates via a link 54 to drive a field generator within the positioning pad 30. The field generator transmits a field signal (e.g., an electromagnetic field or other type of field such as an acoustic field) detected by the sensor 32. The sensor 32 generates a response signal in response to the field signal. The response signal is sensed by the catheter hub 18. The catheter hub 18 communicates with the sensor 32 on one or more tools 10, cameras 44, and position alignment tools 40 via a communication link 42. The communication link 42 can be a wired or wireless link. The catheter hub 18 transmits the response signal or a processed version of the response signal to the workstation 12 via a link 52 (which can be a wired or wireless link).

[0017] The workstation 12 determines the position and physical orientation of the sensor 32 based on the characteristics of the response signal, and therefore determines the position and physical orientation of the object (e.g., the camera 44, one or more tools 10, and the position registration tool 40) in which the sensor 32 is incorporated or attached. In one example, the field generators in the positioning pad 30 each have a known relative position. The sensor 32 receives signals from multiple field generators. The received signals can be distinguished based on time (e.g., different field generators are driven at different times so that the time when the sensor 32 receives the signal can be related to the different field generators), frequency (e.g., different field generators are driven with signals of different frequencies so that the frequency of the signal received by the sensor 32 identifies a single field generator), or based on other characteristics of the signals generated by the field generators.

[0018] As described above, the catheter hub 18 transmits the signal (or a processed version of the signal) received from the sensor 32 to the workstation 12 for processing. The workstation 12 processes the signal to determine the position of the sensor 32 relative to the field generator of the positioning pad 30. The processing performed to determine the position of the sensor 32 depends on the type of signal emitted by the field generator. In some examples, the processing determines the amplitude of the signal received in response to each of the field generators. A larger amplitude indicates a smaller distance to the field generator, while a lower amplitude indicates a larger distance to the field generator. Using the distance determination of multiple field generators for each sensor 32 (e.g., 3), the position relative to the field generator can be determined by triangulation. In an alternative, the field generator of the positioning pad 30 replaces the sensor, and the sensor on the tool replaces the field generator. In such an alternative, the sensor of the positioning pad 30 detects the signal emitted by the tool, and the workstation 12 processes these signals in a similar manner, as if the positioning pad 30 includes a field generator and the tool includes a sensor. Alternatively, any technically feasible technology for determining the position based on the response to the signal can be used. Furthermore, although a particular system for determining the position of a tool has been described, any technically feasible means for determining the position of a tool may be used.

[0019] As described above, the system 100 also includes a position registration tool 40, which can be embodied as a handheld thin stick. The position registration tool 40 is used to associate the three-dimensional model stored in the model data 48 with the position of the field generator in the positioning pad in the registration procedure. In one example, the 3D model of the model data 48 is a computer data representation of the surgical object (e.g., the patient's head 22). This 3D model can be obtained by medical imaging techniques such as computed tomography ("CT") scanning or magnetic resonance imaging ("MRI"), or by any other imaging technology that produces data that can be converted into a 3D model. In order to perform the registration procedure, the position registration tool 40 is placed at a specific location near the object (e.g., the head 22). The workstation 12 then associates the location with the location within the 3D model stored in the model data 48, thereby correlating the point in the 3D model with the point in reality. This association can be made in response to a specific instruction by an operator such as a surgeon. In such a case, the workstation 12 displays the 3D model from the model data 48 on the display 16. The operator moves the position registration tool 40 to a specific position, and then indicates the corresponding position in the 3D model to the workstation 12 via the input device 36. In an alternative, the workstation 12 automatically associates the position in the real space with the position in the 3D model. In one example, this automatic association is completed as follows. The position registration tool 40 moves around the vicinity of the surgical object (e.g., the head 22). The workstation 12 processes the data received from the position registration tool 40 to identify the corresponding position in the 3D model. In one example, the position registration tool 40 includes a camera, and the workstation 12 performs image processing on the image received with the position registration tool 40 to identify the position of the position registration tool 40 in the 3D model. In some examples, multiple related points between the real space and the 3D model are obtained and stored to improve the accuracy of the registration and to achieve rotation and position registration. Although the position registration tool 40 is described as a tool for achieving position registration between the real space and the 3D model, any other tool can be used alternatively, including tools for other purposes (such as any one of the camera 44 or the tool 10).

[0020] The camera 44 provides the operator with a view in the display 16. The view includes an actual image of the subject (represented by the head 22). More specifically, the camera 44 takes a series of images of the subject (head 22) and displays such images on the display 16.

[0021] In addition, the workstation 12 can overlay specific features on the image provided by the camera 44. More specifically, the camera 44 includes a position sensor 32, which is combined with the positioning pad 30 (or alternatively combined with some other mechanism), and cooperates with the catheter hub 18 to provide a signal from which the position and orientation of the camera 44 can be provided to the workstation 12. This position information can be derived in a similar manner as described above with respect to the cooperation of the sensors 32 on the positioning pad 30 and the tool 10 and the position registration tool 40. The rotation of the camera 44 can be detected in any technically feasible manner. In one example, a plurality of different micro sensors are included in the sensor 32 of the camera 44, and relative measurements made with the different micro sensors are used to determine the rotation of the camera 44.

[0022] The workstation 12 uses the position and rotation information obtained for the camera 44 to provide one or more overlay features on the display 16, which are synthesized with the image generated by the camera. More specifically, based on the position and orientation of the camera 44, the workstation 12 is able to position one or more virtual objects (e.g., anatomical labels) in the three-dimensional space of the 3D model of the model data 48. Due to the link between the 3D space of the 3D model and the real-world coordinate system (established via a position registration process), the virtual objects (also referred to herein as "overlay features") connected to the 3D space of the 3D model can be displayed in the image captured by the camera.

[0023] Conventional 3D imaging techniques can be used to display some of these laminated features in an image captured with a camera. In a simple example, the workstation 12 has the position of the camera 44 in real space, and has related the geometry of the 3D model to the real space via the above-mentioned registration process. Therefore, the workstation 12 knows the relative position of the 3D model and the camera 44 in the real space coordinate system. Given the position of the object in the coordinate system and the position of the camera in the coordinate system, conventional 3D rendering techniques such as those implemented in standard commercially available 3D rendering graphics cards (such as GeForce series cards purchased from Nvidiacorporation of Santa Clara CA, or Radeon series cards purchased from Advanced Micro Devices, Inc. of Sunnyvale, CA) can draw three-dimensional objects to screen space. Such techniques can be used to reproduce a 3D model of model data 48 in screen space displayed on the display 16. In some operating modes, the workstation 12 reproduces the 3D model of model data 48 and / or other laminated features discussed below on the image provided by the camera 44. Thus, in such an operating mode, the workstation 12 generates an image including an overlay based on the 3D model, which is synthesized with the image generated by the camera 44. In some operating modes, the workstation 12 does not display any part of the 3D model, displays some parts or all parts, and does not display other overlay features or displays some other overlay features. Overlay features include information related to the procedure performed on the object (e.g., head 22) and / or information related to the anatomical structure of the object. Overlay features include user (e.g., surgeon) generated marks for marking specific locations in the three-dimensional space of the object. Overlay features alternatively or additionally include computer-generated features generated based on analysis of images obtained via the camera 44. Overlay features may include any other data that can be displayed on the display 16, which indicates information related to the procedure performed on the object or any other aspect of the use of the system 100. In one example, overlay features include marks predefined by the surgeon as points of interest, as anatomical structure identifiers, as specifically defined text, or as any other features.

[0024] In another example, the stacked feature includes a graphical representation of one or more tools 10. More specifically, the workstation 12 has the position of the sensor 32 on the tool relative to the camera 44. The workstation 12 can use similar techniques as described above to reproduce the graphical representation of the tool in the image generated by the camera 44. In addition to the position of the tool 10, the way the graphical representation is reproduced can also depend on the orientation (e.g., rotation) of the tool 10 so that the physician understands the geometric position of the tool 10 relative to the patient's anatomy.

[0025] The display 16 may be a conventional display or may be virtual reality glasses. The virtual reality glasses may have appropriate shielding such as X-ray shielding provided by lead if the procedure involves imaging suitable for such shielding. The display 16 may also be a remote display (i.e., a display located far from the patient), which may facilitate remote surgery in conjunction with the remote control tool 10.

[0026] Additional details of the system 100 will now be described with reference to the remaining figures. These additional figures show aspects of the system 100 in the context of otolaryngology procedures, although it should be understood that the general principles described may be applied to other areas of the human body.

[0027] Figure 2 2 shows a portion of an exemplary position tracking and navigation system 200 including tools related to otolaryngology. System 200 is an example of system 100 in which one of tools 10 is an endoscope 202. Position tracking and navigation system 200 includes a system similar to Figure 1 The display 16 of the display 16, and similar Figure 1 The location tracking and navigation system 200 also includes

[0028] Figure 1 other components of system 100, even if such components are not Figure 2 Although system 200 is described as including both endoscope 202 and camera 44, only endoscope 202 may be used without camera 44.

[0029] Figure 2 One of the tools 10 is an endoscope 202, which includes a probe 206 that obtains an image at a distal end 208 and provides the image to a camera 204 connected to the endoscope 202. The camera 204 transmits the image to the workstation 12, which processes the image for display. The workstation 12 can display the image obtained using the camera 204 integrated with the endoscope 202 on a display showing the use of Figure 1The display of images obtained by camera 44 may be shown in the same display (e.g., on display 16) as the display of images obtained by camera 44, or the images may be shown in a separate endoscope display 205. If shown on display 16, the workstation 12 performs operations so that display 16 may be shared between camera 204 and camera 44. In one example, in response to input from an operator, display 16 switches between showing images from camera 44 and camera 204. For example, when camera 204 is powered off, display 16 may show images from camera 44, and when camera 204 is powered on, display 16 may show images from camera 204. Alternatively, selection of camera 44 or camera 204 may occur in response to an explicit camera selection input that explicitly selects which camera to show images. If endoscope display 205 is used to show images from camera 204 of endoscope 202, there is no competition between the displays—display 16 shows images from camera 44, and endoscope display 205 shows images from camera 204 of endoscope 202. By Figure 1 The image generated by camera 44 including the stacked features is overview image 210 (shown on display 16 ).

[0030] As with the camera 44, the workstation 12 obtains the image obtained by the endoscope 202 from the camera 204 and displays the image on the display along with an endoscope overlay feature. The endoscope overlay feature includes information related to the procedure being performed and / or the anatomy of the subject (e.g., the head 22). Also, as with the system 100, one or more tools 10 (e.g., surgical tools) may be present in the vicinity of the subject (e.g., the head 22). Figure 2 One tool 10 is shown with a sensor 32. Also, at the end of the probe 206 of the endoscope 202 is the sensor 32. The workstation 12, together with the positioning pad 30 and the positioning pad driver 20, detects the position and orientation of the sensor 32, and thus the tool 10 and the endoscopic probe 206. It should be understood that Figure 2 One exemplary embodiment is shown, and any number of tools 10, each having a sensor 32, may be used in a procedure. The image generated by the camera 204 and including the endoscopic overlay features comprises an endoscopic image 212, which may then be displayed on either the display 16 or the endoscopic display 205.

[0031] The endoscope overlay features include an indication of the position of a tool (such as the tool 10 shown or other tools), an indication of the location or position of an anatomical structure, or other indications related to a procedure or an anatomical structure. In some operating modes, the workstation 12 processes the position data generated based on the sensor 32 and uses the position data to generate overlay features of the endoscopic image 212. Such overlay features include visual images indicating the position and orientation of the tool 10. As described with respect to Figure 1 As with the process described above, the workstation 12 uses the position of the sensor 32 (determined using the positioning mat 30) and the position and orientation of the distal end 208 of the endoscopic probe 206 (also determined using the positioning mat 30) to determine the position and shape of the stacked features in the endoscopic image 212. Figure 1 As described above, given the camera position and orientation and the object position and orientation, a standard three-dimensional reconstruction algorithm can generate an image. Using such an algorithm, the workstation 12 can generate images corresponding to the stacked features of different tools based on the position of the tool 10 relative to the distal end 208 of the endoscopic probe 206. The sensor 32 on the endoscopic probe 206 is positioned at a known position relative to the distal end 208 of the endoscopic probe 206, so that the position of the distal end 208 of the endoscopic probe 206 can be derived from the position data obtained by the sensor 32 on the endoscopic probe 206.

[0032] In addition to tool position and orientation, in some modes of operation, the overlay features displayed by the workstation 12 in the endoscopic image 212 also include text and graphical indicia related to the anatomical structure. Figure 3 A 3D model 300 of a head is shown for illustrating the context in which stacked features exist and / or for illustrating the manner in which at least some of the stacked features are generated, according to an example. The 3D model 300 may be obtained by standard medical imaging techniques such as CT scans or MRIs, and data obtained by these techniques may be converted into a three-dimensional model using known techniques. Such scans may be used in a computer program product. Figure 1 System 100 or Figure 2 For the purpose of simplicity and clarity, only some anatomical structures are shown in the 3D model 300. However, it should be understood that the 3D model generated based on the scan and used for medical procedures can substantially reflect all anatomical structures of the object.

[0033] In some modes of operation, at least some of the overlay features synthesized onto the endoscopic image 212 are based on markers 302 input by a human operator. More specifically, prior to the procedure, a 3D model of the object will be generated, for example, based on a medical scan. Subsequently, a human operator, such as a surgeon, will view the 3D model and will input markers 302 within the three-dimensional coordinate system of the 3D model. Any user interface may be used to allow a human operator to input markers 302. Some examples of these markers 302 are shown in FIG. Figure 3 Marker 302(1) is associated with the maxillary sinus ostium, while marker 302(2) is associated with the sphenoid sinus ostium. Other markers 302 may also be present in the 3D model.

[0034] Note that the computer system where the markings 302 are input into the 3D model need not be the workstation 12. In other words, the 3D model may be analyzed and manipulated (markings 302 are added) by a human operator on a computer system that is different from the computer system (workstation 12) that generates images (overview image 210, endoscopic image 212) for viewing by the human operator during the procedure. For this reason, while the workstation 12 is sometimes described herein as performing specific tasks related to adding markings 302 to the 3D model 300, or performing other tasks related to adding, removing, editing, or otherwise manipulating data of stacked features, it should be understood that such actions may alternatively be performed using a different computer system.

[0035] In some operating modes, the workstation 12 automatically generates the labels. Some labels can be added based on the analysis of the 3D model 300 and based on well-known anatomical structure related information. For example, the workstation 12 can generate labels 302 for one or more specific anatomical structures based on a template 3D model (which includes a general three-dimensional model of a human head) and based on additional analysis, such as a comparison of the 3D model of the human object being operated with the template model. In some examples, the labels 302 are both automatically generated and input by humans. More specifically, the workstation 12 automatically generates the labels, and the human adjusts the position of these labels. Alternatively, the workstation 12 presents a list of anatomical landmarks for which the workstation 12 can automatically generate labels 302. In response, the human operator selects one or more such landmarks, and the workstation 12 generates labels 302 based on these selected landmarks. The human operator can then adjust the position of these labels based on the actual structure of the object's head. Any other technically feasible means for generating such labels 302 is also possible. The labels 302 may also include associated text, which is input by the human operator. These text labels may be generated during the input of the label 302 itself. For example, when the human operator selects a point in the 3D model as the label 302, the human operator may also input a text label.

[0036] Some of the overlay features, such as at least some of the markers 302, are displayed during the procedure in either or both of the overview image 210 or the endoscopic image 212. As described above, because the workstation 12 knows the position of the camera 44, the position of the distal end 208 of the endoscopic probe 206, and the position of the subject's anatomy, the workstation 12 is able to reproduce indicators for one or more of the sensor markers 302 based on the registration procedure.

[0037] FIG. 4A to FIG. 4C An exemplary endoscopic image 212 is shown. In image 400(1), the endoscopic probe 206 is inserted into the nasal cavity pointing toward the maxillary sinus ostium and is visible in image 400(1). The 3D model includes a marker 302 at the maxillary sinus ostium. By analyzing the position of the marker 302 in comparison with the position of the distal end 208 of the endoscopic probe 206, the workstation 12 determines that the marker 302(1) is within the field of view of the endoscopic camera 204. In response, the workstation 12 displays an overlay feature 404(1) associated with the marker 302 in the endoscopic image 212. In image 400(1), the overlay feature includes an arrow 406(1) and text 408(1), however, in other embodiments, the overlay may include other graphical features. The graphical features of the overlay are configured to indicate the location of the marker 302. In image 400(1), this indication is via an arrow 406 extending from the approximate location of marker 302 to text 408 associated with marker 302. Other modes of operation may include different types of graphics configured to illustrate the location of marker 302 in different ways.

[0038] In a configuration in which the position of the mark 302 is indicated by an arrow 406 and text 408, the workstation 12 is configured to select an appropriate position for the text 408, and then draw an arrow 406 from the position corresponding to the mark 302 to the position selected for the text 408. Any technically feasible device for analyzing an image to determine an appropriate position for the text can be used. The term "appropriate" generally means that the text 408 is readable. In one example, a portion of the image 404 with a roughly uniform color is selected for the text. In one example, the roughly uniform color is determined based on the variance of the average color values ​​of the pixels in the area. The higher the variance, the more uneven the area. In one example, an area with a variance below a certain threshold indicates that the area is suitable for text. It should be noted that the above-mentioned technology represents some examples of technologies for displaying overlays, and any technically feasible device for displaying features of the position of the identification mark 302 can be used.

[0039] Figure 4B and Figure 4COther exemplary images 400 at other locations are shown. Specifically, image 400(2) represents a view taken by endoscope 202, wherein endoscopic probe 206 is inserted into the nasal cavity and directed toward the sphenoid sinus ostium. Based on marker 302(2), workstation 12 uses a similar method to that described with respect to Figure 4A Text 408(2) and arrow 406(2) are generated in the manner described. Image 400(3) represents a view taken by endoscope 202, wherein endoscopic probe 206 is inserted into the nasal cavity and directed toward a nasal polyp. As in the case of images 400(1) and 400(2), workstation 12 generates image 400(3) based on position data from sensor 32, based on the 3D model, and based on marker 302.

[0040] Figure 4D Another exemplary image 400(4) is shown. Specifically, image 400(4) represents a view captured by endoscope 202, wherein endoscopic probe 206 is inserted into the nasal cavity and directed toward a nasal polyp. Based on marker 302(4), workstation 12 generates text 408(4) and arrow 406(4). In addition, workstation 12 has information indicating the shape and location of blood vessel 460. Workstation 12 determines that the blood vessel is located within the field of view corresponding to the image captured by endoscope 202, and therefore displays a graphical representation of blood vessel 460 in image 400(4). This allows the practitioner to avoid blood vessel 460 during the procedure.

[0041] FIG. 5A to FIG. 5B An exemplary image 500 is shown that includes a type of laminated features referred to herein as "anatomical feature outlines." These anatomical feature outlines 502 are automatically generated by the workstation 12 and are used to indicate the location of certain anatomical features. Image 500(1) shows arrows 406(4) and text 408(4), as well as outlines 502(1). Outlines 502(1) are automatically generated by the workstation 12 through image processing, possibly in combination with position information received from the sensor 32 and 3D model information stored in the model data 48.

[0042] The workstation 12 may perform any technically feasible method for identifying the geometric outline of a particular anatomical feature of interest. In one example, a template model of an "ideal" or "reference" anatomical structure is compared to a 3D model obtained from an actual scan of the object. Based on a positional comparison between the geometry of the 3D model and the geometry of the template model, known anatomical features are identified within the 3D model. In another example, after scanning to generate the 3D model, a human operator observes the 3D model and identifies the anatomical structure of interest in a manner similar to generating the marker 302.

[0043] During the procedure, the workstation 12 processes images obtained using the endoscopic camera 204 to identify pixels of the image associated with specific identified anatomical structures. In some examples, different anatomical structures are identified using different image processing techniques. In one technique, the workstation 12 identifies a dark area in an otherwise bright image (taking into account illumination-based and / or camera-based vignetting) as an orifice. The workstation 12 determines which orifice to identify based on the position and orientation of the distal end 208 of the endoscopic probe 206. For example, based on the position within the 3D model, the workstation 12 may be able to identify that the distal end 208 of the endoscopic probe 206 is within a particular area such as the nasal cavity, and based on the position of the distal end 208 of the endoscopic probe 206 within the nasal cavity, identify that the identified orifice is the maxillary sinus orifice. In response, the workstation 12 draws a contour around the darkened area and provides a label identifying the contour as corresponding to the maxillary sinus orifice.

[0044] In another technique, the workstation 12 identifies areas of a particular color as a particular type of anatomical structure. Figure 5B In the image, the slightly yellow structure is identified as a nasal polyp based on its color and position. Specifically, the workstation 12 determines that the distal end 208 of the endoscopic probe 206 is located within the nasal cavity based on the position determined via interaction with the sensor 32, and therefore determines that the yellow object in the image captured by the endoscopic camera 204 is a nasal polyp.

[0045] It should be noted that although certain stacking features are described herein, the present disclosure is not limited to these specific stacking features. Other stacking features that are technically drawable may be displayed alternatively or additionally. The display of the stacking features may be adjusted in position, form, and / or format based on image analysis of the images provided by the endoscope camera 204.

[0046] Figure 6 is a flow chart of a method 600 for displaying an augmented reality image according to an example. Figures 1 to 5B The system described is described, but those skilled in the art will understand that any system configured to perform the method in any technically feasible sequence of steps falls within the scope of the present disclosure. The method 600 shown can be performed for either or both of the camera 44 or the endoscopic camera 204, or for any other camera included in the system 100. Figure 6 In the discussion of FIG. 1 , the term “camera” without a reference number refers to any of these cameras.

[0047] As shown, method 600 begins at step 602, where workstation 12 detects motion of the camera. The motion is detected via positioning mat 30, which transmits a signal that is received by one or more sensors 32 attached to the camera, as described above. Motion detection corresponds to detecting that the position of the camera is different from the immediately previous position.

[0048] At step 604, the workstation 12 determines the position of the camera. Again, this position detection is based on the signal received by the sensor 32 in response to the signal transmitted by the positioning mat 30. Various aspects of the signal relate to the position and orientation of the camera and can therefore be used to determine the position and orientation of the camera. For example, multiple transmitters within the positioning mat 30 can transmit different signals, each with different distinguishing characteristics (such as frequency). The sensor 32 detects the amplitude of the signals of different frequencies. The amplitude is associated with the distance from each of the transmitters within the positioning mat 30. Each transmitter has a known position in the positioning mat. Multiple distances are used to triangulate the position of the sensor 32. The sensor 32 may have individual microsensors that each receive slightly different signals. The difference in the signals received by the microsensors can be used to determine the orientation of the sensor 32. Via this analysis, the position and orientation of the sensor 32 relative to the positioning mat 30 is determined. The position and orientation of the positioning mat 30 relative to the object (e.g., the patient's head) was previously established based on a registration procedure, in which an operator, such as a human operator, moves the tool 10 with the sensor 32 around the object and correlates one or more points on the object with points in the 3D model of the object, substantially as described above.

[0049] At step 606, the workstation 12 determines the field of view of the camera. An image generated by a camera is associated with a particular field of view both in the actual image (i.e., the image actually generated by the camera) and in the 3D coordinate system corresponding to the 3D model and the sensor 32 measurements. Any particular camera system has a known field of view, which is typically measured in degrees and is determined based on the characteristics of the lens and the sensor. An additional parameter associated with the field of view is shape, some examples of which are rectangular and circular. Because the field of view is known based on the physical characteristics of the camera and sensor, determining the field of view of a camera involves looking up the camera's field of view from stored data, calculating the field of view based on other known parameters of the camera, or identifying the field of view by other known techniques - any technically feasible technique may be used.

[0050] At step 608, the workstation 12 identifies an overlay feature corresponding to the determined field of view. As described above, an overlay feature includes a graphical element associated with a specific location in the 3D space of the model stored in the model data 48. One type of overlay feature is a marker 302 associated with a specific point in the 3D space of the model. Another type of overlay feature includes an automatically generated indicator based on anatomy, such as FIG. 5A to FIG. 5B 502 of anatomical features. This list of stacked features should not be viewed as limiting. Any type of features or aspects related to a procedure and / or anatomy may be identified for synthesis into an image to be viewed (e.g., overview image 210 or endoscopic image 212).

[0051] At step 610, the workstation 12 composites the overlay features with the camera image and displays the composite image. Which display the workstation 12 displays the positive image on may depend on which camera the image is from and whether another camera of the system 100 is active and providing image data to the workstation 12 for display. In one example, if both the camera 44 and the endoscope camera 204 are active and providing images for composite, the workstation 12 displays the image from the camera 44 on the display 16 as the overview image 210 and the image from the endoscope camera 204 on the endoscope display 205 as the endoscope image 212. In another example, if both the camera 44 and the endoscope camera 204 are active and providing images for composite, the images are both displayed on the display 16 using time-based switching between the images or control by a human operator to select which image. In another example, both images are displayed on a specific display, such as the display 16 or the endoscope display 205. In such cases, the specific image displayed may be controlled by a human operator. Alternatively, one camera may take priority over the other cameras so that if one camera is on, the display always shows images from that camera and not images from the other cameras.

[0052] The overlay feature synthesis includes positioning the overlay features identified at step 608 within an image associated with the camera and synthesizing the overlay features within the image received from the camera. Synthesizing the overlay features includes generating a graphical representation of the overlay features and displaying the graphical representation with the image from the camera (e.g., over the image from the camera or blending pixels with pixels of the image from the camera). For the mark 302, the graphical representation may include, for example, FIG. 4A to FIG. 4C The generated arrows 406 and text 408. Other stacking features may include FIG. 5A to FIG. 5B The generated anatomical feature contours. Some of the overlay features include a visual indication of the tool 10. Specifically, the workstation 12 knows the position of the camera in the 3D space of the model and knows the position of the tool 10 in the 3D space of the model. If the tool is within the field of view of the camera, the workstation 12 can generate a graphical indicator of the tool to be synthesized into the image generated by the camera.

[0053] The result following step 610 is an image from the camera superimposed with one or more graphical features derived from data related to the 3D model and / or other stored data related to the anatomy or procedure being performed. The composite image may be displayed on the display 16 or the endoscope display 205.

[0054] The methods provided include specific implementations in a general purpose computer, a processor, or a processor core. By way of example, suitable processors include a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and / or a state machine. Such processors can be manufactured by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediate data including a network table (such instructions can be stored on a computer readable medium). The results of such processing can be mask work, which is then used in a semiconductor manufacturing process to manufacture a processor that implements the methods described herein.

[0055] The method or flow chart provided herein can be implemented in a computer program, software or firmware incorporated into a non-transient computer-readable storage medium for execution by a general-purpose computer or processor. Examples of non-transient computer-readable storage media include ROM, random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs).

Claims

1. A system for providing an augmented reality display of an object, the system comprising: workstation; and camera, The workstation is configured as follows: obtaining an image from the camera; detecting a position of the camera; identifying a field of view of the camera based on the position of the camera; generating one or more stacked features corresponding to the field of view; combining the one or more laminated features with the image from the camera to form a composite image; as well as displaying the composite image on a display; wherein the one or more overlay features are associated with an anatomical structure of the subject, wherein the workstation is configured to generate the one or more overlay features by: generating an overlay feature indicative of an anatomical feature; wherein the workstation is configured to generate the stacked feature indicative of the anatomical feature by: determining a position of a marker within the field of view of the camera based on the relative position and orientation of a locator mat and the position and orientation of the object, and based on the relative position of a marker and a model of the object, wherein the marker is a point in the model of the object associated with the anatomical feature; as well as rendering a graphical representation corresponding to the marker based on the position of the marker within the field of view of the camera; wherein the graphical representation corresponding to the marker comprises text at a selected location on the image and an arrow indicated by the marker, the selected location being based on a variance of average color values ​​for a plurality of pixels in an area of ​​the image being below a predetermined threshold, wherein the text indicates a corresponding anatomical feature and the arrow indicates the location of the marker within the field of view of the camera.

2. The system of claim 1, further comprising a positioning pad, wherein the workstation is configured to detect the position of the camera by: performing a pre-procedural registration to correlate the relative position and orientation of the locator pad to the position and orientation of the subject; receiving one or more signals with a sensor of the positioning mat, the one or more signals generated by one or more field generators coupled to the camera; as well as The position of the camera is identified based on the one or more signals.

3. The system of claim 1 , wherein the workstation is configured to generate the one or more stack-up features by: Generates a graphical representation of the tool.

4. The system of claim 3, wherein the workstation is configured to generate the graphical representation of the tool by: determining a position of the tool within the field of view of the camera; and The graphical representation of the tool is rendered based on the position of the tool within the field of view of the camera.

5. The system of claim 1, wherein the workstation is further configured to: identifying a mouth based on detecting a darkened portion of the image, Wherein the graphical representation corresponding to the marking comprises an outline of an anatomical feature.

6. A system for providing an augmented reality display of an object, the system comprising: workstation; monitor; one or more tools; camera; Endoscope; and Endoscopic Camera, The workstation is configured as follows: obtaining an image from the camera; detecting a position of the camera; identifying a field of view of the camera based on the position of the camera; generating one or more stacked features corresponding to the field of view of the camera, the one or more stacked features corresponding to the field of view of the camera being associated with an anatomical structure of the object or associated with the one or more tools, wherein the workstation is configured to generate the one or more stacked features by: generating stacked features indicative of anatomical features; combining the one or more laminate features corresponding to the field of view of the camera with the image from the camera to form a composite image; as well as displaying the composite image on the display; wherein the workstation is configured to generate the stacked feature indicative of the anatomical feature by: determining a position of a marker within the field of view of the camera based on the relative position and orientation of a locator mat and the position and orientation of the object, and based on the relative position of a marker and a model of the object, wherein the marker is a point in the model of the object associated with the anatomical feature; as well as rendering a graphical representation corresponding to the marker based on the position of the marker within the field of view of the camera; wherein the graphical representation corresponding to the marker comprises text at a selected location on the image and an arrow indicated by the marker, the selected location being based on a variance of average color values ​​for a plurality of pixels in an area of ​​the image being below a predetermined threshold, wherein the text indicates a corresponding anatomical feature and the arrow indicates the location of the marker within the field of view of the camera.

7. The system of claim 6, wherein the workstation is further configured to: obtaining an image from said endoscopic camera; detecting a position of the endoscopic camera; identifying a field of view of the endoscopic camera based on the position of the endoscopic camera; generating one or more overlay features corresponding to the field of view of the endoscopic camera, the one or more overlay features corresponding to the field of view of the endoscopic camera being associated with anatomy of the subject or with the one or more tools; synthesizing the one or more stacked features corresponding to the field of view of the endoscopic camera with the image from the endoscopic camera to form a synthesized endoscopic image; as well as The composite image is displayed on the display or on an endoscope display.