Vision system for use in a surgical environment - Patent Application 20070122997

Abstract

The present invention relates to a vision system for use in a surgical environment. Various real object detection devices detect the location of real objects in the real environment, such as the patient and their body parts, medical staff, robots, cutting tools on the robot, implants transferred by the robot into the body part, surgical tools, and disposable items. A map generator generates a map that forms a digital representation or digital twin of the real environment. Various guidance modules, including a room setup module, an anatomical structure registration module, a surgical planning module, and a surgical execution module, utilize the digital representation and navigate virtual or real objects based on the digital representation.

Description

[Technical Field]

[0001] This application claims priority to U.S. Provisional Patent Application No. 62 / 936,383, filed November 15, 2019, the entirety of which is incorporated herein by reference in its entirety.

[0002] The present invention relates generally to vision systems, and more particularly to vision systems that function within a surgical environment. [Background technology]

[0003] Performing a surgical procedure within a surgical environment, such as a hospital or clinic, requires entirely different contributions from a variety of people, including personnel involved in setting up the room for the surgery, radiology personnel involved in recording radiology data of the patient's body parts, surgeons and other specialists who collaborate to plan the surgery using visualization of the radiology data, surgeons and other surgical staff who perform the surgery, and personnel involved in ventilating the room and replenishing disposable items.

[0004] Radiology data visualization is a static visualization based on previously collected radiology data. Solutions for interacting with the visualization and planning a surgical procedure typically do not exist. After the planning phase is completed, the radiology data is typically not used to perform the surgical procedure. Robots are frequently used to perform surgical procedures. These robots are digitally connected to operators, such as remote surgeons, who can steer the robots over long distances and perform the surgical procedure, including making incisions and placing implants into the human body. These remote operators rely on vision of both the robot and the human body to steer the robot.

[0005] Without a digital twin of the real environment, it is very difficult to achieve any degree of reproducibility or high accuracy in room setup, anatomical alignment, surgical planning, and surgical execution. Summary of the Invention [Means for solving the problem]

[0006] The present invention provides a vision system including a real object detection device positioned to detect locations of real objects in a real environment, at least one processor, a computer-readable medium connected to the processor, a data storage device on the computer-readable medium, and a set of instructions stored on the computer-readable medium and executable by the processor. The set of instructions may include a map generator connected to the real object detection device and executable to receive data of the real environment including the real objects and create a map that forms a digital representation of the real environment including the real objects, a map storage routine executable to store the map on the data storage device, and a guidance module connected to the data storage device and executable to read the map and guide at least one of the virtual object and the real object based on the digital representation.

[0007] The present invention also provides a visualization method including: detecting a location of a real object in a real environment using a real object detection device; executing, using a processor, a map generator connected to the real object detection device to receive data of the real environment including the real object and form a digital representation of the real environment including the real object to create a map; executing, using the processor, a map storage routine to store the map on a data storage device; and executing, using the processor, a guidance module connected to the data storage device to read the map and guide at least one of the virtual object and the real object based on the digital representation.

[0008] The present invention further provides a vision system including a real object detection device positioned to detect locations of real objects within a real environment, at least one processor, a computer-readable medium connected to the processor, a data storage device on the computer-readable medium, and a set of instructions stored on the computer-readable medium and executable by the processor, the set of instructions including: a map generator connected to the real object detection device and executable to receive data of the real environment including the real objects and create a map to form a digital representation of the real environment including the real objects; a map storage routine executable to store the map on the data storage device; a guidance module connected to the plurality of data storage devices and executable to read the map and guide at least one of the virtual objects and the real objects based on the digital representation, the room setup module executable by the processor to: a. set up a room based on the digital representation, the room setup module may include storing a desired room layout and providing an output that digitally superimposes the desired room layout on the real environment; and b. set up a room based on the digital representation of the real environment, the room setup module executable by the processor to a surgical planning module executable by the processor to plan a surgical procedure based on the digital representation, the surgical procedure planning module comprising: storing a digital representation of the patient's body part; displaying the digital representation of the patient's body part along with a virtual object to a user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; and moving the virtual object relative to the digital representation of the body part within the user's view in response to the input from the user; and a surgical planning module executable by the processor to plan a surgical procedure based on the digital representation, the surgical procedure planning module comprising: storing a digital representation of the patient's body part; displaying the digital representation of the patient's body part along with a virtual object to a user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; and moving the virtual object relative to the digital representation of the body part within the user's view in response to the input from the user.and a guidance module executable by the processor to assist in performing a surgical procedure based on the digital representation, the guidance module including at least two of: storing the digital representation of the patient's body part; receiving input from a user and guiding a virtual object relative to the digital representation of the body part; and a surgical procedure execution module that may include, in response to the input from the user, moving the virtual object relative to the digital representation of the body part within the user's view and moving respective ones of the real objects relative to the patient's body part within the real environment.

[0009] The present invention also includes a method for detecting locations of real objects in a real environment using a real object detection device; executing a map generator, connected to the real object detection device, using a processor to receive data of the real environment including the real objects and create a map to form a digital representation of the real environment including the real objects; executing a map storage routine, using the processor, to store the map on a data storage device; and executing a plurality of guidance modules, connected to the data storage device, using the processor to read the map and guide at least one of the virtual object and the real object based on the digital representation, wherein the guidance modules may include: a. executable by the processor to configure a room based on the digital representation, storing a desired room layout, and providing an output that digitally superimposes the desired room layout on the real environment. a surgical planning module; b. the plurality of guidance modules is an anatomical structure registration module, executable by the processor to perform anatomical structure registration based on the digital representation, and may include storing a location of a patient's body part, where the location of the body part is based on a location of a real object detected by a real object detection device; c. a surgical planning module executable by the processor to plan a surgical procedure based on the digital representation, and may include storing a digital representation of the patient's body part, displaying the digital representation of the patient's body part along with a virtual object to a user, receiving input from the user and guiding the virtual object relative to the digital representation of the body part, and moving the virtual object within the user's view relative to the digital representation of the body part in response to the input from the user; and d.and a surgical operation execution module that is executable by a processor to assist in performing a surgical operation based on the digital representation, and that may include storing a digital representation of a patient's body part, receiving input from a user and directing a virtual object relative to the digital representation of the body part, and, in response to the input from the user, moving the virtual object relative to the digital representation of the body part within the user's view, and moving respective ones of real objects relative to the patient's body part within the real environment.

[0010] Forward and reflected waves are used in the field of radiology for the purpose of imaging a patient. For example, X-ray machines and computed tomography (CT) machines use X-ray waves, ultrasound machines use ultrasound waves, and magnetic resonance imaging (MRI) machines use alternating magnetic fields or waves and radio waves in a forward and reflected manner to detect a patient's internal structures.

[0011] In the context of radiology data, the present invention further provides a vision system including a real object detection device positioned to detect the location of a real object in a real environment, at least one processor, a computer-readable medium connected to the processor, a data storage device on the computer-readable medium, and a set of instructions stored on the computer-readable medium and executable by the processor. The set of instructions may include a map generator connected to the real object detection device and executable to receive data of the real environment including the real objects and create a map that forms a digital representation of the real environment including the real objects; a map storage routine executable to store the map on a data storage device; a head-mountable frame, wherein the optical waveguide is affixed to the head-mountable frame; a raw data receiving unit that receives raw data of the reflected waves; an image generation unit connected to the data storage device and processes the raw data of the reflected waves to create image data representing an image and store the image data in the data storage device; an image data receiving unit that receives the image data from the data storage device; at least one projector connected to the image data receiving unit and that receives the image data, the projector generating light in a pattern that represents the image data and based on the map; and at least one optical waveguide connected to the projector and affixed to the head-mountable frame that directs light to the retina of the user's eye so that the user sees a rendering of the image data.

[0012] The present invention also provides a method of viewing, including using a real object detection device to detect a location of a real object in a real environment; using a processor to execute a map generator connected to the real object detection device, receiving data of the real environment including the real object, and creating a map to form a digital representation of the real environment including the real object; using the processor to execute a map storage routine and store the map on a data storage device; mounting a head-mountable frame on a head of a viewer; storing raw data of reflected waves in the data storage device; processing the raw data of reflected waves to create image data; storing the image data in the data storage device; receiving the image data from the data storage device; generating light in a pattern representative of the image data and based on the map; and using an optical waveguide affixed to the head-mountable frame to direct the light to the retina of the viewer's eye so that the viewer sees a rendering of the image data.

[0013] The present invention further provides a vision system including a real object detection device positioned to detect the location of a real object in a real environment, at least one processor, a computer-readable medium connected to the processor, a data storage device on the computer-readable medium, and a set of instructions stored on the computer-readable medium and executable by the processor. The set of instructions may include a map generator coupled to the real object detection device and executable to receive data of a real environment including real objects and create a map that forms a digital representation of the real environment including the real objects; a map storage routine that stores a first map having a plurality of anchors, each anchor of the first map having a set of coordinates; an anchor identification system coupled to the real object detection device and that detects anchors of a second map based on locations of the real objects, each anchor of the second map having a set of coordinates; and a localization module coupled to the first map and the second map and executable to localize the second map relative to the first map by matching a first anchor of the second map to a first anchor of the first map and matching a second anchor of the second map to a second anchor of the first map.

[0014] The present invention also provides a viewing method including: detecting a location of a real object in a real environment using a real object detection device; using a processor to execute a map generator connected to the real object detection device, receiving data of the real environment including the real object, and creating a map to form a digital representation of the real environment including the real object; storing a first map having a plurality of anchors, each anchor of the first map having a set of coordinates; detecting anchors of a second map based on the locations of the real objects, each anchor of the second map having a set of coordinates; and locating the second map relative to the first map by matching the first anchor of the second map with the first anchor of the first map and matching the second anchor of the second map with the second anchor of the first map. The present invention provides, for example, the following. (Item 1) 1. A vision system comprising: a real object detection device positioned to detect the location of a real object in a real environment; at least one processor; a computer-readable medium coupled to the processor; a data storage device on said computer readable medium; a set of instructions stored on the computer-readable medium and executable by the processor; a map generator connected to the real object detection device, the map generator being executable to receive data of the real environment including the real objects and to create a map forming a digital representation of the real environment including the real objects; a map storage routine executable to store the map on the data storage device; a navigation module coupled to the data storage device and executable to read the map and navigate at least one of a virtual object and a real object based on the digital representation; and A set of instructions, including A vision system comprising: (Item 2) the guidance module is a room setup module executable by the processor to setup a room based on the digital representation; storing a desired room layout; providing an output digitally superimposing the desired room layout onto the real environment; Item 1. The vision system of item 1, comprising: (Item 3) Item 3. The vision system of item 2, wherein the room setting module generates an image based on the desired room layout and superimposes the image over the real environment. (Item 4) 4. The vision system of claim 3, wherein the room setup module generates an image of a desired installation of one of the real objects within the desired room layout and superimposes the image over the real environment. (Item 5) Item 5. The vision system of item 4, wherein the room setup module provides an output indicating that individual real objects have been moved to positions consistent with the desired installation. (Item 6) Item 6. The vision system of item 5, wherein the room setting module tracks the movement of the individual real objects as the real object detection device detects the individual real objects and the map generator updates the map as the individual real objects move. (Item 7) 7. The vision system of claim 6, wherein the room setting module provides a visual output indicating that the individual real object has been moved to a position that matches the desired setting by changing a color of an image of the desired setting. (Item 8) Item 3. The vision system of item 2, wherein the room setup module provides an output that digitally superimposes the desired room exploded layout onto the real environment. (Item 9) Item 3. The vision system of item 2, wherein the room setting module tracks discarded items during a surgical procedure and provides an output of the items for replacement. (Item 10) The apparatus further includes a viewing device, the viewing device comprising: A head unit, the head unit comprising: A head mountable frame; a data channel for receiving image data of the image; a light guide, the light guide being a transparent light guide positioned between the user's eye and the external surface of the real object such that light from the external surface of the real object remains transmitted to the retina of the eye, such that the user sees the external surface of the real object augmented in the rendering of the image including at least one object in the desired room layout; A head unit comprising: Item 3. The vision system of item 2, comprising: (Item 11) the guidance module is an anatomical registration module executable by the processor to perform anatomical registration based on the digital representation; storing a location of a body part of a patient, the location of the body part being based on a location of a real object by the real object detection device; Item 1. The vision system of item 1, comprising: (Item 12) Item 12. The vision system of item 11, further comprising a probe, the probe being a guided real object, the probe having a probe tip and a detectable surface, wherein when a user positions the probe tip relative to the body part, the real object detection device detects the detectable surface, and the anatomical structure registration module calculates a location of the probe tip based on a location of the detectable surface. (Item 13) Item 13. The vision system of item 12, wherein the anatomical structure registration module displays target points to the user that are superimposed on the body part and guides the user to specific locations on the body part. (Item 14) Item 14. The vision system of item 13, wherein the anatomical structure registration module displays a plurality of target points to the user and guides the user to a plurality of distinct specific locations on the body part, and the anatomical structure registration module calculates a distinct location of the probe tip based on a distinct location of the detectable surface when the probe tip is at the distinct specific location. (Item 15) Item 13. The vision system of item 12, wherein the anatomical structure registration module calculates an orientation of the detectable surface and uses the orientation to calculate a location of the probe tip. (Item 16) The apparatus further includes a viewing device, the viewing device comprising: A head unit, the head unit comprising: A head mountable frame; a data channel for receiving image data of the image; an optical waveguide, the optical waveguide being a transparent optical waveguide positioned between the user's eye and the external surface of the real object such that light from the external surface of the real object remains transmitted to the retina of the eye so that the user sees the external surface of the real object augmented with the rendering of the image including the rendering of the target point; A head unit comprising: Item 14. The vision system of item 13, comprising: (Item 17) the guidance module is a surgical planning module executable by the processor to plan a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; displaying a digital representation of the patient's body part together with the virtual object to the user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; moving the virtual object relative to the digital representation of the body part within the user's view in response to input from the user; Item 1. The vision system of item 1, comprising: (Item 18) Item 18. The vision system of item 17, wherein the user is simultaneously provided with at least two views of the digital representation of the body part. (Item 19) Item 18. The vision system of item 17, wherein the virtual object is a digital representation of an implant that moves within the digital representation of the body. (Item 20) 20. The visualization system of claim 19, wherein the surgical planning module displays measurements of the digital representation of the implant, the measurements being adjustable by adjusting the digital representation of the implant. (Item 21) The apparatus further includes a viewing device, the viewing device comprising: a data channel for receiving image data of the image; a display connected to the data channel such that the display allows the user to view the image including the digital representation of the patient's body part together with the virtual object; a user input device through which the user provides the input and guides the virtual object; Item 18. The vision system of item 17, comprising: (Item 22) the viewing device is a first viewing device, and the user is a first user; further comprising a second viewing device, said second viewing device comprising: a data channel for receiving image data of the image; a display connected to the data channel such that the image including the digital representation of the patient's body part together with the virtual object can be viewed by a second user; a user input device through which the second user provides the input and guides the virtual object; and Item 22. The vision system of item 21, comprising: (Item 23) 23. The viewing system of claim 22, wherein the surgical planning module displays the second user's rendering within the display of the first viewing device. (Item 24) the guidance module is a surgical procedure execution module executable by the processor to assist in performing a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; In response to an input from the user, moving the virtual object relative to the digital representation of the body part in a view of the user; moving, within the real environment, individual ones of the real objects relative to body parts of the patient; Item 1. The vision system of item 1, comprising: (Item 25) 25. The vision system of claim 24, wherein one of the real objects detected by the real object detection device is a body part of the patient. (Item 26) 25. The vision system of claim 24, wherein one of the real objects detected by the real object detection device is a medical staff member. (Item 27) Item 25. The vision system of item 24, wherein one of the real objects detected by the real object detection device is a robot. (Item 28) 25. The vision system of claim 24, wherein the real object is a cutting tool that is moved into the patient's body part. (Item 29) 25. The vision system of item 24, wherein the real object to be moved is an implant to be inserted into the patient's body part. (Item 30) Item 25. The vision system of item 24, wherein one of the real objects detected by the real object detection device is a disposable item. (Item 31) 25. The vision system of claim 24, wherein the surgical execution module tracks the movement of the individual real objects as the real object detection device detects the individual real objects and the map generator updates the map as the individual real objects move. (Item 32) a head-mountable frame, wherein the optical waveguide is fixed to the head-mountable frame; a raw data receiving unit for receiving raw data; an image generation unit, connected to the data storage device, that processes raw data of the reflected waves, creates image data representing an image, and stores the image data in the data storage device; an image data receiving unit for receiving the image data from the data storage device; at least one projector connected to the image data receiving unit to receive the image data, the projector generating light in a pattern representing the image data; at least one light guide connected to the projector and secured to the head-mountable frame, the at least one light guide directing the light to a retina of the user's eye so that the user sees a rendering of the image data; and Item 1. The vision system of item 1, further comprising: (Item 33) a map storage routine for storing a first map having a plurality of anchors, each anchor of the first map having a set of coordinates; an anchor identification system connected to the real object detection device, the anchor identification system detecting anchors in a second map based on the locations of the real objects, each anchor in the second map having a set of coordinates; a location determination module coupled to the first map and the second map and operable to locate the second map relative to the first map by matching a first anchor of the second map with a first anchor of the first map and matching a second anchor of the second map with a second anchor of the first map; Item 1. The vision system of item 1, further comprising: (Item 34) A visual recognition method comprising: Detecting a location of a real object in a real environment using a real object detection device; running, with a processor, a map generator connected to the real object detection device, to receive data of the real environment including the real object and to create a map forming a digital representation of the real environment including the real object; using said processor to execute a map storage routine to store said map on said data storage device; using the processor to execute a navigation module coupled to the data storage device to retrieve the map and navigate at least one of a virtual object and a real object based on the digital representation; A method comprising: (Item 35) the guidance module is a room setup module executable by the processor to setup a room based on the digital representation; storing a desired room layout; providing an output digitally superimposing the desired room layout onto the real environment; Item 35. The method according to Item 34, comprising: (Item 36) Item 36. The method of item 35, wherein the room setup module generates an image based on the desired room layout and superimposes the image over the real environment. (Item 37) Item 37. The method of item 36, wherein the room setup module generates an image of a desired placement of one of the real objects in the desired room layout and superimposes the image over the real environment. (Item 38) Item 38. The method of claim 37, wherein the room setup module provides an output indicating that individual real objects have been moved to positions consistent with the desired installation. (Item 39) Item 39. The method of item 38, wherein the room setting module tracks the movement of the individual real objects as the real object detection device detects the individual real objects and the map generator updates the map as the individual real objects move. (Item 40) 40. The method of claim 39, wherein the room setup module provides a visual output indicating that the individual real objects have been moved to positions that match the desired setup by changing a color of an image of the desired setup. (Item 41) Item 36. The method of item 35, wherein the room setup module provides an output that digitally overlays the exploded layout of the desired room onto the real environment. (Item 42) Item 36. The method of item 35, wherein the room setup module tracks discarded items during a surgical procedure and provides an output of the items for replacement. (Item 43) Mounting a head-mountable frame on a viewer's head; generating light in a pattern representative of the image data; using optical waveguides affixed to the head-mountable frame to direct the light to the retina of the viewer's eye such that the rendering of the image data appears to the viewer as a digital representation of the exterior surfaces of the real objects augmented with the rendering of the image including at least one object in the desired room layout; Item 36. The method of item 35, further comprising: (Item 44) the guidance module is an anatomical registration module executable by the processor to perform anatomical registration based on the digital representation; storing a location of a body part of a patient, the location of the body part being based on a location of a real object by the real object detection device; Item 35. The method according to Item 34, comprising: (Item 45) Item 45. The method of item 44, further comprising a probe, the probe being a guided real object, the probe having a probe tip and a detectable surface, wherein when a user positions the probe tip relative to the body part, the real object detection device detects the detectable surface, and the anatomical structure registration module calculates a location of the probe tip based on a location of the detectable surface. (Item 46) 46. ​​The method of claim 45, wherein the anatomical structure registration module displays target points to the user that are superimposed on the body part and guides the user to specific locations on the body part. (Item 47) Item 47. The method of item 46, wherein the anatomical structure registration module displays a plurality of target points to the user and guides the user to a plurality of distinct specific locations on the body part, and the anatomical structure registration module calculates a distinct location of the probe tip based on a distinct location of the detectable surface when the probe tip is at the distinct specific location. (Item 48) 46. ​​The method of claim 45, wherein the anatomical structure registration module calculates an orientation of the detectable surface and uses the orientation to calculate a location of the probe tip. (Item 49) Mounting a head-mountable frame on a viewer's head; generating light in a pattern representative of the image data; using optical waveguides affixed to the head-mountable frame to direct the light to the retina of the viewer's eye such that the rendering of the image data appears to the viewer as a digital representation of the exterior surface of the real object augmented with a rendering of the image including a rendering of the target point; Item 47. The method of item 46, further comprising: (Item 50) the guidance module is a surgical planning module executable by the processor to plan a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; displaying a digital representation of the patient's body part together with the virtual object to the user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; moving the virtual object relative to the digital representation of the body part within the user's view in response to input from the user; Item 35. The method according to Item 34, comprising: (Item 51) Item 51. The method of item 50, wherein the user is simultaneously provided with at least two views of the digital representation of the body part. (Item 52) Item 51. The method of item 50, wherein the virtual object is a digital representation of an implant that moves within the digital representation of the body. (Item 53) 53. The method of claim 52, wherein the surgical planning module displays measurements of the digital representation of the implant, the measurements being adjustable by adjusting the digital representation of the implant. (Item 54) receiving image data for the image on a data channel of a viewing device; displaying the image data such that the image including the digital representation of the patient's body part together with the virtual object is visible to the user through a display connected to the data channel; using a user input device to transmit the input provided by the user through the user input device to guide the virtual object; 51. The method of claim 50, further comprising: (Item 55) the viewing device is a first viewing device, and the user is a first user; receiving image data for the image on a data channel of a second viewing device; displaying the image data to the second user through a display connected to the data channel of the second viewing device such that the image including the digital representation of the patient's body part together with the virtual object is visible to the second user; using a user input device of the second viewing device to transmit the input provided by the user through the user input device to guide the virtual object; Item 55. The method of item 54, further comprising: (Item 56) Item 56. The method of item 55, wherein the surgical planning module displays the second user's rendering within the display of the first viewing device. (Item 57) the guidance module is a surgical procedure execution module executable by the processor to assist in performing a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; In response to an input from the user, moving the virtual object relative to the digital representation of the body part in a view of the user; moving, within the real environment, individual ones of the real objects relative to body parts of the patient; Item 35. The method according to Item 34, comprising: (Item 58) Item 58. The method of item 57, wherein one of the real objects detected by the real object detection device is a body part of the patient. (Item 59) Item 58. The method of item 57, wherein one of the real objects detected by the real object detection device is a medical staff member. (Item 60) Item 58. The method according to Item 57, wherein one of the real objects detected by the real object detection device is a robot. (Item 61) Item 58. The method of item 57, wherein the real object is a cutting tool that is moved into the patient's body part. (Item 62) 58. The method of claim 57, wherein the real object to be moved is an implant to be inserted into the patient's body part. (Item 63) Item 58. The method of item 57, wherein one of the real objects detected by the real object detection device is a disposable item. (Item 64) Item 58. The method of item 57, wherein the surgical operation execution module tracks the movement of the individual real objects as the real object detection device detects the individual real objects and the map generator updates the map as the individual real objects move. (Item 65) Mounting a head-mountable frame on a viewer's head; storing the raw data in a data storage device; processing the raw reflected wave data to generate image data; storing the image data in the data storage device; receiving the image data from the data storage device; generating light in a pattern representative of the image data; using an optical waveguide affixed to the head-mountable frame to direct the light to the retina of the viewer's eye so that the viewer sees a rendering of the image data; Item 35. The method of item 34, further comprising: (Item 66) storing a first map having a plurality of anchors, each anchor of the first map having a set of coordinates; detecting anchors in a second map based on the locations of the real objects, each anchor in the second map having a set of coordinates; locating the second map relative to the first map by matching a first anchor of the second map with a first anchor of the first map and matching a second anchor of the second map with a second anchor of the first map; 33. The method of claim 32, further comprising: (Item 67) 1. A vision system comprising: a real object detection device positioned to detect the location of a real object in a real environment; at least one processor; a computer-readable medium coupled to the processor; a data storage device on said computer readable medium; a set of instructions stored on the computer-readable medium and executable by the processor; wherein the set of instructions comprises: a map generator connected to the real object detection device, the map generator being executable to receive data of the real environment including the real objects and to create a map forming a digital representation of the real environment including the real objects; a map storage routine executable to store the map on the data storage device; a plurality of navigation modules coupled to the data storage device and executable to read the map and navigate at least one of a virtual object and a real object based on the digital representation, the navigation modules comprising: a. a room setup module, the room setup module executable by the processor to setup a room based on the digital representation; storing a desired room layout; providing an output digitally superimposing the desired room layout onto the real environment; a room setting module including: b. an anatomical registration module, the anatomical registration module executable by the processor to perform anatomical registration based on the digital representation; storing a location of a body part of a patient, the location of the body part being based on a location of a real object by the real object detection device; an anatomical structure registration module, c. a surgical planning module, said surgical planning module executable by said processor to plan a surgical procedure based on said digital representation; storing a digital representation of the patient's body part; displaying a digital representation of the patient's body part together with the virtual object to the user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; moving the virtual object relative to the digital representation of the body part within the user's view in response to input from the user; a surgical planning module including: d. a surgical procedure execution module, said surgical procedure execution module executable by said processor to assist in performing a surgical procedure based on said digital representation; storing a digital representation of the patient's body part; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; In response to an input from the user, moving the virtual object relative to the digital representation of the body part in a view of the user; moving, within the real environment, individual ones of the real objects relative to body parts of the patient; a surgical execution module, a plurality of guidance modules, including at least two of:

[0023] A vision system, including: (Item 68) Item 68. The vision system of item 67, wherein the plurality of guidance modules includes at least three of the modules. (Item 69) Item 69. The vision system of item 68, wherein the plurality of guidance modules includes at least four of the modules. (Item 70) Item 68. The vision system of item 67, further comprising an interface for receiving a selection from a user to execute each of the modules. (Item 71) A visual recognition method comprising: Detecting a location of a real object in a real environment using a real object detection device; running, with a processor, a map generator connected to the real object detection device, to receive data of the real environment including the real object and to create a map forming a digital representation of the real environment including the real object; using said processor to execute a map storage routine to store said map on said data storage device; and executing, with the processor, a plurality of navigation modules coupled to the data storage device, to retrieve the map and navigate at least one of a virtual object and a real object based on the digital representation, the navigation modules comprising: a. a room setup module, the room setup module executable by the processor to setup a room based on the digital representation; storing a desired room layout; providing an output digitally superimposing the desired room layout onto the real environment; a room setting module including: b. the guidance module is an anatomical registration module, the anatomical registration module being executable by the processor to perform anatomical registration based on the digital representation; storing a location of a body part of a patient, the location of the body part being based on a location of a real object by the real object detection device; an anatomical structure registration module, c. a surgical planning module, said surgical planning module executable by said processor to plan a surgical procedure based on said digital representation; storing a digital representation of the patient's body part; displaying a digital representation of the patient's body part together with the virtual object to the user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; moving the virtual object relative to the digital representation of the body part within the user's view in response to input from the user; a surgical planning module including: d. a surgical procedure execution module, said surgical procedure execution module executable by said processor to assist in performing a surgical procedure based on said digital representation; storing a digital representation of the patient's body part; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; In response to an input from the user, moving the virtual object relative to the digital representation of the body part in a view of the user; moving, within the real environment, individual ones of the real objects relative to body parts of the patient; a surgical execution module, and A method comprising: (Item 72) Item 72. The method of item 71, wherein the plurality of induction modules includes at least three of the modules. (Item 73) the plurality of induction modules includes at least four of the modules; receiving, through an interface, a selection from a user to execute each of said modules; Item 73. The method of item 72, further comprising: (Item 74) 1. A vision system comprising: a real object detection device positioned to detect the location of a real object in a real environment; at least one processor; a computer-readable medium coupled to the processor; a data storage device on said computer readable medium; a set of instructions stored on the computer-readable medium and executable by the processor; a map generator connected to the real object detection device, the map generator being executable to receive data of the real environment including the real objects and to create a map forming a digital representation of the real environment including the real objects; a map storage routine executable to store the map on the data storage device; a head-mountable frame, wherein the optical waveguide is fixed to the head-mountable frame; a raw data receiving unit for receiving raw data of the reflected wave; an image generation unit, connected to the data storage device, that processes raw data of the reflected waves, creates image data representing an image, and stores the image data in the data storage device; an image data receiving unit for receiving the image data from the data storage device; at least one projector connected to the image data receiving unit to receive the image data, the projector generating light in a pattern representative of the image data and based on the map; at least one light guide connected to the projector and secured to the head-mountable frame, the at least one light guide directing the light to a retina of the user's eye so that the user sees a rendering of the image data; and A set of instructions, including A vision system comprising: (Item 75) a navigation module coupled to the data storage device and executable to read the map and navigate at least one of a virtual object and a real object based on the digital representation; Item 75. The vision system of item 74, further comprising: (Item 76) the guidance module is a room setup module executable by the processor to setup a room based on the digital representation; storing a desired room layout; providing an output digitally superimposing the desired room layout onto the real environment; Item 76. The vision system of item 75, comprising: (Item 77) the guidance module is an anatomical registration module executable by the processor to perform anatomical registration based on the digital representation; storing a location of a body part of a patient, the location of the body part being based on a location of a real object by the real object detection device; Item 76. The vision system of item 75, comprising: (Item 78) the guidance module is a surgical planning module executable by the processor to plan a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; displaying a digital representation of the patient's body part together with the virtual object to the user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; moving the virtual object relative to the digital representation of the body part within the user's view in response to input from the user; Item 76. The vision system of item 75, comprising: (Item 79) the guidance module is a surgical procedure execution module executable by the processor to assist in performing a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; In response to an input from the user, moving the virtual object relative to the digital representation of the body part in a view of the user; moving, within the real environment, individual ones of the real objects relative to body parts of the patient; Item 76. The vision system of item 75, comprising: (Item 80) 1. A computed tomography (CT) scanner, comprising: A base and a platform for the patient; and a rotator mounted to the base for rotation about the patient, the transmitter being an X-ray transmitter affixed to the rotator and transmitting X-ray waves, the receiver being an X-ray detector affixed to the rotator and detecting the X-ray waves, and movement of the platform relative to the base permits movement of the patient relative to a plane extending from the X-ray transmitter to the X-ray detector to acquire the raw data; Computed tomography (CT) scanners, including Item 75. The vision system of item 74, further comprising: (Item 81) Item 75. The vision system of item 74, wherein the light guide is a transparent light guide positioned between the eye and the external surface of the real object such that light from the external surface of the real object remains transmitted to the retina of the eye so that the user can see the external surface of the real object augmented by the rendering of the digital representation. (Item 82) the projector is a left projector, the light guide is a left light guide, and the eye is the left eye of the user; a right projector connected to the image data receiving unit to receive the image data, the right projector generating light in a pattern representing the image data; a right optical waveguide connected to the right projector, the right optical waveguide directing light from the right projector to the retina of the user's right eye while transmitting light from the external surface of the body to the retina of the right eye so that the user sees the external surface of the object augmented in the rendering of the image with the right eye; and Item 82. The vision system of item 81, further comprising: (Item 83) The set of instructions a stereoscopic analyzer connected to the data receiving unit and configured to receive the image data, the stereoscopic analyzer determining left and right image datasets, the left and right projectors projecting the left and right image datasets, respectively, the left and right image datasets being different from each other and providing the user with a perception of a three-dimensional rendering; Item 83. The vision system of item 82, comprising: (Item 84) a head unit detection device for detecting movement of the head-mountable frame; and wherein the set of instructions further comprises: a display adjustment algorithm connected to the head unit detection device, the display adjustment algorithm receiving measurements based on movements detected by the head unit detection device and calculating a setpoint value; a display positioning algorithm that modifies the position of the body part within the view of the eye based on the setting value; Item 75. The vision system of item 74, comprising: (Item 85) The head unit detection device a head unit inertial measurement unit (IMU) mounted on the head-mountable frame, the head unit IMU including a motion sensor that detects movement of the head-mountable frame; Item 85. The vision system of item 84, comprising: (Item 86) The head unit detection device a head unit camera mounted on the head-mountable frame, the head unit camera detecting movement of the head-mountable frame by capturing images of objects within the view of the head unit camera; the set of instructions comprising: an image processing system for analyzing the image and detecting the pose position of the head-mountable frame; Item 85. The vision system of item 84, comprising: (Item 87) a map storage routine for storing a first map having a plurality of anchors, each anchor of the first map having a set of coordinates; an anchor identification system connected to the real object detection device, the anchor identification system detecting anchors in a second map based on the locations of the real objects, each anchor in the second map having a set of coordinates; a location determination module coupled to the first map and the second map and operable to locate the second map relative to the first map by matching a first anchor of the second map with a first anchor of the first map and matching a second anchor of the second map with a second anchor of the first map; Item 75. The vision system of item 74, further comprising: (Item 88) A visual recognition method comprising: Detecting a location of a real object in a real environment using a real object detection device; running, with a processor, a map generator connected to the real object detection device, to receive data of the real environment including the real object and to create a map forming a digital representation of the real environment including the real object; using said processor to execute a map storage routine to store said map on said data storage device; Mounting a head-mountable frame on a viewer's head; storing raw data of the reflected waves in a data storage device; processing the raw reflected wave data to generate image data; storing the image data in the data storage device; receiving the image data from the data storage device; generating light in a pattern representative of the image data and based on the map; using an optical waveguide affixed to the head-mountable frame to direct the light to the retina of the viewer's eye so that the viewer sees a rendering of the image data; A method comprising: (Item 89) Executing with the processor a navigation module coupled to the data storage device to retrieve the map and navigate at least one of a virtual object and a real object based on the digital representation. Item 89. The method of item 88, further comprising: (Item 90) the guidance module is a room setup module executable by the processor to setup a room based on the digital representation; storing a desired room layout; providing an output digitally superimposing the desired room layout onto the real environment; Item 89. The method according to item 88, comprising: (Item 91) the guidance module is an anatomical registration module executable by the processor to perform anatomical registration based on the digital representation; storing a location of a body part of a patient, the location of the body part being based on a location of a real object by the real object detection device; Item 89. The method according to item 88, comprising: (Item 92) the guidance module is a surgical planning module executable by the processor to plan a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; displaying a digital representation of the patient's body part together with the virtual object to the user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; moving the virtual object relative to the digital representation of the body part within the user's view in response to input from the user; Item 89. The method according to item 88, comprising: (Item 93) the guidance module is a surgical procedure execution module executable by the processor to assist in performing a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; In response to an input from the user, moving the virtual object relative to the digital representation of the body part in a view of the user; moving, within the real environment, individual ones of the real objects relative to body parts of the patient; Item 89. The method according to item 88, comprising: (Item 94) 1. Operating a computed tomography (CT) scanner, the computed tomography (CT) scanner comprising: A base and a platform for the patient; and a rotator mounted to the base for rotation about the patient, the transmitter being an X-ray transmitter affixed to the rotator and transmitting X-ray waves, the receiver being an X-ray detector affixed to the rotator and detecting the X-ray waves, and movement of the platform relative to the base permits movement of the patient relative to a plane extending from the X-ray transmitter to the X-ray detector to acquire the raw data; Including Item 89. The method of item 88, further comprising: (Item 95) Item 89. The method of item 88, wherein the light guide is a transparent light guide positioned between the eye and the external surface of the real object such that light from the external surface of the real object remains transmitted to the retina of the eye so that the viewer sees the external surface of the real object augmented with the rendering of the digital representation. (Item 96) the eye is the viewer's left eye, directing the light to the retina of the viewer's right eye while transmitting light from the external surface of the body to the retina of the right eye so that the viewer sees, with the right eye, the external surface of the body augmented with the rendering of the body part. Item 96. The method of item 95, further comprising: (Item 97) determining left and right image data sets, the left and right image data sets being different from one another to give the viewer a perception of a three-dimensional rendering; Item 97. The method of item 96, further comprising: (Item 98) detecting movement of the head-mountable frame; calculating a setpoint value based on the detected movement; and modifying the position of the body part within the view of the eye based on the setting value; Item 89. The method of item 88, further comprising: (Item 99) Item 99. The method of item 98, wherein the movement is detected using a motion sensor in a head unit inertial measurement unit (IMU) mounted on the head-mountable frame. (Item 100) the movement is detected using a head unit camera mounted on the head-mountable frame, the head unit camera detecting movement of the head-mountable frame by capturing images of objects within a view of the head unit camera; analyzing the image and detecting a posture position of the head-mountable frame; Item 99. The method of item 98, further comprising: (Item 101) storing a first map having a plurality of anchors, each anchor of the first map having a set of coordinates; detecting anchors in a second map based on the locations of the real objects, each anchor in the second map having a set of coordinates; locating the second map relative to the first map by matching a first anchor of the second map with a first anchor of the first map and matching a second anchor of the second map with a second anchor of the first map; Item 89. The method of item 88, further comprising: (Item 102) 1. A vision system comprising: a real object detection device positioned to detect the location of a real object in a real environment; at least one processor; a computer-readable medium coupled to the processor; a data storage device on said computer readable medium; a set of instructions stored on the computer-readable medium and executable by the processor; a map generator connected to the real object detection device, the map generator being executable to receive data of the real environment including the real objects and to create a map forming a digital representation of the real environment including the real objects; a map storage routine for storing a first map having a plurality of anchors, each anchor of the first map having a set of coordinates; an anchor identification system connected to the real object detection device, the anchor identification system detecting anchors in a second map based on the locations of the real objects, each anchor in the second map having a set of coordinates; a location determination module coupled to the first map and the second map and operable to locate the second map relative to the first map by matching a first anchor of the second map with a first anchor of the first map and matching a second anchor of the second map with a second anchor of the first map; A set of instructions, including A vision system comprising: (Item 103) a navigation module coupled to the data storage device and executable to read the map and navigate at least one of a virtual object and a real object based on the digital representation; Item 103. The vision system of item 102, further comprising: (Item 104) the guidance module is a room setup module executable by the processor to setup a room based on the digital representation; storing a desired room layout; providing an output digitally superimposing the desired room layout onto the real environment; Item 104. The vision system of item 103, comprising: (Item 105) the guidance module is an anatomical registration module executable by the processor to perform anatomical registration based on the digital representation; storing a location of a body part of a patient, the location of the body part being based on a location of a real object by the real object detection device; Item 104. The vision system of item 103, comprising: (Item 106) the guidance module is a surgical planning module executable by the processor to plan a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; displaying a digital representation of the patient's body part together with the virtual object to the user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; moving the virtual object relative to the digital representation of the body part within the user's view in response to input from the user; Item 104. The vision system of item 103, comprising: (Item 107) the guidance module is a surgical procedure execution module executable by the processor to assist in performing a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; In response to an input from the user, moving the virtual object relative to the digital representation of the body part in a view of the user; moving, within the real environment, individual ones of the real objects relative to body parts of the patient; Item 104. The vision system of item 103, comprising: (Item 108) Item 103. The vision system of item 102, wherein the real object detection device is a real object detection camera. (Item 109) a reference map generator coupled to the first map and the second map, the reference map generator being operable to create a reference map by incorporating a third anchor of the first map into the second map. Item 103. The vision system of item 102, further comprising: (Item 110) The apparatus further includes a viewing device, the viewing device comprising: A head unit, the head unit comprising: a head-mountable frame, wherein the real object detection device is mounted on the head-mountable frame; a data channel for receiving image data of local content; a local content location system coupled to said data channel and operable to associate said local content with an anchor of said reference map; a display system connected to the local content location system and configured to display the local content; A head unit comprising: Item 103. The vision system of item 102, comprising: (Item 111) a local-to-world coordinate transformer that transforms the local coordinate frame of the local content into the world coordinate frame of the second map; Item 111. The vision system of item 110, further comprising: (Item 112) a first world frame determination routine that calculates a first world coordinate frame based on anchors of the second map; a first world frame storage instruction for storing the world coordinate frame; a head frame determination routine for calculating a head coordinate frame that changes in response to movement of the head-mountable frame; a head frame storage instruction for storing the first head coordinate frame; a world / head coordinate converter that converts the world coordinate frame into the head coordinate frame; Item 111. The vision system of item 110, further comprising: (Item 113) Item 113. The vision system of item 112, wherein the head coordinate frame changes relative to the world coordinate frame as the head-mountable frame moves. (Item 114) Item 111. The vision system of item 110, further comprising at least one sound element associated with at least one anchor of the second map. (Item 115) Item 111. The vision system of item 110, wherein the first and second maps are created by the vision device. (Item 116) Further comprising first and second viewing devices, each viewing device comprising: A head unit, the head unit comprising: a head-mountable frame, wherein the real object detection device is mounted on the head-mountable frame; a data channel for receiving image data of local content; a local content location system coupled to said data channel and operable to associate said local content with an anchor of said reference map; a display system connected to the local content location system for displaying the local content; A head unit comprising: Item 103. The vision system of item 102, comprising: (Item 117) Item 117. The vision system of item 116, wherein the first vision device creates an anchor for the first map, the second vision device creates an anchor for the second map, and the location identification module forms part of the second vision device. (Item 118) Item 118. The vision system of item 117, wherein the first and second maps are created in a first and second session, respectively. (Item 119) Item 118. The vision system of item 117, wherein the first and second maps are created in the same session. (Item 120) A server; a map download system forming part of the viewing device and configured to download the first map from a server via a network; Item 111. The vision system of item 110, further comprising: (Item 121) Item 103. The vision system of item 102, wherein the localization module repeatedly attempts to localize the second map relative to the first map. (Item 122) an anchor projector on the server that projects anchors of the second map onto the first map; a map quality analyzer on the server that determines a metric score representative of the quality of the first map and a metric representative of the quality of the second map; a map enhancer that assigns a higher one of the first map and the second map for transmission to a viewing device based on the metric score; Item 103. The vision system of item 102, further comprising: (Item 123) a data storage device containing digital representations of a plurality of completed surgical procedures; Item 103. The vision system of item 102, further comprising: (Item 124) A visual recognition method comprising: Detecting a location of a real object in a real environment using a real object detection device; running, with a processor, a map generator connected to the real object detection device, to receive data of the real environment including the real object and to create a map forming a digital representation of the real environment including the real object; storing a first map having a plurality of anchors, each anchor of the first map having a set of coordinates; detecting anchors in a second map based on the locations of the real objects, each anchor in the second map having a set of coordinates; locating the second map relative to the first map by matching a first anchor of the second map with a first anchor of the first map and matching a second anchor of the second map with a second anchor of the first map; A method comprising: (Item 125) Executing with the processor a navigation module coupled to the data storage device to retrieve the map and navigate at least one of a virtual object and a real object based on the digital representation. Item 125. The method of item 124, further comprising: (Item 126) the guidance module is a room setup module executable by the processor to setup a room based on the digital representation; storing a desired room layout; providing an output digitally superimposing the desired room layout onto the real environment; Item 126. The method according to Item 125, comprising: (Item 127) the guidance module is an anatomical registration module executable by the processor to perform anatomical registration based on the digital representation; storing a location of a body part of a patient, the location of the body part being based on a location of a real object by the real object detection device; Item 126. The method according to Item 125, comprising: (Item 128) the guidance module is a surgical planning module executable by the processor to plan a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; displaying a digital representation of the patient's body part together with the virtual object to the user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; moving the virtual object relative to the digital representation of the body part within the user's view in response to input from the user; Item 126. The method according to Item 125, comprising: (Item 129) the guidance module is a surgical procedure execution module executable by the processor to assist in performing a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; In response to an input from the user, moving the virtual object relative to the digital representation of the body part in a view of the user; moving, within the real environment, individual ones of the real objects relative to body parts of the patient; Item 126. The method according to Item 125, comprising: (Item 130) Item 125. The method of item 124, wherein the real object detection device is a real object detection camera. (Item 131) executing a reference map generator connected to the first map and the second map and executable to create a reference map by incorporating a third anchor of the first map into the second map; Item 125. The method of item 124, further comprising: (Item 132) The method is performed, at least in part, on a viewing device, the viewing device comprising: A head unit, the head unit comprising: a head-mountable frame, wherein the real object detection device is mounted on the head-mountable frame; a data channel for receiving image data of local content; a local content location system coupled to said data channel and operable to associate said local content with an anchor of said reference map; a display system connected to the local content location system for displaying the local content; A head unit comprising: Item 125. The method according to Item 124, comprising: (Item 133) performing a local-to-world coordinate converter that converts the local coordinate frame of the local content into the world coordinate frame of the second map; Item 133. The method of item 132, further comprising: (Item 134) executing a first world frame determination routine to calculate a first world coordinate frame based on anchors of the second map; executing a first world frame store instruction to store the world coordinate frame; executing a head frame determination routine to calculate a head coordinate frame that changes in response to movement of the head-mountable frame; executing a head frame storage instruction to store the first head coordinate frame; performing a world-to-head coordinate converter to transform the world coordinate frame into the head coordinate frame; Item 133. The method of item 132, further comprising: (Item 135) Item 135. The method of item 134, wherein the head coordinate frame changes relative to the world coordinate frame as the head-mountable frame moves. (Item 136) Associating at least one sound element with at least one anchor in said second map. Item 133. The method of item 132, further comprising: (Item 137) Item 133. The method of item 132, wherein the first and second maps are created by the viewing device. (Item 138) The method is performed, at least in part, on first and second viewing devices, each viewing device comprising: A head unit, the head unit comprising: a head-mountable frame, wherein the real object detection device is mounted on the head-mountable frame; a data channel for receiving image data of local content; a local content location system coupled to said data channel and operable to associate said local content with an anchor of said reference map; a display system connected to the local content location system for displaying the local content; A head unit comprising: Item 125. The method according to Item 124, comprising: (Item 139) Item 139. The method of item 138, wherein the first viewing device creates an anchor for the first map, the second viewing device creates an anchor for the second map, and the location identification module forms part of the second viewing device. (Item 140) Item 139. The method of item 139, wherein the first and second maps are created in a first and second session, respectively. (Item 141) Item 139. The method of item 139, wherein the first and second maps are created in the same session. (Item 142) executing a map download system forming part of said viewing device for downloading said first map from a server via a network; Item 133. The method of item 132, further comprising: (Item 143) Item 125. The method of item 124, wherein the location module repeatedly attempts to locate the second map relative to the first map. (Item 144) executing an anchor projector on the server that projects anchors of the second map onto the first map; executing a map quality analyzer on the server to determine a metric score representative of a quality of the first map and a metric representative of a quality of the second map; executing a map enhancer that assigns a higher one of the first map and the second map for transmission to a viewing device based on the metric score; Item 125. The method of item 124, further comprising: (Item 145) Item 125. The method of item 124, further comprising storing digital representations of the plurality of completed surgical procedures in a data storage device. [Brief explanation of the drawings]

[0015] The invention will now be further described, by way of example only, with reference to the accompanying drawings in which:

[0016] [Figure 1] FIG. 1 is a block diagram of a patient viewing system according to one embodiment of the present invention.

[0017] [Figure 2] FIG. 2 is a partial perspective view and partial block diagram of a CT scanner, a data receiving unit, an image generating unit, and a data storage device that form part of a patient viewing system.

[0018] [Figure 3] FIG. 3 is a partial perspective and partial block diagram of a display system, catheter, and data storage device that form part of a patient viewing system.

[0019] [Figure 4] FIG. 4 is a block diagram illustrating a catheter integrated system forming part of the display system of FIG. 3, further illustrating a catheter.

[0020] [Figure 5] FIG. 5 is a perspective view illustrating a viewer in the form of a surgeon, who sees a rendering of the patient's body and internal body parts of the patient, and also sees a rendering of the tip of the catheter and the past path of the tip.

[0021] [Figure 6] FIG. 6 is a top plan view of FIG.

[0022] [Figure 7] FIG. 7 is the view as seen by a viewer.

[0023] [Figure 8] FIG. 8 is a view similar to FIG. 6 after the viewer has moved counterclockwise around the patient's body and moved their head counterclockwise to maintain a view of the patient's body.

[0024] [Figure 9] FIG. 9 is a view similar to FIG. 7 showing the patient's body and how the rendering is modified within the view.

[0025] [Figure 10] FIG. 10 illustrates in enlarged detail the renderings shown to the viewer in FIGS.

[0026] [Figure 11] FIG. 11 is a partial perspective and partial block diagram illustrating a vision system, a real-world object in the form of a table, and a first user interacting with the vision system, according to an embodiment of the present invention.

[0027] [Figure 12] FIG. 12 is a block diagram of a first vision device forming part of the vision system.

[0028] [Figure 13]FIG. 13 is a schematic diagram illustrating how an origin coordinate frame is transformed into a destination coordinate frame for the purposes of correct rendering of local content.

[0029] [Figure 14] FIG. 14 is a top plan view illustrating a pupil-based coordinate frame.

[0030] [Figure 15] FIG. 15 is a top plan view illustrating the camera coordinate frame, including all pupil positions.

[0031] [Figure 16] FIG. 16 is a block diagram of the server's vision data and algorithms and the first and second vision devices of the vision system.

[0032] [Figure 17] FIG. 17 is a two-dimensional representation of a three-dimensional first local tracking map (Map 1) generated by a first viewing device.

[0033] [Figure 18] FIG. 18 is a block diagram illustrating the upload of Map 1 from a first viewing device to a server.

[0034] [Figure 19] FIG. 19 is a view similar to FIG. 11 after a first user has finished a first session and a second user has started a second session using a second viewing device that forms part of the viewing system.

[0035] [Figure 20] FIG. 20 is a block diagram illustrating the downloading of a reference map from a server to a second viewing device.

[0036] [Figure 21]FIG. 21 illustrates a two-dimensional representation of a second tracking map (Map 2) generated by a second viewing device, and further illustrates a localization attempt made to locate Map 2 relative to a reference map.

[0037] [Figure 22] FIG. 22 is a view similar to FIG. 21 after Map 2 has been further expanded and local content has been associated with anchors in Map 2.

[0038] [Figure 23] FIG. 23 is a view similar to FIG. 22 after successful location of map 2 relative to the reference map.

[0039] [Figure 24] FIG. 24 is a view similar to FIG. 23 after an anchor or anchors from the reference map have been included in Map 2 to generate the reference map.

[0040] [Figure 25] FIG. 25 is a view similar to FIG. 24 illustrating a further expansion of Map 2 on a second viewing device.

[0041] [Figure 26] FIG. 26 is a diagram similar to FIG. 25 illustrating the uploading of map 2 from a second viewing device to a server.

[0042] [Figure 27] FIG. 27 is a diagram similar to FIG. 26, merging Map 2 and the reference map.

[0043] [Figure 28] FIG. 28 is a diagram similar to FIG. 27 illustrating the transmission of a new reference map from a server to a first and second viewing device.

[0044] [Figure 29]FIG. 29 is a two-dimensional representation of Map 2 and the head coordinate frame of a second viewing device referenced to Map 2.

[0045] [Figure 30] FIG. 30 is a diagram similar to FIG. 29 illustrating the adjustments of the head coordinate frame that can occur in two dimensions and six degrees of freedom.

[0046] [Figure 31] FIG. 31 illustrates a reference map on a second viewing device, with sounds localized relative to anchors on map 2.

[0047] [Figure 32] 32 and 33 are perspective and block diagrams illustrating the use of a vision system according to another embodiment, where a first user has finished a first session and the first user has started a second session using the vision system. [Figure 33] 32 and 33 are perspective and block diagrams illustrating the use of a vision system according to another embodiment, where a first user has finished a first session and the first user has started a second session using the vision system.

[0048] [Figure 34] 34 and 35 are perspective and block diagrams illustrating the use of a vision system according to a further embodiment of the present invention, with three users using the vision system simultaneously within the same session. [Figure 35] 34 and 35 are perspective and block diagrams illustrating the use of a vision system according to a further embodiment of the present invention, with three users using the vision system simultaneously within the same session.

[0049] [Figure 36] FIG. 36 is a perspective view of a vision system that may be used by various users within a surgical environment.

[0050] [Figure 37] FIG. 37 is a block diagram of a more comprehensive vision system including multiple head units, sensors, a guidance module, and a computer.

[0051] [Figure 38] FIG. 38 is a flow chart illustrating the functionality of the room settings module.

[0052] [Figure 39] 39 and 40 are perspective views of a room and a user tasked with setting up the room for surgery using a vision system. [Figure 40] 39 and 40 are perspective views of a room and a user tasked with setting up the room for surgery using a vision system.

[0053] [Figure 41] FIG. 41 is a perspective view of a surgical robotic system.

[0054] [Figure 42] FIG. 42 is a block diagram illustrating various aspects of the room settings module.

[0055] [Figure 43] 43 and 44 show the interface presented to the user for selecting the execution of various guidance modules. [Figure 44] 43 and 44 show the interface presented to the user for selecting the execution of various guidance modules.

[0056] [Figure 45] FIG. 45 is a flow chart illustrating the functionality of the anatomical registration module.

[0057] [Figure 46]FIG. 46 is a perspective view showing a user using a probe to align points on the anatomy of a body part.

[0058] [Figure 47] FIG. 47 is a block diagram illustrating various aspects of the anatomical registration module.

[0059] [Figure 48] FIG. 48 is a flow chart illustrating the functionality of the surgical planning module.

[0060] [Figure 49] FIG. 49 is a front view of a user with a representation of the view seen by the user.

[0061] [Figure 50] Figures 50a and 50b illustrate a portion of the view seen through the user's head unit as the user makes adjustments to the image of the digital representation of the implant.

[0062] [Figure 51] FIG. 51 is a view of a user interacting with a digital representation of a remote user.

[0063] [Figure 52] FIG. 52 is a block diagram illustrating various aspects of the surgical planning module.

[0064] [Figure 53] FIG. 53 is a flow chart illustrating the functionality of the perform surgery module.

[0065] [Figure 54] FIG. 54 is a perspective view of objects detected by a real object detection device, including various personnel.

[0066] [Figure 55]FIG. 55 is a perspective view showing the plane presented to the user on which the user sets the cutting plane for the robot.

[0067] [Figure 56] FIG. 56 shows the menu items that are displayed to the user when the robot approaches a location set by the user.

[0068] [Figure 57] FIG. 57 illustrates the view that is presented to the user as they set the location of the implant.

[0069] [Figure 58] 58a, 58b, and 58c illustrate how the finger input surface of the handheld controller component is used by the user to operate the robot.

[0070] [Figure 59] FIG. 59 is a block diagram illustrating various aspects of the surgical execution module.

[0071] [Figure 60] FIG. 60 is a block diagram of a machine in the form of a computer that may find use in the system of the present invention, in accordance with one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0072] Detailed Description FIG. 1 of the accompanying drawings illustrates a viewing system 20 according to an embodiment of the present invention, including a CT scanner 22, a data storage device 24, a catheter 26, and a display system 28.

[0073] The data storage device 24 is connected to the CT scanner 22. Raw data from the CT scanner 22 may be stored in the data storage device 24. The data storage device 24 also stores image data based on the raw data.

[0074] Display system 28 is connected to data storage device 24 so as to be able to read image data from data storage device 24. Catheter 26 is connected to display system 28 so that display system 28 can read measurements and video data from catheter 26 for further processing or for display to a viewer.

[0075] In use, a patient is positioned at station 32 in CT scanner 22. The patient's body 30 is scanned using CT scanner 22 to obtain raw data that CT scanner 22 stores in data storage device 24. The raw data is then processed to obtain 3D image data.

[0076] The patient is transferred from station 32 at CT scanner 22 to station 34 at display system 28. A viewer uses display system 28 to view the patient's body 30. Display system 28 also reads image data from data storage device 24. The viewer uses display system 28 to view an image in the form of a 3D rendering of the patient's body 30. The viewer inserts catheter 26 into body 30. Display system 28 reads data from the tip of catheter 26 for further processing or display to the viewer.

[0077] FIG. 2 illustrates components of the vision system 20, including the CT scanner 22, the data storage device 24, the energy source 36, the data receiving unit 38, and the image generating unit 40.

[0078] The CT scanner 22 includes a base 42 , a platform 44 , a rotator 46 , an x-ray transmitter 48 , and a plurality of x-ray detectors 50 .

[0079] The platform 44 is secured to the base 42 through a mechanism (not shown) that allows translational movement of the platform 44 relative to the base 42. An actuator, such as a stepper motor (not shown), is operable to cause translational movement of the platform 44 relative to the base 42.

[0080] The rotor 46 has an opening 52. An X-ray transmitter 48 is affixed to the rotor 46 on one side of the opening 52, and an X-ray detector 50 is affixed to the rotor 46 on the opposite side of the opening 52. The rotor 46 is mounted to the base 42 about the periphery of the platform 44. The platform 44 moves relative to the opening 52 during its translation. A motor (not shown) is connected between the base 42 and the rotor 46 and is operable to rotate the rotor 46 about the periphery of the platform 44.

[0081] The energy source 36 may be connected to the X-ray transmitter 48 through a switch 54. The X-ray detector 50 is connected to a data receiving unit 38. The data receiving unit 38 may be a software unit residing on a computer-readable medium of a computer. The data storage device 24 resides on a computer-readable medium. The computer-readable medium may be a single computer-readable medium or may be separated in one personal computer or several personal computers interconnected over a network. The data receiving unit 38 is connected to the data storage device 24 either directly or via a network.

[0082] The image generation unit 40 may be a computer program resident on a computer readable medium. The image generation unit 40 is connected to the data storage device 24 either directly or via a network.

[0083] In use, the operator of the CT scanner 22 positions the patient with their body 30 lying on the platform 44. A motor connected between the base 42 and the rotor 46 is then switched on so that the rotor 46 rotates in a direction 58 about the platform 44 and the patient's body 30. The operator also switches on a motor that moves the platform 44 in a translational direction relative to the base 42 so that the platform 44 moves in a direction 60 relative to the rotor 46. The operator then connects the switch 54 between the energy source 36 and the x-ray transmitter 48, activating the x-ray transmitter 48. The x-ray transmitter then generates a forward-moving x-ray wave 62.

[0084] The patient's body 30 is positioned relative to the X-ray transmitter 48 so that the forward X-ray waves 62 penetrate the body 30 to a body part (not shown) within the body 30. For purposes of this example, the body part being scanned is the patient's lungs. The lungs have many bronchi through which a catheter may advance. It is also possible that a catheter may advance through hollow passages within the heart, arteries, and veins, such as the blood circulatory system. The systems described herein may also find use for viewing internal body parts without the use of a catheter for vision, surgery, or intervention, such as viewing growths within the abdomen or analyzing the internal function of a knee. The body part reduces the energy of the forward X-ray waves 62. Different materials within the body part reduce the energy by different amounts. One of the X-ray detectors 50 is positioned relative to the body 30 to detect reflected X-ray waves 64 from the body part. A reflected x-ray wave 64 from the body part is detected in response to the forward x-ray wave 62 and is essentially the forward x-ray wave 62 with a reduced power due to the reduction in power by the body part. A further forward x-ray wave 66 is also shown. Further x-ray waves are generated between the forward x-ray waves 62 and 66 and are detected by respective ones of the x-ray detectors 50. In this manner, reflected x-ray waves are received from different parts of the body part.

[0085] The x-ray transmitter 48 and x-ray detector 50 rotate with the rotor 46 around the body part within the patient's body 30. In this manner, the body part can be scanned from different angles to generate two-dimensional "slices" of the anatomy. CT scans can show bones, organs, and soft tissue. Subsequent slices are taken by moving the platform 44 in direction 60. Each slice therefore represents two-dimensional data, and together the slices represent data for the body part in three dimensions.

[0086] The data receiving unit 38 receives raw data of reflected X-ray waves 64 from the X-ray detectors 50. The raw data includes a time series correlation between the angle of the X-ray transmitter 48 relative to the body part within the patient's body 30, the energy detected by each of the X-ray detectors 50, the location of each of the X-ray detectors 50, and the position of the platform 44. The data receiving unit 38 stores the raw data as raw data 68 of reflected X-ray waves detected by the X-ray detectors 50.

[0087] When sufficient raw data 68 of the body part has been collected, the operator disconnects switch 54, stopping platform 44. The operator then stops rotator 46 and removes the patient from platform 44.

[0088] The image generation unit 40 reads the raw data 68 from the data storage device 24. The image generation unit 40 generates image data based on the raw data 68. The image data includes a three-dimensional rendering of the body part. The image generation unit 40 then stores the image data as image data 70 in the data storage device 24. The data storage device 24 may be a single data storage device or may be distributed among platforms; thus, the raw data 68 and the image data 70 may be located in a single data storage device within a personal computer or in several data storage devices within several personal computers.

[0089] FIG. 3 illustrates the components of the viewing system 20 in more detail, showing the data storage device 24 (which holds the image data 70), the catheter 26, and the display system 28.

[0090] The catheter 26 includes a lumen 76 and a tip 78 attached to the end of the lumen 76. The lumen is an elongated member (e.g., a hollow tubular member) that forms the majority of the length of the catheter 26. The lumen 76 contains a mechanism (not shown) that is operable to move the tip 78 in at least four orthogonal directions and in all directions between the orthogonal directions. The tip 78 is therefore steerable using the mechanism within the lumen 76. The lumen has a hollow bore that is large enough to hold the mechanism used to steer the tip, along with any electrical cables and / or optical fibers that may be required to relay signals from the tip to the display system 28 through the lumen 76.

[0091] The catheter 26 further includes a catheter inertial measurement unit (IMU) 80 and a catheter camera 82 affixed to the tip 78. The catheter IMU 80 may be, for example, a semiconductor chip having several measurement devices formed therein. The measurement devices include one or more gyroscopes and one or more accelerometers. Measurements from the gyroscopes and accelerometers, individually or in combination, provide data indicative of movement of the tip 78. Such movement can be tracked in six degrees of freedom, e.g., translation in the x, y, and z directions, and rotation about the x, y, and z axes.

[0092] The catheter camera 82 has a lens (not shown) on the side of the tip 78 facing the lumen 76. The catheter camera 82 is positioned to capture images in the form of live video data within the area in front of the tip 78, i.e., on the side facing the lumen 76. There may be multiple light sources and multiple cameras on different sides of the camera tip, but for ease of discussion, it will be assumed that there is only a single camera, e.g., an integrated camera and light source on the distal end of the catheter.

[0093] Display system 28 includes a head-mountable frame 86, left and right projectors 88A and 88B, left and right waveguides 90A and 90B, a detection device 92, and a vision algorithm 94. Left and right projectors 88A and 88B, left and right waveguides 90A and 90B, and detection device 92 are affixed to head-mountable frame 86. Head-mountable frame 86 is shaped to be mounted on the head of a viewer. Components of head-mountable frame 86 may include, for example, a strap (not shown) that wraps around the back of the viewer's head.

[0094] The left and right projectors 88A and 88B are connected to a power supply. Each projector 88A or 88B has an individual input through which image data is provided to the individual projector 88A or 88B. When powered, the individual projector 88A or 88B generates and emits light in a two-dimensional pattern. Left and right waveguides 90A and 90B are positioned to receive light from the left and right projectors 88A and 88B, respectively. The left and right waveguides 90A and 90B are transparent waveguides.

[0095] The sensing device 92 includes a head unit IMU 100 (more than one IMU) and one or more head unit cameras 102. The head unit IMU 100 includes one or more gyroscopes and one or more accelerometers. The gyroscopes and accelerometers are typically formed in semiconductor chips and are capable of detecting movement of the head unit IMU 100 and head-mountable frame 86, including movement along and rotation about three orthogonal axes.

[0096] The head unit camera 102 continuously captures images from the environment surrounding the head-mountable frame 86. The images can be compared to each other to detect movement of the head-mountable frame 86 and the viewer's head.

[0097] The vision algorithm 94 includes an image data receiving unit 106, a display positioning algorithm 108, a catheter integration system 110, a display adjustment algorithm 112, an image processing system 114, and a stereoscopic analyzer 116. The image data receiving unit 106 is connected to the data storage device 24 through a direct connection or via a network. The components of the vision algorithm 94 are linked to each other through subroutines or calls. Through such subroutines or calls, the image data receiving unit 106 is linked to the stereoscopic analyzer 116 via the display positioning algorithm 108.

[0098] The catheter integrated system 110 may be connected to the catheter IMU 80 and the catheter camera 82 through conductors within the lumen 76. Those skilled in the art will appreciate that the vision algorithms 94 may reside on a computing system, the catheter integrated system 110 may receive signals from the catheter camera 82 and the catheter IMU 80, and such signals may be converted from analog or digital data to computer software data. The catheter integrated system 110 may be connected to the stereo vision analyzer 116 through subroutines or calls.

[0099] The display adjustment algorithm 112 and the image processing system 114 are connected to the head unit IMU 100 and the head unit camera 102, respectively. Such connections are through conductors and, where applicable, through inverters that convert analog or digital data to computer software data. The display adjustment algorithm 112 may be connected to the display positioning algorithm 108 through subroutines and calls. The image processing system 114 may be connected to the display adjustment algorithm 112 through calls and subroutines.

[0100] In use, a viewer mounts head-mountable frame 86 on their head. Left and right waveguides 90A and 90B are then positioned in front of the viewer's left and right eyes 120A and 120B.

[0101] The image data receiving unit 106 reads image data 70 from the data storage device 24 and provides the image data 70 to the display positioning algorithm 108. The display positioning algorithm 108 inputs the image data 70 into a stereoscopic analyzer 116. The image data 70 is three-dimensional image data of a body part as described above. The stereoscopic analyzer 116 analyzes the image data 70 and determines left and right image data sets based on the image data 70. The left and right image data sets are data sets representing two-dimensional images that differ slightly from each other for the purpose of giving the viewer the perception of a three-dimensional rendering. The image data 70 is a static data set that does not change over time.

[0102] Stereoscopic analyzer 116 projects left and right image data sets into left and right projectors 88A and 88B. Left and right projectors 88A and 88B then create left and right light patterns 122A and 122B. The components of display system 28 are shown in plan view, and left and right light patterns 122A and 122B are shown in front elevation view. Each light pattern 122A and 122B includes multiple pixels. For purposes of illustration, light rays 124A and 126A from two of the pixels are shown exiting left projector 88A and entering left waveguide 90A. Light rays 124A and 126A reflect off the sides of left waveguide 90A. Although light rays 124A and 126A are shown propagating from left to right within left waveguide 90A through internal reflections, it should be understood that light rays 124A and 126A also propagate into the page using a system of refraction and reflection. Light rays 124A and 126A exit left optical waveguide 90A through pupil 128A and enter left eye 120A through left eye pupil 130A. Light rays 124A and 126A then impinge on retina 132A of left eye 120A. In this manner, left light pattern 122A impinges on retina 132A of left eye 120A. The viewer is given the perception that the pixels formed on retina 132A are pixels 134A and 136A, which the viewer perceives as being a distance away on the side of left waveguide 90A facing left eye 120A.

[0103] In a similar manner, stereoscopic analyzer 116 projects a right image data set into right projector 88B. Right projector 88B transmits right light pattern 122B, represented by pixels in the form of light rays 124B and 126B. Light rays 124B and 126B reflect within right waveguide 90B and exit through pupil 128B. Light rays 124B and 126B then enter through pupil 130B of right eye 120B and impinge on retina 132B of right eye 120B. The pixels of light rays 124B and 126B are perceived as pixels 134B and 136B behind right light waveguide 90B.

[0104] The patterns created on the retinas 132A and 132B are perceived individually as left and right images 140A and 140B shown in front elevation, which differ slightly from each other due to the function of the stereoscopic analyzer 116. The left and right images 140A and 140B are perceived as a three-dimensional rendering in the viewer's brain.

[0105] As mentioned, the left and right waveguides 90A and 90B are transparent. Light from real objects on the sides of the left and right waveguides 90A and 90B facing the eyes 120A and 120B can project through the left and right waveguides 90A and 90B and impinge on the retinas 132A and 132B. In particular, light from the surface of the patient's body 30 impinges on the retinas 132A and 132B so that the surface of the patient's body 30 is visible to the viewer. An augmented reality is created in which the surface of the patient's body 30 as seen by the viewer is augmented with a three-dimensional rendering perceived by the viewer due to the left and right images 140A and 140B perceived by the viewer in combination.

[0106] The head unit IMU 100 detects the overall movement of the viewer's head. If the viewer, for example, moves counterclockwise around the patient's body 30 while simultaneously rotating their head counterclockwise and continuing to look at the patient's body 30, such movement will be detected by the gyroscope and accelerometer within the head unit IMU 100. The head unit IMU 100 provides measurements from the gyroscope and accelerometer to a display adjustment algorithm 112. The display adjustment algorithm 112 calculates and provides the adjustment value to a display positioning algorithm 108. The display positioning algorithm 108 modifies the image data 70 to compensate for the viewer's head movement. The display positioning algorithm 108 provides the modified image data 70 to a stereoscopic analyzer 116 for display to the viewer.

[0107] The head unit camera 102 continuously captures images as the viewer moves their head. The image processing system 114 analyzes the images by identifying images of objects within the images. The image processing system 114 analyzes the object movement and determines the pose position of the head-mountable frame 86. The image processing system 114 provides the pose position to the display accommodation algorithm 112. The display accommodation algorithm 112 uses the pose position to further refine the setting values ​​that it provides to the display positioning algorithm 108. The display positioning algorithm 108 therefore modifies the image data 70 based on a combination of the motion sensors in the head unit IMU 100 and the images captured by the head unit camera 102.

[0108] The catheter integrated system 110 may detect the location of the tip 78 of the catheter 26 before the viewer inserts the tip 78 into the patient's body 30. The viewer subsequently inserts the tip 78 into the patient's body 30. The tip 78 then becomes invisible to the viewer. The catheter IMU 80 provides a signal indicative of the total movement of the tip 78 to the catheter integrated system 110. The catheter integrated system 110 can therefore track the position of the tip 78 using motion sensors within the catheter IMU 80. Unlike the image data 70, which is static, the position of the tip 78 changes over time. The catheter integrated system 110 provides the position of the tip 78 to the stereoscopic analyzer 116. The position of the tip 78 can be dynamic in that it changes over time and moves in three dimensions. The stereoscopic analyzer 116 locates the tip 78 within the left and right image data sets that are inserted into the left and right projectors 88A and 88B. The viewer can therefore see the location of tip 78 within left and right images 140A and 140B. The location of tip 78 varies slightly within left and right images 140A and 140B such that the viewer perceives the location of tip 78 in three dimensions. The rendering of the location of tip 78 as provided by left and right images 140A and 140B changes over time as tip 78 passes through patient body 30. Such movement of the location of tip 78 as a rendering changes in three dimensions such that the viewer perceives the rendering of tip 78 as it moves in three dimensions, i.e., left, right, up, down, front, back, etc.

[0109] The catheter camera 82 continues to capture video data and provides it to the catheter integrated system 110. The catheter integrated system 110 provides the video data to the stereoscopic analyzer 116. The stereoscopic analyzer 116 places the video data at a fixed location within the viewer's view unless or until a user interaction event is detected that indicates the location should change. The video data changes over time as different images are captured by the catheter camera 82.

[0110] The vision algorithm 94 is a set of instructions stored on a computer-readable medium in conjunction with the data storage device 24. The set of instructions is executable by a processor to perform the methods described above. The computer-readable medium storing the vision algorithm 94 may be located on a belt pack worn by the viewer.

[0111] FIG. 4 illustrates in more detail the components of the vision system 20, particularly the components of the catheter integrated system 110 and their relationship to the catheter IMU 80 and catheter camera 82 and stereoscopic analyzer 116 within the tip 78.

[0112] The catheter integrated system 110 includes a catheter tracking system 150, a past path calculator 152, a mesh generator 154, a predicted path calculator 156, a video data receiving unit 158, and a catheter display integrator 160. The catheter tracking system 150 is connected to the catheter IMU 80. The catheter tracking system 150 calculates the position of the tip 78 based on movements detected by the catheter IMU 80. The catheter IMU 80 includes several tip tracking devices, including several gyroscopes and accelerometers for tracking its movements in six degrees of freedom. The catheter tracking system 150 stores the current position of the tip 78 as a position 162 in the data storage device 24. The catheter tracking system 150 continues to monitor the catheter IMU 80, calculate the current position of the tip 78, and store the current position of the tip 78 as a current position 162 in the data storage device 24.

[0113] The catheter display integrator 160 receives the current position 162 from the data store 24 and provides the current position 162 to the stereoscopic analyzer 116. The stereoscopic analyzer 116 displays the current position 162 of the tip 78 to the viewer as a rendering such that the viewer can see the position of the tip 78 as a three-dimensional rendering.

[0114] The past path calculator 152 reads all positions 162 at all instants from the data store 24. The past path calculator 152 calculates the past path of the tip 78 in three dimensions and stores the past path as a past path 164 in the data store 24. The catheter display integrator 160 receives the past path 164 from the data store 24 and provides the past path 164 to the stereo analyzer 116. The stereo analyzer 116 displays the past path 164 to the viewer as a three-dimensional rendering.

[0115] The mesh generator 154 retrieves the past path 164 from the data store and generates a three-dimensional mesh around the past path 164. The mesh generator 154 then stores the mesh as mesh 166 in the data store 24. The catheter display integrator 160 retrieves the mesh 166 from the data store 24 and provides the mesh 166 to the stereoscopic analyzer 116. The stereoscopic analyzer 116 displays the mesh 166 to the viewer. The stereoscopic analyzer 116, in some embodiments, creates a three-dimensional rendering of the mesh 166 that overlays the past path 164.

[0116] The predicted path calculator 156 reads the overall position 162 of the tip 78 from the data store 24 and calculates a future path of the tip 78 based on the positions 162 and the past positions read from the data store 24. The predicted path calculator 156 then stores the future path as a future path 168 in the data store 24. The catheter display integrator 160 reads the future path 168 from the data store 24 and provides the future path 168 to the stereoscopic analyzer 116. The stereoscopic analyzer 116 displays the future path 168 to the viewer as a three-dimensional rendering.

[0117] The video data receiving unit 158 ​​receives live video from the catheter camera 82. The video data receiving unit 158 ​​provides the live video data to a catheter display integrator 160. The catheter display integrator 160 provides the live video data to a stereoscopic analyzer 116. The stereoscopic analyzer 116 displays the live video data to a viewer. The live video data is a two-dimensional display that is displayed to the viewer at a predetermined distance in three-dimensional space. The catheter display integrator also integrates a mesh 166 with the video data from the video data receiving unit 158 ​​such that the mesh 166 is displayed on the video data. As the video data changes with the changing position of the catheter 26 within the patient's body 30, the mesh 166 also changes accordingly.

[0118] FIG. 5 illustrates the use of a visualization system 20 as previously described by a viewer 172 in the form of a surgeon using catheter 26 as a bronchoscope for the purpose of examining a body part 174, including the segmental bronchi within a patient's lungs.

[0119] A viewer 172 can see the patient's body 30 through the left and right waveguides 90A and 90B. A body part 174 is inside the patient's body 30, and therefore the viewer cannot see the actual (i.e., physical) body part 174.

[0120] The viewer 172 also sees a three-dimensional rendering 176 based on the image data 70 as described above. In certain embodiments, the rendering 176 is located next to the patient's body 30. The rendering 176 is included in the figure to show where the viewer 172 perceives the rendering 176 relative to the patient's body 30, but it should be understood that from the perspective of the reader of this document, the rendering 176 does not exist in the real world. An inset 180 shows that the viewer 172 may see a three-dimensional rendering 182 of the body part 174 as part of the rendering 176.

[0121] The viewer 172 inserts the tip 78 of the catheter 26 into the patient's mouth. The viewer 172 then advances the tip 78 into the body part 174. The location of the tip 78 is monitored at closely spaced instances in time as described above, and its past path is stored in three dimensions. The sampling time may vary depending on the use case, with possible optimizations such as capturing data while the endoscope is inside the patient's body or only after the user activates the "Start Recording / Sampling" feature. Inset 184 shows that rendering 176 includes a rendering 186 of the location of the tip 78 in three dimensions and a rendering 188 of the past path of the tip 78 in three dimensions. Renderings 182, 186, and 188 may also be displayed simultaneously to the viewer 172, such that the viewer sees renderings 186 and 188 within rendering 182.

[0122] 6 is a top plan view showing the location of viewer 172 relative to patient body 30, and further illustrates the location of rendering 176 within the view of viewer 172. Rendering 176 may be placed in any position relative to patient body 30 based on user preference, pre-programmed default settings, or any other suitable means. The particular relative location of patient body 30 with respect to rendering 176 in FIG. 6 is for illustrative purposes only and should not be considered limiting in any way.

[0123] Figure 7 illustrates a view 192 as seen by viewer 172 of Figure 6. Viewer 172 may see the patient's actual body 30 and a rendering 176. View 192 also includes live video based on video data captured by catheter camera 82 of Figure 4. View 192 also shows a mesh 196 overlaying video 194. Mesh 196 is a representation of mesh 166 of Figure 4.

[0124] 8, viewer 172 is moving counterclockwise around patient body 30 and also rotating his or her head counterclockwise to keep patient body 30 in view. Display accommodation algorithm 112 detects the movement of viewer's 172's head and adjusts the position of rendering 176 accordingly so that rendering 176 appears to remain stationary relative to patient body 30 in view of viewer 172.

[0125] In FIG. 9, the patient's body 30 has been rotated clockwise relative to FIG. 7. The rendering 176 has also been rotated clockwise so as to remain stationary relative to the patient's body 30. However, the location of the live video 194 does not change from the view 192 in FIG. 7 to the view 192 in FIG. 9. The viewer 172 therefore sees the live video 194 and the mesh 196 in the same location, and these components do not move in response to movement of the viewer's 172's head. The viewer 172 can therefore view the patient's body 30 and the rendering 176 from different sides and angles without losing view of the live video 194 and the mesh 196. The purpose of the mesh 196 may be to help the viewer 172 guide the tip 78 of the catheter 26 when the viewer 172 inserts the tip 78 into the passageway within the body part 174 a second time after the mesh is created or during catheter removal as the catheter moves through the same path in the opposite direction. Some embodiments may have different viewing configurations for the virtual content (e.g., mesh 196, live video 194, rendering 176) where some or all of the virtual content is fixed relative to real-world coordinates or fixed relative to the viewer.

[0126] 10 shows components of rendering 176 displayed to the viewer that are too small to see in the views of FIGS. 7 and 9. Viewer 172 sees renderings 182, 186, and 188 of body part 174, tip 78, and the tip's past path. The viewer also sees a three-dimensional rendering of mesh 196. Mesh 196 is shown separate from renderings 182, 186, and 188 for illustrative purposes, but it should be understood that mesh 196 may be overlaid on rendering 182 of body part 174.

[0127] The implementation described above uses a CT scanner 22 to scan the body part 174. The CT scanner has a transmitter in the form of an X-ray transmitter and a receiver in the form of an X-ray detector to transmit and receive waves in the form of X-ray waves. Other scanning devices using other transmitters and receivers and transmitting and detecting different waves are possible. For example, a sonar system uses an audio transmitter to transmit sound waves and an audio receiver to receive sound waves. A vision system may include a light source inserted into the body part that transmits light waves and a camera located within the body part that captures light waves reflected from the body part.

[0128] However, CT scanners are preferred over other scanning devices because they provide extremely high-detail raw data of the body part in three dimensions, and such data can be easily transformed using an image generation unit to create three-dimensional image data. CT data also has the advantage that it may include data regarding specific substances, materials, and material densities. The described implementation shows a rendering 176 placed next to the patient's body 30 within the view 192 of a viewer 172. It may also be possible to match the rendering with the patient's body 30, such that the rendering of the body part is where the actual body part is, and the rendering of the catheter tip is where the actual position of the catheter tip is.

[0129] Aspects of the invention can also be implemented without a catheter. For example, it is possible to scan a patient's body to determine growth and for a viewer to use a display system to overlay a rendering of the growth in three dimensions onto the patient's actual body. In this way, the viewer can "see" the growth "within" the patient's actual body.

[0130] Figure 11 of the accompanying drawings illustrates a vision system 210 according to another embodiment of the invention, including a first vision device 212.1 worn by a first user 214.1, a real object in the form of a table 216, a network 218, and a server 220.

[0131] The first viewing device 212.1 includes a head unit 222, a belt pack 224, and a cable connection 226. The first user 214.1 has the head unit 222 secured to their head and the belt pack 224 on their waist, remote from the head unit 222. The cable connection 226 connects the head unit 222 to the belt pack 224. The head unit 222 includes technology used to display a virtual object or objects to the first user 214.1 while still allowing the first user 214.1 to see real objects, such as the table 216. The belt pack 224 primarily contains the processing and communication capabilities of the first viewing device 212.1. In another embodiment, the processing and communication capabilities may reside entirely within the head unit 222, thus eliminating the need for the belt pack 224, or may be located in another device, such as a backpack.

[0132] The belt pack 224 is connected to the network 218 via a wireless connection. The server 220 is connected to the network 218 and holds data representing local content. The belt pack 224 downloads the data representing the local content from the server 220 via the network 218. The belt pack 224 provides the data to the head unit 222 via a cable connection 226. The head unit 222 typically includes a display having a light source, e.g., a laser light source or a light emitting diode (LED) light source, and a waveguide to guide the light.

[0133] In use, a first user 214.1 wears the head unit 222 on their head and the belt pack 224 on their waist. The belt pack 224 downloads image data from the server 220 via the network 218. The first user 214.1 can view the table 216 through the display of the head unit 222. A projector forming part of the head unit 222 receives the image data from the belt pack 224 and generates light based on the image data. The light travels through one or more waveguides forming part of the display of the head unit 222. The light then exits the waveguides and propagates onto the retina of the first user's 214.1 eye. The projector generates light in a pattern that is replicated on the retina of the first user's 214.1 eye. The light that falls on the retina of the eye of the first user 214.1 has a selected depth of field so that the first user 214.1 perceives an image at a preselected depth behind the waveguide. In addition, the first user's 214.1 eyes receive slightly different images so that the first user's 214.1 brain perceives a three-dimensional image or multiple images at a selected distance from the head unit 222. In this example, the first user 214.1 perceives the local content 228 as augmented while viewing the table 216. The proportion of the local content 228 and its location and distance from the first user 214.1 are determined by the data representing the local content 228 and the various coordinate frames used to display the local content 228 to the first user 214.1.

[0134] The local content 228 is invisible from the perspective of the drawing and is only visible to the first user 214.1 due to their use of the first viewing device 212.1. The local content 228 initially resides as a data structure in the visual data and algorithms within the belt pack 224. The data structure then appears as light when the projector in the head unit 222 generates light based on the data structure. Although the local content 228 does not exist in three-dimensional space in front of the first user 214.1, the local content 228 is still represented in three-dimensional space in FIG. 1. Visualization of computer data in three-dimensional space is used throughout this description to illustrate how data structures interrelate among the data structures within the belt pack 224, facilitating rendering as perceived by one or more users.

[0135] FIG. 12 illustrates the components of the first viewing device 12.1 in more detail, including a head unit 222, a rendering engine 230, and various components that form part of the visual data and algorithms, including various coordinate systems 232, various origin and destination coordinate frames 234, and various origin / destination coordinate frame converters 236.

[0136] The head unit 222 includes a head-mountable frame 240 , a display system 242 , a real object detection camera 244 , a motion tracking camera 246 , and an inertial measurement unit 248 .

[0137] The head-mountable frame 240 has a shape that allows it to be attached to the head of a first user 214.1 in Figure 11. The display system 242, the real object detection camera 244, the mobile tracking camera 246, and the inertial measurement unit 248 are mounted on the head-mountable frame 240 and therefore move with the head-mountable frame 240.

[0138] The coordinate system 232 includes a local data system 252 , a world frame system 254 , a head frame system 256 , and a camera frame system 258 .

[0139] Local data system 252 includes a data channel 262, a local frame determination routine 264, and a local frame storage instruction 266. Data channel 262 can be an internal software routine, a hardware component such as an external cable or radio frequency receiver, or a hybrid component such as an open port. Data channel 262 can receive image data 268, which represents local content.

[0140] A local frame determination routine 264 is connected to the data channel 262. The local frame determination routine 264 determines a local coordinate frame 270. The local coordinate frame may be based, for example, on the top edge relative to the bottom edge of the browser window, the character's head and feet, etc. A local frame storage instruction 266 is connected to the local frame determination routine 264. Those skilled in the art will understand that software modules and routines are "connected" to one another through subroutines, calls, etc. The local frame storage instruction 266 stores the local coordinate frame 270 as a local coordinate frame 272 within the origin and destination coordinate frame 234.

[0141] The rendering engine 230 is connected to the data channel 262. The rendering engine 230 receives image data 268 from the data channel 262.

[0142] A display system 242 is connected to the rendering engine 230. The display system 242 includes components that convert image data 268 into visible light. The visible light forms two patterns, one for each eye. The visible light enters the eye of the first user 214.1 in FIG. 11 and is detected on the retina of the eye of the first user 214.1.

[0143] The real object detection camera 244 represents one or more cameras that capture images from different sides of the head-mountable frame 240. The moving tracking camera 246 also represents one or more cameras that capture images on either side of the head-mountable frame 240. One camera may be used instead of two cameras, representing the real object detection camera 244 and the moving tracking camera 246.

[0144] The inertial measurement unit 248 includes several devices used to detect movement of the head unit 222. The inertial measurement unit 248 may include a gravity sensor, one or more accelerometers, and one or more gyroscopes. The sensors of the inertial measurement unit 248 combine to track movement of the head unit 222 in at least three orthogonal directions and about at least three orthogonal axes.

[0145] The world frame system 254 includes a world surface determination routine 278, a world frame determination routine 280, and world frame storage instructions 282. The world surface determination routine 278 is connected to the real object detection camera 244. The world surface determination routine 278 receives images captured by the real object detection camera 244, processes the images, and identifies surfaces within the images. A depth sensor (not shown) determines the distance to the surface. The surface is therefore represented by data in three dimensions, including its size, shape, and distance from the real object detection camera. The world frame determination routine 280 is connected to the world surface determination routine 278 and determines a world coordinate frame 284 based on the location of the surface as determined by the world surface determination routine 278. The world frame storage instructions 282 are connected to the world frame determination routine 280 and receive the world coordinate frame 284 from the world frame determination routine 280. The world frame storage instructions 282 store the world coordinate frame 284 in the origin and destination coordinate frame 234 as a world coordinate frame 286.

[0146] The head frame system 256 includes a head frame determination routine 290 and head frame storage instructions 292. The head frame determination routine 290 is connected to the moving tracking camera 246 and the inertial measurement unit 248. The head frame determination routine 290 uses data from the moving tracking camera 246 and the inertial measurement unit 248 to calculate a head coordinate frame 294. For example, the inertial measurement unit 248 has a gravity sensor that determines the direction of gravity relative to the head unit 222. The moving tracking camera 246 continuously captures images that are used by the head frame determination routine 290 to refine the head coordinate frame 294. The head unit 222 moves as the first user 214.1 in FIG. 11 moves their head. The moving tracking camera 246 and the inertial measurement unit 248 continuously provide data to the head frame determination routine 290 so that the head frame determination routine 290 can update the head coordinate frame 294.

[0147] The head frame store instruction 292 is connected to the head frame determination routine 290 and receives a head coordinate frame 294 from the head frame determination routine 290. The head frame store instruction 292 stores the head coordinate frame 294 as a head coordinate frame 296 in the origin and destination coordinate frame 234. When the head frame determination routine 290 recalculates the head coordinate frame 294, the head frame store instruction 292 repeatedly stores the updated head coordinate frame 294 as the head coordinate frame 296.

[0148] The camera frame system 258 includes camera intrinsics 298, which are the dimensions of the head unit 222 that are characteristic of its design and manufacture. The camera intrinsics 298 are used to calculate a camera coordinate frame 300, which is stored in the origin and destination coordinate frame 234.

[0149] The camera coordinate frame 300 includes all pupil positions for the left eye of the first user 214.1 in Figure 11. As the left eye moves left to right or up and down, the pupil positions for the left eye are located within the camera coordinate frame 300. In addition, the pupil positions for the right eye are located within the camera coordinate frame 300 for the right eye.

[0150] The origin / destination coordinate frame converter 236 includes a local / world coordinate converter 304, a world / head coordinate converter 306, and a head / camera coordinate converter 308. The local / world coordinate converter 304 receives the local coordinate frame 272 and converts the local coordinate frame 272 to a world coordinate frame 286. The transformation of the local coordinate frame 272 to the world coordinate frame 286 is represented as a local coordinate frame that is transformed to a world coordinate frame 310 within the world coordinate frame 286.

[0151] The world / head coordinate converter 306 transforms from the world coordinate frame 286 to the head coordinate frame 296. The world / head coordinate converter 306 transforms the local coordinate frame, which is transformed into the world coordinate frame 310, to the head coordinate frame 296, and the transformation is expressed as a local coordinate frame, which is transformed into the head coordinate frame 312 within the head coordinate frame 296.

[0152] The head / camera coordinate converter 308 converts from the head coordinate frame 296 to the camera coordinate frame 300. The head / camera coordinate converter 308 converts the local coordinate frame, which is converted to the head coordinate frame 312, to a local coordinate frame, which is converted to the camera coordinate frame 314 in the camera coordinate frame 300. The local coordinate frame, which is converted to the camera coordinate frame 314, is input into the rendering engine 230. The rendering engine 230 displays the image data 268, which represents the local content 228, based on the local coordinate frame, which is converted to the camera coordinate frame 314.

[0153] Figure 13 is a spatial representation of the various origin and destination coordinate frames 234. The local coordinate frame 272, world coordinate frame 286, head coordinate frame 296, and camera coordinate frame 300 are represented in the figure. Each camera has its own camera coordinate frame 300 that encompasses all pupil positions for one eye. Reference numerals 304A and 306A represent the transformations performed by local-to-world coordinate converter 304, world-to-head coordinate converter 306, and head-to-camera coordinate converter 308 in Figure 12, respectively.

[0154] By giving virtual content its own coordinate frame, as opposed to being measured directly relative to the world coordinate frame, the virtual content can be given a more persistent frame position. For example, if a virtual lamp is placed on a table, there can be multiple data points on the table to provide placement input for the virtual lamp's relative positioning that does not substantially change over time. In contrast, if the world map is created as a function of certain orientation and position, and the user changes position or orientation, thus forcing a new world coordinate frame, the virtual lamp may continue to utilize the same local coordinate frame rather than adjusting for the new world frame, which can introduce jitter or position shifts into the lamp's appearance.

[0155] FIG. 14 depicts a camera rendering protocol for transforming from the head coordinate frame to the camera coordinate frame. The pupil for one eye moves from position A to B. A virtual object, intended to appear stationary, would be projected onto the depth plane at one of two positions A or B, depending on the position of the pupil (assuming the camera is configured to use the pupil as its coordinate frame). As a result, using a pupil coordinate frame that is transformed into the head coordinate frame would cause jitter in the stationary virtual object as the eye moves from position A to position B. This situation is referred to as view-dependent display or projection.

[0156] As depicted in Figure 15, the camera rendering (CR) frame is positioned to encompass all pupil positions, and the object projection will now be consistent regardless of pupil positions A and B. The head coordinate frame is transformed into the CR frame, which is referred to as view-independent display or projection. Image reprojection may be applied to the virtual content to account for changes in eye position; however, because the rendering is still in the same position, jitter is minimized.

[0157] 16 illustrates in more detail the first viewing device 212.1, the second viewing device 212.2, and the visual data and algorithms of the server 220. Although not shown, the first viewing device 212.1 is configured identically to the second viewing device 212.2.

[0158] The server 220 includes a map storage routine 318, a reference map 320, a map transmitter 322, and a map merging algorithm 324 that acts as a central server-side map generator.

[0159] In use, the first viewing device 212.1 generates a local tracking map (hereafter referred to as "Map 1"), and the map storage routine 318 receives Map 1 from the first viewing device 212.1. The map storage routine 318 then stores Map 1 on the storage device of the server 220 as the reference map 320.

[0160] The second viewing device 212.2 includes a map download system 326, an anchor identification system 328, a location module 330, a reference map embedder 332, a local content location system 334, and a map publisher 336.

[0161] In use, the map transmitter 322 transmits the reference map 320 to the second viewing device 212.2, and the map download system 326 downloads the reference map 320 from the server 220 and stores it as the reference map 333.

[0162] The anchor identification system 328 is connected to the world surface determination routine 278. The anchor identification system 328 identifies anchors based on objects detected by the world surface determination routine 278. The anchor identification system 328 uses the anchors to generate a second map (Map 2). As shown by cycle 338, the anchor identification system 328 continues to identify anchors and update Map 2. The locations of the anchors are recorded as three-dimensional data based on the data provided by the world surface determination routine 278. The world surface determination routine 278 receives images from the real object detection camera 244 and depth data from the depth sensor 335 and determines the location of the surface and its relative distance from the depth sensor 335.

[0163] Location module 330 is connected to reference map 333 and map 2. Location module 330 repeatedly attempts to locate map 2 relative to reference map 333. Reference map embedder 332 is connected to reference map 333 and map 2. Once location module 330 locates map 2 relative to reference map 333, reference map embedder 332 embeds reference map 333 into the anchor of map 2. Map 2 is then updated with the missing data contained in the reference map.

[0164] A local content positioning system 334 is connected to map 2. The local content positioning system 334 may be, for example, a system in which a user can locate local content at a specific location in a world coordinate frame. The local content then attaches itself to an anchor in map 2. The local-to-world coordinate converter 304 converts the local coordinate frame to the world coordinate frame based on the settings of the local content positioning system 334. The functionality of the rendering engine 230, the display system 242, and the data channel 262 is described with reference to FIG. 12.

[0165] Map publisher 336 uploads Map 2 to server 220. Map storage routine 318 of server 220 then stores Map 2 in the server's 220 storage medium.

[0166] A map merge algorithm 324 merges map 2 with the reference map 320. When more than two maps are stored, for example, three or four maps, the map merge algorithm 324 merges all maps into the reference map 320 and renders the new reference map 320. The map transmitter 322 then transmits the new reference map 320 to every device 212.1 and 212.2 that is within the area represented by the new reference map 320. Once devices 212.1 and 212.2 locate their respective maps relative to the reference map 320, the reference map 320 becomes the elevation map.

[0167] FIG. 17 illustrates map 1 and local content (content 123 and content 456) on a first viewing device 212.1. Map 1 includes several anchors (anchor a-anchor d). From the perspective of the first viewing device 212.1, anchor a, for example, has X, Y, and Z coordinates of (0,0,0), and anchor b has X, Y, and Z coordinates of (-1,0,0). Content 123 is associated with anchor a. Content 123 may be, for example, a virtual object, such as a virtual implant, that needs to be related to anchor a so that all users will see the virtual implant in the same location after their individual systems are localized relative to the reference map. In this example, content 123 has an X, Y, and Z relationship to the anchor at (1,0,0). Content 456 has a relationship to anchor b. In this example, content 456 has an X, Y, and Z relationship of (1,0,0) to anchor b. Map 1 also has an origin (origin 1).

[0168] In Figure 18, a first viewing device 212.1 uploads Map 1 to a server 220. The server 220 now has a reference map based on Map 1. The first viewing device 212.1 has a reference map that is empty at this stage. The server 220, for purposes of discussion, does not contain any other maps other than Map 1. No maps are stored on the second viewing device 212.2.

[0169] The first viewing device 212.1 also transmits its Wi-Fi signature data to the server 220. The server 220 may use the Wi-Fi signature data to determine the general location of the first viewing device 212.1 based on intelligence gathered from other devices that have previously connected to the server 220 or other servers, along with the recorded GPS locations of such other devices.

[0170] The first viewing device 212.1 may now end the first session (see FIG. 11) and disconnect from the server 220.

[0171] FIG. 19 illustrates the start of a second session by a second user 214.2. The first user 214.1 is shown in phantom because the first session by the first user 214.1 has ended. The second viewing device 212.2 begins recording objects. Various systems, with varying granularity, may be used by the server 220 to determine that the second session by the second viewing device 212.2 is in the same vicinity as the first session by the first viewing device 212.1. For example, Wi-Fi signature data, Global Positioning System (GPS) positioning data, GPS data based on the Wi-Fi signature data, or any other data indicating location may be included within the first and second viewing devices 212.1 and 212.2 to record their locations. Alternatively, the anchors identified by the second viewing device 212.2 may show similarity to anchors on Map 1.

[0172] 20, the first and second viewing devices 212.1 and 212.2 download a reference map 320 from the server 220. Map 1 on the second viewing device 212.2 includes an anchor ad and an origin 1. The server 220 may have multiple reference maps for various locations and may determine that the second viewing device 212.2 is in the same vicinity as the first viewing device 212.1 during the first session and send the reference map for that vicinity to the second viewing device 212.2.

[0173] FIG. 21 shows that the second viewing device 212.2 begins identifying anchors for purposes of generating Map 2. The second viewing device 212.2 has identified only a single anchor, namely, anchor a. The anchor's X, Y, and Z coordinates for the second viewing device 212.2 are (1,1,1). The X, Y, and Z coordinates of anchor a are therefore different for Map 2 of the second viewing device 212.2 than those determined for Map 1 and the reference map of the first viewing device 212.1. Map 2 also has its own origin (Origin 2). The second viewing device 212.2 immediately attempts to locate Map 2 relative to the reference map. Because Map 2 has an insufficient number of anchors for purposes of locating it relative to the reference map, the location attempt fails.

[0174] 22 shows map 2 after the second viewing device 212.2 has identified additional anchors (anchor b, anchor c, and anchor e) on map 2. The second viewing device 212.2 again attempts to locate map 2 relative to the reference map. Because map 2 has at least two anchors (anchor a and anchor b) that match two anchors on the reference map, the location attempt will be successful.

[0175] Additionally, the second viewing device 212.2 has content 123 and content 456 associated with anchors a and b on map 2. Content 123 has X, Y, and Z coordinates relative to anchor a of (1,0,0). The coordinates of content 123 relative to anchor a are therefore the same for the second viewing device 212.2 and the first viewing device 212.1.

[0176] Similarly, the X, Y, and Z coordinates of content 456 relative to anchor b in map 2 are (1,0,0). The X, Y, and Z coordinates of content 456 relative to anchor b for the second viewing device 212.2 are therefore the same as for the first viewing device 212.1 in FIG.

[0177] FIG. 23 illustrates the successful location of map 2 relative to the reference map. Anchors a, b, and c are common to map 1 and the reference map. The reference map also has anchor d, which is not included in map 2, and map 2 has anchor e, which is not included in the reference map. Note that with respect to the second viewing device 212.2, content 123 is co-located with respect to the anchors for the second viewing device 212.2 as illustrated in FIG. 23 as with respect to the first viewing device 212.1 as illustrated in FIG. 17. Content 456 is also co-located with respect to anchor b, with respect to the second viewing device 212.2 and the first viewing device 212.1. The first and second users 214.1 and 214.2 therefore perceive content 123 and content 456 in the same location in the real world.

[0178] 24, the second viewing device 212.2 expands map 2 to include anchor d within map 1. The inclusion of anchor d represents the start of the expansion of map 2.

[0179] 25, the second viewing device 212.2 continues to expand Map 2 as additional anchors (anchors f, g, and h) are identified by the second viewing device 212.2, e.g., as the user walks around the real world. Also, note that Map 1 is not expanded in FIGS. 24 and 25.

[0180] 26, the second viewing device 212.2 uploads Map 2 to the server 220. The server 220 stores Map 2 along with the reference map.

[0181] The reference map in server 220 now includes anchor i, which is not included in map 1 on first viewing device 212.1. The reference map on server 220 may be expanded to include anchor i when a third viewing device (not shown) uploads a map to server 220 and such map includes anchor i.

[0182] 27, the server 220 merges Map 2 with the reference map. The server 220 determines that anchor ad is common to the reference map and Map 2. The server 220 extends the reference map to include anchor eh from Map 2, forming a new reference map. The reference maps on the first and second viewing devices 212.1 and 212.2 are based on Map 1 and are out of date.

[0183] 28, the server 220 transmits the new reference map to the first and second viewing devices 212.1 and 212.2, which proceed to locate their respective local maps (Map 1 and Map 2, respectively) relative to the new reference map, as described above.

[0184] As shown in Figure 29, head coordinate frame 296 or "head pose" is relative to an anchor in map 2. The anchor in map 2 serves as a world coordinate frame, and the transformation from the world coordinate frame to head coordinate frame 296 was described above with reference to Figure 22. Head coordinate frame 296 shown in Figure 29 has only two orthogonal axes at a particular coordinate location relative to the anchor in map 2 and at a particular angle relative to map 2. However, it should be understood that head coordinate frame 296 is at a three-dimensional location relative to the anchor in map 2 and has three orthogonal axes in three-dimensional space.

[0185] In Figure 30, the head coordinate frame 296 is moving relative to the anchor of map 2. The head coordinate frame 296 is moving because the second user 214.2 is moving their head. A user can move their head with six degrees of freedom (6 dof). The head coordinate frame 296 can therefore move in 6 dof, i.e., in three dimensions from its original location in Figure 29 and about three orthogonal axes relative to the anchor of map 2. The head coordinate frame 296 is adjusted as the real object detection camera 244 and the inertial measurement unit 248 in Figure 12 detect movement of the real object and the head unit 222, respectively.

[0186] FIG. 31 shows that sounds can be associated with one or more anchors. A user may wear headphones or earphones, for example, with stereo sound. The location of the sound through the headphones can be simulated using conventional techniques. The location of the sound may be located at a constant position such that as the user rotates their head to the left, the location of the sound rotates to the right, and the user perceives the sound as originating from the same location in the real world. In this example, the location of the sound is represented by sound 123 and sound 456. For purposes of discussion, FIG. 31 is similar in its analysis to FIG. 25. When first and second users 214.1 and 214.2 are located in the same room, at the same or different times, they perceive sound 123 and sound 456 as originating from the same location in the real world.

[0187] 32 and 33 illustrate further implementations of the techniques described above. A first user 214.1 begins a first session as described with reference to FIG. 11. As shown in FIG. 32, the first user 214.1 ends the first session, as indicated by the phantom line. At the end of the first session, the first viewing device 212.1 uploaded Map 1 to the server 220. The first user 214.1 now begins a second session at a time later than the first session. The first viewing device 212.1 does not download Map 1 from the server 220 because Map 1 is already stored on the first viewing device 212.1. If Map 1 is lost, the first viewing device 212.1 downloads Map 1 from the server 220. The first viewing device 212.1 then proceeds to build anchors for Map 2 as described above, locating it relative to Map 1 and further developing the reference map. Map 2 is then used to associate local content, head coordinate frames, local sounds, etc. as described above.

[0188] 34 and 35, it is also possible that more than one user may interact with server 220 in the same session. Multiple users at the same location have the added benefit of leading to more accurate anchors relative to the head coordinate frame. Multiple systems tend to detect more anchors, which leads to more cross-validation of maps and better scoring. In this example, first user 214.1 and second user 214.2 are joined by a third user 214.3 with a third viewing device 212.3. Each viewing device 212.1, 212.2, and 212.3 begins to generate its own map, i.e., Map 1, Map 2, and Map 3, respectively. As viewing devices 212.1, 212.2, and 212.3 continue to develop Maps 1, 2, and 3, the maps are progressively uploaded to server 220. Server 220 merges maps 1, 2, and 3 to form a reference map, which is then transmitted from server 220 to each of viewing devices 212.1, 212.2, and 212.3.

[0189] 36, the vision system features an illustrated head unit 402, a handheld controller component 404, and an interconnected ancillary computing or controller component 406, which may be configured to be worn on a user as a belt pack or equivalent. Each of these components may be operatively coupled 410, 412, 414, 416, 417, 418 to each other and to other connected resources 408, such as cloud computing or cloud storage resources, via wired or wireless communication configurations, such as those defined by IEEE 802.11, Bluetooth® (RTM), and other connectivity standards and configurations. See, for example, U.S. Patent Application Nos. 14 / 555,585, 14 / 690,401, 14 / 331,218, 15 / 481,255, 62 / 627,155, 62 / 518,539, 16 / 229,532, 16 / 155,564, 15 / 413,284, 16 / 020,541, 62,702,322, 62 / 206,765, 15,597,694, 16 / 221,065, and 15 / 968,673. , 62 / 682,788, and 62 / 899,678 (each of which is incorporated herein by reference in its entirety), various aspects of such components are described, such as various embodiments of two depicted optical elements 420 through which a user may view the world around them, along with visual components that may be produced by associated system components, for an augmented reality experience. Such systems may also comprise various sensors, including, but not limited to, various camera-type sensors (such as monochrome, color / RGB, and / or thermal imaging components) 422, 424, 426, a depth camera sensor 428, and / or a sound sensor 430, such as a microphone, configured to provide information about the user's surrounding environment.

[0190] FIG. 37 illustrates a more comprehensive vision system 432 including a head unit 402A, a head unit 402B, a head unit 402C, a fixed sensor 434, a movable sensor 436, a reference map 438, a guidance module 440, a personal computer 442, and a tablet computer 444.

[0191] Each of the head units 402A, 402B, and 402C includes a separate set of head unit sensors 446 and a separate head unit display 448. The head units 402A and 402B may be located in the same room, and the head unit 402C may be located in a different room, remote from the room in which the head units 402A and 402B are located. The head unit display 448, personal computer 442, and table computer 444 represent different display devices 450 through which a user may view two-dimensional or three-dimensional images, as described with reference to FIG. 34 above. The head unit sensors 446 may be used to detect the head frame of each of the head units 402A, 402B, or 402C, as described above. Additionally, the head unit sensors 446 of the head units 402A and 402B may be used to detect objects in the room in which they are located. These sensors are particularly useful for sensing objects the user is looking at. For example, if a user of the head unit 402A is looking at a body part of a patient, the head unit sensor 446 of the head unit 402A will also detect the body part of the patient.

[0192] Fixed sensors 434 are sensors mounted in fixed locations within the room. Fixed sensors 434 may be used to detect stationary objects within the room or, more generally, objects that move within the room, such as surgical personnel, robots, cutting tools on robots, surgical implants, surgical tools, disposable items, patients, and patient body parts. Mobile sensors 436 represent sensors that may be located on mobile objects, such as robots, that move within the room. Fixed sensors 434, mobile sensors 436, and head unit sensors 446 of head units 402A and 402B represent real object detection device 452. At 454, real object detection device 452 is responsible for continuously updating fiducial map 438, as described above with particular reference to FIG. 35 .

[0193] The guidance module 440 includes a room setup module 458, an anatomical alignment module 460, a surgical planning module 462, and a surgical execution module 464. The reference map 438 serves as a digital representation (also sometimes referred to as a "digital twin") of the real object in the real environment as detected by the real object detection device 452. The guidance modules 440 are connected to and can read the reference map 438 from a data storage device that holds the reference map 438. Each of the guidance modules 440 is executable to navigate at least one of the virtual object and the real object based on the digital representation. The guidance modules 440 may also provide output to the display device 450. Because the real object detection device 452 continuously detects real objects, the reference map 438 is continuously updated. The guidance module 440 continuously modifies its guidance and visual output in response to changes in the reference map 438. The guidance module 440 typically resides on a data storage device of a server computer system and is executed by a processor of the server computer system. Part or all of the guidance module 440 may also be executed by another computer system, such as any one of the head units 402A-402C.

[0194] Figure 38 illustrates the functionality of the room settings module 458 in Figure 37. At 470, the processor of the server computer system stores the desired room layout. The desired room layout is typically configured by the operator and depends on the specific proportions of the room and the particular surgical procedure for which the room is intended.

[0195] At 472, the processor provides an output that digitally overlays the desired room layout onto the real environment. The reference map 438 is used to determine the existing real environment. The operator overlays the desired room layout onto the reference map 438.

[0196] At 474, an image of a desired placement of one of the real objects in the desired room layout is generated, and the image is superimposed over the real environment. At 476, the operator moves the real object within the real environment. The operator moves the real object in a direction toward the image of the desired placement of the real object.

[0197] At 478, the movements of the individual real objects are tracked. The real objects are tracked because the real object detection device 452 detects the real objects and because the map generator, e.g., the map merge algorithm 324 in Figure 16, updates the reference map 438 as the individual real objects move.

[0198] At 480, the system provides an output indicating that the individual real objects have been moved to positions that match the desired placement. The output may be, for example, a color change in the image of the desired placement. The operator may repeat steps 474-480 to move additional real objects and continue to match the real objects to the desired room layout. The room may then be used to perform a surgical procedure on the patient.

[0199] Following the patient's surgery, the user may again use the system to disassemble the room. At 482, the system provides an output that digitally overlays the desired room disassembly layout onto the real environment. The operator may then move the real objects in reverse order. Element 484 represents the system tracking items discarded during surgery and providing an output of items for replacement. As an example, if the desired room equipment requires 100 cotton balls and 30 cotton balls were used during surgery, the system provides an output indicating that 30 cotton balls should be replaced.

[0200] 39 illustrates a user 490 being asked to set up a room for a surgical procedure. The user 490, wearing a head unit such as head unit 402A, is presented with a display of a room setup menu 492. The user 490 uses their hand or handheld controller component 404 to select items from the menu 492.

[0201] FIG. 40 illustrates a user 490 and a remote user 494. The remote user 494 is tasked with designing and storing a room layout using one of the personal computers 442. A fixed sensor 434 detects several real objects in the room, e.g., a table 496. An image of a virtual object 498 is displayed to the user 490, representing a desired location for the table 496. The user 490 then moves the table 496 in a direction 500. The movement of the table 496 in the direction 500 is tracked by the fixed sensor 434, the head-unit sensor 446, and potentially also the movable sensor 436 mounted on the table 496. The image 498 is originally one color, e.g., red, and changes to another color, e.g., green, as the table 496 is located across the image 498. The user 490 then repeats the process with other objects until all objects are in their desired locations. Once the surgery is completed, the user 490 is presented with an exploded map that the user 490 can use to return all objects to their original positions.

[0202] FIG. 41 shows one of the real objects located by a user 490: a surgical robotic system 504. The surgical robotic system 504 includes a surgical instrument 534, a movable arm 532, and a movable base 536. Such systems are available from vendors such as Stryker (RTM), Intuitive Surgical (RTM), and Johnson & Johnson (RTM), and once “registered” to the patient's anatomy, may be utilized for a variety of surgical procedures such that detailed and precise geometric relationships between portions of the surgical robotic system, such as the surgical instrument 534, and the patient's anatomy are understood to facilitate their precise coordination with one another in three-dimensional space. In other words, in orthopedic surgery, where it is desired to cut a portion of a patient's bone with a bone-cutting surgical instrument, it is, of course, important to precisely understand the location of the bone in space relative to the instrument. In various embodiments, it is useful for surgical operators to wear a system such as that illustrated in FIG. 36 so that they can not only visualize the operating room around them, but also visualize virtual elements such as pre-operative images, intra-operative images, alternate views of various items, and understand the geometric relationships of various objects, such as through the use of a common coordinate system (or “persistent coordinate frame” or “PCF”) that can be established and utilized by one or more spatial computing users, to which certain anatomical structures of the users and also certain aspects of the surgical instruments or systems may also be registered. With one or more user views registered to the PCF, along with surgical instruments and anatomical structures, virtual elements may be presented to the users to assist in not only planning but also execution of the surgical procedure.

[0203] FIG. 42 illustrates various aspects of the room settings module 458, including data inputs, data outputs, various actors, etc.

[0204] FIG. 43 shows an interactive interface presented to a user of one of the head units 402A, 402B, or 402C. A rendering of the patient's body part is shown rotated about a vertical axis at the left of the view. The rendering may be based on radiological data, which may have been collected using a CT scanner, as described with reference to FIG. 2, and may be supplemented with additional images following execution of the surgical planning module 462 in FIG. 37, or with any additional data or changes detected using the real object detection device 452. The right side of the view includes a menu with options selectable to access various guidance modules 440. The user may select a "Robot Setup" option to access the room setup module 458, an "Align Anatomy" option to execute the Align Anatomy module 460, a "Pre-Surgical Planning" or "Intra-Surgical Planning" option to execute the surgical planning module 462, or a "Perform Surgery" option to execute the perform surgical procedure module 464. All these options are available to a single user using a single head unit.

[0205] 44, the user makes a selection using the handheld controller component 404. A selection wand 538 appears in the user's view. The user can move the selection wand 538 to select one of the options. In this example, the user selects the "Align Anatomy" option. The user may, for example, tap the thumb interface of the handheld controller component 404 to make the selection.

[0206] FIG. 45 illustrates the functionality of the anatomical structure registration module 460. At 540, a digital representation of a patient's body part is stored. At 542, multiple target points are displayed to a user. The target points are superimposed on the body part to guide the user to multiple distinct, specific locations on the body part. At 544, the user locates the probe tip relative to the body part. At 546, one of the real object detection devices 452 in FIG. 37 detects a detectable surface. At 548, the location and orientation of the detectable surface are calculated. At 550, the location of the probe tip is calculated based on the location and orientation of the detectable surface. Steps 544-550 are repeated, for example, five times for five target points.

[0207] 46 illustrates the functionality of the anatomical registration module 460 in more detail. A user 552, wearing a first head unit 402A, holds a probe 554. The probe 554 has a probe tip 556 and a reference object 558. The reference object 558 has a detectable surface 560. The detectable surface 560 has an image that is detectable by the head unit sensor 446 in FIG.

[0208] Also shown is a patient's body part 562. The system presents and overlays five target points 564 on the body part 562. The user 552 is guided by the target points 564 to locate the probe tip 556 over each of the target points 564. By detecting the detectable surface 560, the system is able to calculate the location of the probe tip 556 and precisely align the five locations on the body part 562. The user is also provided with a visual output 566, which indicates to the user how many target points have been aligned.

[0209] It should be noted that although the probe tip 556 moves and the reference map 438 changes to reflect the movement of the probe tip 556, it is also possible that the body part 562 is not stationary and the reference map 438 changes in response to the movement of the body part 562. Such movements of the body part 562 will be recorded by the head unit sensor 446 of the head unit 402A primarily because they span the field of view 570 in the direction the user 552 is looking and the body part 562 is within the field of view 570.

[0210] 47 illustrates various aspects of the anatomical registration module 460, including data inputs, data outputs, various operators, etc. The functionality of the anatomical registration module 460 is described in the context of a user facilitating the registration of body parts. However, it should be understood that the registration may alternatively be performed entirely automatically, i.e., without user assistance, using various sensors, computer vision, and other mechanisms.

[0211] FIG. 48 illustrates the functionality of the surgical planning module 462. At 580, a digital representation of a patient's body part is stored. The digital representation stored at 580 may be the same digital representation stored at 540 in FIG. 45. At 582A, multiple simultaneous views are displayed on the head unit 402A. The views are different views of the digital representation of the patient's body part along with the surgical implant (virtual object) and measurements of the digital representation of the implant. At 584A, a user may provide input that is received by the surgical planning module 462 to guide the surgical implant relative to the digital representation of the body part. At 586A, the surgical implant is moved relative to the digital representation of the body part within the user's view in response to input from the user. At 588A, a user may adjust measurements by adjusting the digital representation of the surgical implant.

[0212] The user of head unit 402A may at any time perform a handoff to the user of head unit 402C at 590. The user of head unit 402C may then perform any one or more of steps 582B, 584B, 586B, and 588B. The user of head unit 402C may perform a handoff at 592, returning control to the user of head unit 402A.

[0213] 49 shows a user 552, wearing a head unit 402A, using their handheld controller component 404 to view and manipulate three different views 596A, 596B, and 596C, which represent coronal, transverse, and sagittal views of the knee, respectively.

[0214] 50a and 50b show one of the views seen by a user 552 through the head unit 402A. The individual views include a rendered view 600 of a digital representation of a body part and a rendered view 602 of a surgical implant. Because the head unit 402A is a see-through head unit, the user can also see the body part 562. Additionally, a rendered view 604 of the surgical implant is shown over the actual body part 562. The actual body part 562 is not cluttered with additional renderings of the digital representation of the body part. The user can then adjust the surgical implant, for example, by moving the rendered view 602 of the surgical implant. In this example, moving the rendered view 602 causes a change in measurement from 8.0 mm to 7.5 mm.

[0215] FIG. 51 shows a rendered view 608 of a user of head unit 402C. The user of head unit 402C is located at a remote location. User 552 can consult with the user of head unit 402C. Both users can see each other in the same session, as described with reference to FIG. 34 above, and can hear each other from the appropriate location where they are located, as described with reference to FIG. 31. The user represented by rendered view 608 may also take over control from user 552, as described with reference to FIG. 48.

[0216] FIG. 52 illustrates various aspects of the surgical planning module 462, including data inputs, data outputs, various actors, and the like.

[0217] 53 shows in further detail the functionality of the surgical operation execution module 464. At 620, a digital representation of the patient's body part is stored, as described above. At 622, real objects are detected by various real object detection devices 452. The detected real objects include the patient and the patient's body part, medical staff, one or more robots, cutting tools on the robot, implants, surgical tools, and disposable items transferred into the body part by the robot.

[0218] At 624, the system receives input from the user to guide the virtual object relative to the digital representation of the body part. At 626, the virtual object is moved relative to the digital representation of the body part in the user's view. Note that an actual digital representation of the body part does not have to be rendered for viewing by the user. At 628, in the real environment, individual ones of the real objects are moved relative to the patient's body part. At 630, the movement of all individual real objects is tracked as the real object detection device 452 detects the individual real objects and the map generator updates the map as the individual real objects move.

[0219] 54 illustrates some of the real objects tracked by the real object detection device 452, including the patient, various medical staff (anesthesiologist, operating surgeon, assistant surgeon, care nurse, direct care nurse). It should be understood that other real objects that may be tracked, although not shown in detail, may include patient body parts, robots, robotic cutting tools, surgical implants, disposable items, handheld surgical tools, etc.

[0220] 55 illustrates how a user 552 plans and positions the cutting plane of the robot's cutting blade using the handheld controller component 404. The cutting plane 634 is displayed to the user 552, and the cutting plane 634 moves as the user 552 moves the handheld controller component.

[0221] FIG. 56 displays a message that is shown to the user 552 when the cutting blade is close to the desired location.

[0222] 57 illustrates how a user 552 selects placement of an implant rendering 636 using the handheld controller component 404. The user 552 is also provided with visualization of a plane 638, an end effector 640, and a robotic arm 642 of the implant 636.

[0223] 58a, 58b, and 58c show using the finger input surface of the handheld controller component 404 to move the robotic arm so that the cutting tool or implant is incrementally moved or rotated until it matches the desired location on the body part.

[0224] FIG. 59 illustrates various aspects of the surgical operation execution module 464, including data inputs, data outputs, various actors, and the like.

[0225] From the above description, it should be clear that a digital representation of the real environment, such as represented by the reference map 438 in FIG. 37 , serves as the basis that the guidance module 440 uses to guide virtual and / or real objects. The digital representation is sometimes visible to the user and sometimes invisible. The room setup module 458 does not display a digital representation of the reference map 438, but instead displays target locations for the objects. However, these target locations are based on the digital representations in the reference map 438, and the user is guided 100% digitally. Such digital guidance provides greater accuracy and repeatability in the room setup. The anatomical structure registration module 460 does not display a digital representation of the reference map 438, but instead displays target points based on the digital representation. As the user then positions the probe tip on the body part, the reference map 438 can be updated with great accuracy, making the surgical procedure more precise. The surgical planning module 462 displays a digital representation of the body part. The digital representation of the body part may be obtained from radiology data that exists outside of the reference map 438. However, after the body part is registered using the anatomical registration module 460, the digital representation of the body part is based on the reference map 438 and the visualization of the body part is based on the reference map 438. In particular, such visualization of the body part in its precise location within the reference map 438 leads to more accurate planning of the surgical procedure. The surgical procedure execution module 464 uses the digital representation of the reference map 438 to guide the robot without necessarily displaying a visualization of the reference map 438 to the user. Because the reference map 438 forms a digital twin of the real environment, the robot can be accurately guided based on the digital representation.

[0226] Spatial computing has many applications and use cases in hospitals. Below is a sampling of representative use cases where spatial computing can have the most impact, many of which, when interconnected, represent elements of a complete platform that can drive many clinical and surgical transformations.

[0227] General - Training and Surgery

[0228] Expert Capture: On-the-fly training that is captured on-device by an expert on a task or other workflow and delivered on-device to operators and clinical staff as needed.

[0229] Remote Assistance: Involve remote experts to provide remote assistance through video, avatars, or 3D renderings. The virtual visit can also include sharing digital content in 3D, placing objects in dedicated rooms (for reuse), and customized views to exchange screen and physical information. This can also be used to support clinical tasks such as expert consultations from another facility, medical device specialists providing assistance to facilitate procedures being performed using the device, or remote technical assistance and application training. This same capability can be used to augment field service technicians with "what-you-see" knowledge back at the control center, overlaying documentation and schematics, identifying parts and components, etc.

[0230] Medical Simulation: Current medical simulation implementations require a large physical footprint and equipment. Simulations using place-based spatial computing can utilize a smaller physical footprint and minimal equipment, allowing a smaller overall space to be used for multiple scenarios. In this use case, a "blank room" can be filled with digital content, adaptive and reactive simulation scenarios can be delivered via devices, and student performance can be measured and tracked using on-board sensors. Simulations and performances can be played back to students for further understanding and scrutiny with instructors.

[0231] Physical Planning and Design: Using devices to map, plan, and visualize new structures or innovations in 3D, including operating room design with placement of devices and equipment for optimal workflow and throughput.

[0232] Tumor Board / Expert Consultation: A panel of experts from various geographic locations can gather in a virtual room as avatars or 3D renderings and view various 3D and other content. The content can be retained in the room and reviewed again later, including with the patient.

[0233] Hospital Services and Patient Engagement

[0234] Patient Consultation: A patient can have a remote consultation with a specialist (at home, in a clinic, or at a local doctor's office) prior to a surgery or other procedure being performed at the hospital. Before surgery, the consultation can include a 3D visualization of the patient's condition using radiology images, a hands-on review of the anatomy, and a discussion of the surgical approach with the surgeon. This can be done in a remote clinic connected to a major hospital.

[0235] Pre-approval: Pre-approval and patient education can be delivered on the device, including pre-surgical 3D visualization, procedure explanation, risk and benefit explanation. The consultation can be recorded and documented for future use, including patient and family education, and legal documentation.

[0236] Data Visualization: Spatial computing presents many opportunities to incorporate electronic health records and other clinical and administrative data and leverage 3D spatial visualization to better integrate data sources, i.e., new ways of understanding those data and therefore, new insights.

[0237] Integrated Operating Room ("OR") and Interventional Room

[0238] Augmented Assistant: The device can be used to provide virtual instructions and guidance, perform all clinical workflow steps, reduce physical interaction with software and hardware, and, through an improved intuitive human interface, facilitate independent troubleshooting of most problems typically solved through human-to-human interaction. This can make workflow more standardized and accurate, and can help reduce the overall amount of personnel required to be physically present in the room. The same underlying capabilities can be used for virtual surgical checklists, including full walk-throughs and documentation.

[0239] Pre-surgical planning: Traditional 2D images can be ported into 3D space for enhanced visualization and collaboration with others, allowing the same content to be viewed from the same location, eventually replacing physical monitors and screens throughout hospitals and physician offices and clinics. The first manifestation of this is the Digital Imaging and Communications in Medicine (DICOM) viewer, which can be used to capture models from 2D surgical planning workstations and move the entire experience into a Magic Leap device, with the ability for multiple people to participate in a single session, view segmentation, scroll through multiple slices, adjust the size of the image for better visualization, and more. Images can be annotated and recorded. Surgeons can plan and practice their approach and visualize potential areas of complications that may otherwise be difficult or impossible to see in 2D.

[0240] Registration, Planning, and Execution: Radiological images and models can be registered to physical anatomy for improved planning and preparation, including placement and alignment of anatomy, implants, and robots. This also includes anatomical structure identification, landmark capture guidance, incision guidance, and registration point overlay. During surgical execution, the device can provide guidance, tissue interference detection, assembly instructions, and incision selection advice.

[0241] Device Control: Along with control or other input mechanisms such as eye tracking, hand gestures, voice commands, etc., the device is used to control medical devices such as surgical robots, surgical lights, surgical beds, and other tools.

[0242] Screenless Display: Provides on-device views of scenes from various medical devices, either in the operating room, patient room, or other areas of the hospital. This allows surgeons, clinical workers, and other staff to aggregate multiple scenes into one optimized display. This reduces the overall number of monitors and allows for more ergonomic viewing of data. On-device displays can be customized based on individual user preferences or more specific use cases to provide an optimal virtual cockpit.

[0243] Live Video: Through the device, live video feeds from a variety of sources can be visualized, including stored and live ultrasound, endoscopy, and microscopy images. This video can be overlaid on top of other clinical content.

[0244] Digital Twin: A complete digital twin of an individual room or an entire hospital can be created. This will enable tracking of objects, understanding and optimizing workflows, and providing AI-driven enhancements to workers and workflows. This will serve as the basis for wider IoT implementations, which will enable equipment control and operation through devices. Other use cases include marker-less tracking of instruments and tools with a visual depiction of the tracked marker, better overall placement of devices and equipment, and improved training, instrument labeling and configuration, inventory management / asset tracking. When coupled with AI and other IoT sensors at the edge, this can provide real-time workflow optimization and corrections.

[0245] Radiation therapy

[0246] Patient Positioning: Current patient positioning tools are expanded to include overlay on top of the patient's anatomy and continuous monitoring and visualization of the patient's position.

[0247] Equipment Positioning: Accurate positioning of devices and / or tables using digital overlay and continuous monitoring and visualization.

[0248] Telemedicine, patient diagnosis, and therapy

[0249] Biomarkers: The device can be used to measure a variety of biomarkers, including eye movement, pupil size, gait, balance, eye / hand / finger coordination, and in the future, various respiratory and cardiac measurements.

[0250] Neurology and Ophthalmology: Using some of the biomarkers, clinical studies are being conducted to validate diagnostic tests related to neurophysiological conditions, including concussion, vestibular disorders, anxiety, PTSD, and other neurophysiological conditions.

[0251] Neuromuscular: Using some of the biomarkers, clinical studies are being conducted to validate diagnostic tests, monitoring protocols, and digital therapeutics related to several movement disorders such as Parkinson's disease or neurological, brain, nervous system, or neurodegenerative disorders.

[0252] Telemedicine: Leveraging biomarkers, external sensors, and avatars or 3D stereoscopic communication tools to deliver virtual, distributed health visits. Some examples include patients in remote clinics seeing specialists in urban hospitals, patients receiving primary care visits in their homes, chronic disease monitoring, or device delivery assistance for physical therapy. This can be extended to concierge medicine, medical tourism, or global specialist consultations. It is also possible to set up small, multi-purpose "blank" rooms within clinics or retail locations, which can then leverage digital content delivered via devices (coupled with external sensors) and translate them into and enable the delivery of various digital health services.

[0253] Following the completion of each surgical procedure, a time-based record of changes in the digital representation or "digital twin" of the entire surgical procedure is stored in a database or data storage device. Live data from all past surgeries, including all previous and future surgeries, is then grown into a live, dynamic database and data system that is used to provide personnel with enhancements during every case. This data is also mapped to patient outcomes and other data about the patient and the surgical procedure, which can then help identify factors that led to good outcomes during the surgical procedure (such as why a good surgeon is a good surgeon). This database is also filled with synthetic training that resembles a real-life experience and ultimately becomes a continuous, artificial intelligence (AI)-driven "guidance" that surgeons and other clinical personnel can utilize in real time throughout the case.

[0254] 60 illustrates a graphical representation of a machine within which a set of instructions may be executed, causing the machine, in the exemplary form of a computer system 900, to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine may operate as a stand-alone device or may be connected (e.g., networked) to other machines. Moreover, while only a single machine is illustrated, the term "machine" shall also be interpreted to include any collection of machines that individually or jointly execute a set (or sets) of instructions that perform any one or more of the methodologies discussed herein.

[0255] The exemplary computer system 1900 includes a processor 1902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 1904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), and a static memory 1906 (e.g., flash memory, static random access memory (SRAM), etc.), which communicate with each other via a bus 1908.

[0256] The computer system 1900 may further include a disk drive unit 1916 and a network interface device 1920 .

[0257] Disk drive unit 1916 includes a machine-readable medium 1922 on which is stored one or more sets of instructions 1924 (e.g., software) that embody one or more of any of the methodologies or functions described herein. Software may reside, completely or at least partially, within main memory 1904 and / or within processor 1902 during its execution by computer system 1900, main memory 1904, and processor 1902, which also constitute machine-readable media.

[0258] The software may also be transmitted or received over a network 1928 via the network interface device 1920 .

[0259] The computer system 1900 includes a laser driver chip 1950 that is used to drive a projector to generate laser light. The laser driver chip 1950 includes its own data storage device 1960 and its own processor 1962.

[0260] Although machine-readable medium 1922 is shown in the exemplary embodiment to be a single medium, the term "machine-readable medium" should be interpreted to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable medium" should also be interpreted to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by a machine, causing the machine to perform any one or more of the methodologies of the present invention. The term "machine-readable medium" should therefore be interpreted to include, but is not limited to, solid-state memory, optical and magnetic media, and carrier wave signals.

[0261] While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are illustrative only and are not limitations of the present invention, and that the invention is not limited to the specific construction and arrangements shown and described, as modifications may occur to those skilled in the art.

Claims

1. 1. A vision system comprising: a real object detection device positioned to detect locations of real objects in a real environment, said real objects including at least a plurality of objects constituting a layout of a room; at least one processor; a computer-readable medium coupled to the processor; a data storage device on said computer readable medium; a set of instructions stored on the computer-readable medium and executable by the processor; a map generator connected to the real object detection device, the map generator being executable to receive data of the real environment including the real objects and to create a map forming a digital representation of the real environment including the real objects; a map storage routine executable to store the map on the data storage device; a guidance module coupled to the data storage device and executable to read the map and to guide at least one of a virtual object and a real object based on the digital representation; and a room setting module executable by the processor to set up the room based on the digital representation before a surgical procedure, wherein setting up the room includes: storing a desired room layout; providing an output digitally superimposing the desired room layout onto the real environment; an induction module including: A set of instructions, including wherein the room setup module tracks discarded items during a surgical procedure and provides an output of the items for replacement.

2. The vision system of claim 1 , wherein the room setup module generates an image and overlays the image over the real environment based on the desired room layout.

3. The vision system of claim 2 , wherein the room setup module generates an image of a desired placement of one of the plurality of objects within the desired room layout and superimposes the image over the real environment.

4. The vision system of claim 3 , wherein the room setup module provides an output indicating that individual objects have been moved to positions consistent with the desired installation.

5. 5. The vision system of claim 4, wherein the room setup module tracks the movement of the individual objects as the real object detection device detects the individual objects and the map generator updates the map as the individual objects move.

6. 6. The vision system of claim 5, wherein the room setup module provides a visual output indicating that the individual object has been moved to a position consistent with the desired setup by changing a color of the image of the desired setup.

7. The vision system of claim 1 , wherein the room setup module provides an output that digitally overlays the desired exploded room layout onto the real environment after a surgical procedure.

8. The apparatus further includes a viewing device, the viewing device comprising: A head unit, the head unit comprising: A head mountable frame; a data channel for receiving image data of the image; a light guide, the light guide being a transparent light guide positioned between the eye of the user and the external surface of the real object, such that light from the external surface of the real object remains transmitted to the retina of the eye, such that the user sees the external surface of the real object augmented in the rendering of the image including at least one object in the desired room layout; A head unit comprising: The vision system of claim 1 , comprising:

9. the guidance module includes at least an anatomical registration module executable by the processor to perform anatomical registration based on the digital representation; storing a location of a body part of a patient, the location of the body part being based on a location of a real object by the real object detection device; The vision system of claim 1 , comprising:

10. 10. The vision system of claim 9, further comprising a probe, the probe being a guided real object, the probe having a probe tip and a detectable surface, wherein when a user positions the probe tip relative to the body part, the real object detection device detects the detectable surface, and the anatomical structure registration module calculates a location of the probe tip based on a location of the detectable surface.

11. The vision system of claim 10 , wherein the anatomical registration module displays target points to the user that are superimposed on the body part to guide the user to specific locations on the body part.

12. 12. The vision system of claim 11, wherein the anatomical structure registration module displays a plurality of target points to the user and guides the user to a plurality of distinct specific locations on the body part, and the anatomical structure registration module calculates a distinct location of the probe tip based on a distinct location of the detectable surface when the probe tip is at the distinct specific location.

13. The vision system of claim 10 , wherein the anatomical registration module calculates an orientation of the detectable surface and uses the orientation to calculate a location of the probe tip.

14. The apparatus further includes a viewing device, the viewing device comprising: A head unit, the head unit comprising: A head mountable frame; a data channel for receiving image data of the image; an optical waveguide, the optical waveguide being a transparent optical waveguide positioned between the user's eye and the external surface of the real object such that light from the external surface of the real object remains transmitted to the retina of the user's eye so that the user can see the external surface of the real object augmented with the rendering of the image including the rendering of the target point; A head unit comprising: The vision system of claim 11 , comprising:

15. The guidance module includes at least a surgical planning module executable by the processor to plan a surgical procedure based on the digital representation, wherein planning the surgical procedure includes: storing a digital representation of the patient's body part; displaying the digital representation of the body part of the patient together with the virtual object to a user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; moving the virtual object relative to the digital representation of the body part within a view of the user in response to the input from the user; The vision system of claim 1 , comprising:

16. The vision system of claim 15 , wherein the user is simultaneously provided with at least two views of the digital representation of the body part.

17. The vision system of claim 15 , wherein the virtual object is a digital representation of an implant that moves into the digital representation of the body.

18. A vision system, comprising: a real object detection device positioned to detect locations of real objects in a real environment, said real objects including at least a plurality of objects constituting a layout of a room; at least one processor; a computer-readable medium coupled to the processor; a data storage device on said computer readable medium; a set of instructions stored on the computer-readable medium and executable by the processor; a map generator connected to the real object detection device, the map generator being executable to receive data of the real environment including the real objects and to create a map forming a digital representation of the real environment including the real objects; a map storage routine executable to store the map on the data storage device; a guidance module coupled to the data storage device and executable to read the map and to guide at least one of a virtual object and a real object based on the digital representation; and a room setting module executable by the processor to set up the room based on the digital representation before a surgical procedure, wherein setting up the room includes: storing a desired room layout; providing an output digitally superimposing the desired room layout onto the real environment; an induction module including: A set of instructions, including the guidance module includes at least a surgical planning module executable by the processor to plan a surgical procedure based on the digital representation, and planning the surgical procedure includes: storing a digital representation of the patient's body part; displaying the digital representation of the body part of the patient together with the virtual object to a user; receiving input from the user and guiding the virtual object relative to the digital representation of the body part; moving the virtual object relative to the digital representation of the body part within a view of the user in response to the input from the user; wherein the virtual object is a digital representation of an implant that moves into the digital representation of the body; A visualization system wherein the surgical planning module displays measurements of the digital representation of the implant, the measurements being adjustable by adjusting the digital representation of the implant.

19. The apparatus further includes a viewing device, the viewing device comprising: a data channel for receiving image data of the image; a display connected to the data channel such that the display allows the user to view the image including the digital representation of the body part of the patient together with the virtual object; a user input device through which the user provides the input and guides the virtual object; and 16. The vision system of claim 15, comprising:

20. the viewing device is a first viewing device, and the user is a first user; further comprising a second viewing device, said second viewing device comprising: a data channel for receiving image data of the image; a display connected to the data channel such that the image including the digital representation of the body part of the patient together with the virtual object can be viewed by a second user; a user input device through which the second user provides the input and guides the virtual object; and 20. The vision system of claim 19, comprising:

21. The viewing system of claim 20 , wherein the surgical planning module displays the second user's rendering within the display of the first viewing device.

22. the guidance module includes at least a surgical procedure execution module executable by the processor to assist in performing a surgical procedure based on the digital representation; storing a digital representation of the patient's body part; receiving input from a user and guiding the virtual object relative to the digital representation of the body part; In response to the input from the user, moving the virtual object relative to the digital representation of the body part within a view of the user; moving, within the real environment, individual ones of the real objects relative to the body parts of the patient; The vision system of claim 1 , comprising:

23. 23. The vision system of claim 22, wherein one of the real objects detected by the real object detection device is the body part of the patient.

24. 23. The vision system of claim 22, wherein one of the real objects detected by the real object detection device is a medical staff member.

25. 23. The vision system of claim 22, wherein one of the real objects detected by the real object detection device is a robot.

26. 23. The vision system of claim 22, wherein the real object is a cutting tool that is moved into the body part of the patient.

27. 23. The vision system of claim 22, wherein the real object to be moved is an implant to be inserted into the body part of the patient.

28. 23. The vision system of claim 22, wherein one of the real objects detected by the real object detection device is a disposable item.

29. 23. The vision system of claim 22, wherein the surgical execution module tracks movement of the individual real objects as the real object detection device detects the individual real objects and the map generator updates the map as the individual real objects move.

30. A head mountable frame; a raw data receiving unit for receiving raw data; an image generation unit, connected to the data storage device, that processes raw data of the reflected waves, creates image data representing an image, and stores the image data in the data storage device; an image data receiving unit for receiving the image data from the data storage device; at least one projector connected to the image data receiving unit to receive the image data, the projector generating light in a pattern representing the image data; at least one light guide connected to the projector and secured to the head-mountable frame, the at least one light guide directing the light to a retina of the user's eye so that the user can see a rendering of the image data; and The vision system of claim 1 further comprising:

31. a map storage routine for storing a first map having a plurality of anchors, each anchor of the first map having a set of coordinates; an anchor identification system connected to the real object detection device and configured to detect anchors in a second map based on the locations of the real objects, each anchor in the second map having a set of coordinates; a location determination module coupled to the first map and the second map and operable to locate the second map relative to the first map by matching a first anchor of the second map with a first anchor of the first map and matching a second anchor of the second map with a second anchor of the first map; The vision system of claim 1 further comprising:

Images