Tracking System for Image-Guided Surgery

Abstract

Apparatus and methods are described that include tracking a portion of a tool (22) and a patient marker (38) from a first line of sight using a first tracking device (34) disposed on a first head-mounted device (28) including a display (30). The portion of the tool and the patient marker are tracked from a second line of sight using a second tracking device (34' / 60). When both the portion of the patient marker and the portion of the tool are within the first line of sight, an augmented reality image is generated on the first display based on data received from the first tracking device and without using data from the second tracking device. When at least the portion of the patient marker and the portion of the tool are not both within the first line of sight, a virtual image of the tool and the patient's anatomical structure is generated using data received from the second tracking device. Other embodiments are also described.

Description

Technical Field

[0001] The present invention generally relates to augmented reality systems, and more particularly to a tracking system for an augmented reality system used to perform image-guided surgery.

Background Art

[0002] A head-mounted display may be used as part of an augmented reality system. The display is used to generate an augmented reality scene in which the scene seen by the user of the head-mounted display is modified by being typically augmented or supplemented. The modification is generated by a computer and typically includes presenting real-time video and / or non-real-time images to the user while the user is looking at the scene.

[0003] In some cases, an augmented reality system is used to perform image-guided surgery as part of a medical procedure. For example, computer-generated images may be presented to a medical professional performing the procedure. The images are presented on a head-mounted display and the images may be aligned with the anatomical part of the patient undergoing the procedure. Some misalignment between the image and the patient's body may be acceptable, but for a satisfactory presentation of the image, the misalignment typically should not exceed about 2-3 mm. To account for such limitations on misalignment between the presented image and the patient's anatomical structure, the position of the patient's body or a part thereof is typically tracked.

[0004] In some cases, an image of a tool used to perform the procedure is incorporated into the image displayed on the head-mounted display. To incorporate the image of the tool into the image, the position of the tool or a part thereof is typically tracked so that the position of the tool relative to the image and / or the patient's anatomical structure is accurately reflected.

[0005] Triangulation techniques are commonly used to track the position of a patient's body or tools. In such techniques, multiple imaging devices arranged in known positions relative to each other are used to detect features (such as markers) on the patient's body and / or tools. Next, the arrangement of the features is derived using a combination of the known arrangement of the imaging devices and the arrangement of the features detected by each imaging device.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

[0007] According to some embodiments of the present invention, a first medical professional (e.g., a surgeon performing a procedure) wears a first head-mounted device. Typically, the head-mounted device includes one or more head-mounted displays. In some embodiments, the head-mounted display is generally similar to that described in U.S. Patent No. 9,928,629 to Benishty (Patent Document 1), which is hereby incorporated by reference. For example, the head-mounted display may include a combiner controlled by a computer processor for, among other things, displaying augmented reality images to the medical professional. In some embodiments, the image generated by the computer is projected onto a first portion of the display, and the image is displayed on the head-mounted display such that the second portion of the display allows the anatomical portion of the patient undergoing the procedure to be visible and the image generated by the computer is aligned with the anatomical portion of the patient. Typically, the image generated by the computer includes a virtual image of a tool superimposed on a virtual image of the patient's anatomical structure. In some embodiments, a portion of the tool that is not visible to the medical professional (e.g., because it is hidden by the patient's anatomical structure) is included in the image generated by the computer.

[0008] Typically, the head-mounted device includes a tracking device configured to facilitate determination of the placement and orientation of the head-mounted device relative to a part of the patient's body (e.g., the patient's back), and / or the position and orientation of the tool relative to the patient's body. For example, the tracking device may include an image acquisition device such as a camera configured to image patient markers and / or tool markers. Typically, the patient markers are configured such that a computer processor can use the data collected from a single tracking device disposed on the head-mounted display to provide sufficient data to determine the placement and orientation of the head-mounted device relative to a part of the patient's body. For example, the patient markers may include an array of elements visible from the tracking device of the head-mounted device, and it is configured such that in any placement and orientation of the head-mounted device relative to the patient markers, the array of elements has a unique appearance in terms of its location and orientation. In this way, the computer processor can determine the placement and orientation of the head-mounted device relative to a part of the patient's body without using triangulation techniques. Usually, a single camera is used in the tracking device of the head-mounted device. In some embodiments, the camera is a high-speed camera. For example, the camera can acquire more than 50 frames per second.

[0009] Typically, to generate an augmented reality image on a head-mounted display, a computer processor determines the placement and orientation of the head-mounted device relative to a part of the patient's body (e.g., the patient's back) and / or the position and orientation of a tool relative to the part of the patient's body. As described above, generally, patient markers are configured to provide sufficient data for a computer processor to determine the placement and orientation of the head-mounted device relative to a part of the patient's body using data collected from a single tracking device disposed on the head-mounted device. However, in some embodiments, at least under certain conditions, the computer processor is configured to incorporate tracking data received from at least one additional tracking device (i.e., a tracking device added to the tracking device included in the head-mounted device of the first medical professional) to generate an image on the head-mounted display of the first medical professional.

[0010] In such an embodiment, the computer processor is configured to incorporate additional data when the first tracking device included in the head-mounted device of the first medical professional loses line of sight with the patient marker and / or the tool marker and / or a portion thereof. For example, the computer processor may be configured to receive data from a tracking device of an additional head-mounted device configured to be worn by an additional medical professional (e.g., a co-surgeon or a nurse) present during the procedure. Typically, the additional head-mounted device is generally similar to the first head-mounted device, and the tracking device of the additional head-mounted device is generally similar to the tracking device of the first head-mounted device. In some embodiments, when at least a portion of the patient marker and a portion of the tool (e.g., the tool marker) are both within the line of sight of the first tracking device, the computer processor generates an augmented reality image on the head-mounted display based on the data received from the first tracking device and without using the data received from the additional tracking device. When at least a portion of the patient marker and a portion of the tool are not both within the line of sight of the first tracking device, the computer processor generates an augmented reality image on the first head-mounted display based at least in part on the data received from the additional tracking device.

[0011] According to some embodiments of the present invention, a method for use with a tool configured to be placed within a part of a patient's body, the method comprising: using a first tracking device disposed on a first head-mounted device worn by a first person to track a part of the tool and patient markers placed on the patient's body from a first line of sight, the first head-mounted device comprising a first head-mounted display; using a second tracking device to track a part of the tool and patient markers placed on the patient's body from a second line of sight; using at least one computer processor to: generate an augmented reality image on the first head-mounted display based on data received from the first tracking device and without using data from the second tracking device when at least a part of the patient markers and a part of the tool are both within the first line of sight, the augmented reality image having (a) a virtual image of the tool and (b) an anatomical structure of the patient superimposed on the patient's body; and generate a virtual image of the tool and the anatomical structure of the patient on the first head-mounted display based at least in part on data received from the second tracking device when at least a part of the patient markers and a part of the tool are not both within the first line of sight. A method is provided, characterized by having the above steps.

[0012] In some embodiments, the step of tracking a part of the tool comprises the step of tracking a tool marker. In some embodiments, the step of using a second tracking device to track a part of the tool and patient markers placed on the patient's body from a second line of sight comprises the step of using a second tracking device disposed in a stationary position to track at least a part of the tool and patient markers. In some embodiments, the step of using a first tracking device to track at least a part of the tool and patient markers comprises the step of using a first camera to track at least a part of the tool and patient markers, and the step of using a second tracking device to track at least a part of the tool and patient markers comprises the step of using a second camera to track at least a part of the tool and patient markers.

[0013] In some embodiments, the step of generating a virtual image of the tool and the patient's anatomical structure on the first head-mounted display, at least in part based on data received from the second tracking device, comprises, in response to a part of the patient marker being within the first line of sight and a part of the tool not being within the first line of sight: determining the placement of the tool relative to the subject's anatomical structure using data received from the second tracking device; generating a virtual image of the tool and the patient's anatomical structure on the first head-mounted display based on the determined placement of the tool relative to the subject's anatomical structure; determining the placement of the patient's body relative to the first head-mounted device based on data received from the first tracking device; and superimposing the virtual image on the patient's body based on the determined placement of the patient's body relative to the first head-mounted device.

[0014] In some embodiments, the step of generating a virtual image of the tool and the patient's anatomical structure on the first head-mounted display, at least in part based on data received from the second tracking device, comprises, in response to a part of the tool being within the first line of sight and a part of the patient marker not being within the first line of sight: determining the placement of the tool relative to the subject's anatomical structure using data received from the second tracking device; and generating a virtual image of the tool and the patient's anatomical structure on the first head-mounted display based on the determined placement of the tool relative to the subject's anatomical structure.

[0015] In some embodiments, the step of generating a virtual image of the tool and the patient's anatomical structure on the first head-mounted display further comprises superimposing the virtual image on the patient's body based on the position of the patient's body relative to the first head-mounted device, determined based on data received from the first tracking device when a part of the patient marker was within the first line of sight.

[0016] In some embodiments, the step of overlaying a virtual image on a patient's body includes tracking the movement of the head-mounted device at times when a portion of the patient marker is within the first line of sight and at times when a portion of the patient marker is not within the first line of sight, using an inertial measurement unit disposed on the first head-mounted device.

[0017] In some embodiments, the step of generating virtual images of the tool and the patient's anatomical structure on the first head-mounted display based at least in part on data received from the second tracking device includes, in response to a portion of the tool and a portion of the patient marker not both being within the first line of sight: determining the position of the tool relative to the subject's anatomical structure using data received from the second tracking device; and generating virtual images of the tool and the patient's anatomical structure on the first head-mounted display based on the determined position of the tool relative to the subject's anatomical structure.

[0018] In some embodiments, the step of generating virtual images of the tool and the patient's anatomical structure on the first head-mounted display further includes overlaying a virtual image on the patient's body based on the position of the patient's body relative to the first head-mounted device, determined based on data received from the first tracking device when a portion of the patient marker was within the first line of sight. In some embodiments, the step of overlaying a virtual image on the patient's body includes tracking the movement of the head-mounted device at times when a portion of the patient marker is within the first line of sight and at times when a portion of the patient marker is not within the first line of sight, using an inertial measurement unit disposed on the first head-mounted device.

[0019] In some embodiments, the step of using a second tracking device to track at least a portion of the tool and the patient marker from a second line of sight comprises using a second tracking device disposed on a second head-mounted device worn by a second person to track at least a portion of the tool and the patient marker from the second line of sight. In some embodiments, the second head-mounted device includes a second head-mounted display, and the method further comprises generating an additional augmented reality image on the second head-mounted display.

[0020] According to some embodiments of the present invention, an apparatus for use with a tool configured to be disposed within a portion of a patient's body, the apparatus comprising: a patient marker configured to be disposed on the patient's body; a first head-mounted device comprising a first head-mounted display and a first tracking device configured to track at least a portion of the tool and the patient marker from a first line of sight; a second tracking device configured to track at least a portion of the tool and the patient marker from a second line of sight; and at least one computer processor configured to: generate an augmented reality image on the first head-mounted display based on data received from the first tracking device and without using data from the second tracking device when at least a portion of the patient marker and a portion of the tool are both within the first line of sight, the augmented reality image having (a) a virtual image of the tool and (b) an anatomical structure of the patient superimposed on the patient's body; and generate a virtual image of the tool and the anatomical structure of the patient on the first head-mounted display based at least in part on data received from the second tracking device when at least a portion of the patient marker and a portion of the tool are not both within the first line of sight.

[0021] According to some embodiments of the present invention, a method for use with a tool configured to be disposed within a part of a patient's body, comprising: using a first tracking device disposed on a first head-mounted device worn by a first person to track a part of the tool and patient markers placed on the patient's body from a first line of sight, wherein the first head-mounted device comprises a first head-mounted display; using a second tracking device disposed on a second head-mounted device worn by a second person to track a part of the tool and patient markers placed on the patient's body from a second line of sight; and using at least one computer processor to generate an augmented reality image on the first head-mounted display based on data received from the first tracking device combined with data received from the second tracking device, the augmented reality image having (a) a virtual image of the tool and the patient's anatomical structure superimposed on (b) the patient's body. A method is further provided, characterized by having the above steps.

[0022] In some embodiments, the step of tracking a part of the tool comprises the step of tracking a tool marker. In some embodiments, the second head-mounted device includes a second head-mounted display, and the method further comprises the step of generating a further augmented reality image on the second head-mounted display. In some embodiments, the step of tracking at least a part of the tool and patient markers using the first tracking device comprises the step of tracking at least a part of the tool and patient markers using a first camera, and the step of tracking at least a part of the tool and patient markers using the second tracking device comprises the step of tracking at least a part of the tool and patient markers using a second camera.

[0023] In some embodiments, the step of generating an augmented reality image on the first head-mounted display includes: determining the placement of the tool relative to the subject's anatomical structure using data received from the second tracking device; generating a virtual image of the tool and the patient's anatomical structure on the first head-mounted display based on the determined placement of the tool relative to the subject's anatomical structure; determining the placement of the patient's body relative to the first head-mounted device based on data received from the first tracking device; and overlaying the virtual image on the patient's body based on the determined placement of the patient's body relative to the first head-mounted device.

[0024] According to some embodiments of the present invention, an apparatus for use with a tool configured to be disposed within a portion of a patient's body, the apparatus comprising: a patient marker configured to be disposed on the patient's body; a first head-mounted device configured to be worn by a first person, the first head-mounted device comprising a first head-mounted display and a first tracking device configured to track at least a portion of the tool and the patient marker from the first line of sight; a second head-mounted device configured to be worn by a second person, the second head-mounted device comprising a second head-mounted display and a second tracking device configured to track at least a portion of the tool and the patient marker from the second line of sight; and at least one computer processor configured to: generate an augmented reality image on the first head-mounted display based on data received from the first tracking device combined with data received from the second tracking device, the augmented reality image having (a) a virtual image of the tool and the patient's anatomical structure and (b) the virtual image overlaid on the patient's body. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be more fully understood from the following detailed description of embodiments with reference to the accompanying drawings:

Figure 1

Figure 2

Figures 3A-3B

Figures 4A-4B

Figures 5A-5B

Figures 6A-6B

[0026] Referring now to FIG. 1, which is a schematic diagram of a medical procedure incorporating an image-guided surgery performed on patient 20 according to some embodiments of the present invention. In the medical procedure shown in FIG. 1, tool 22 is used to perform an operation on the patient's back, and the tool is inserted through incision 24 in the patient's back 27. However, the devices and techniques described herein can be used, with necessary modifications, in any surgical procedure performed on a patient's body. Reference is also made to FIG. 2, which is a schematic diagram of a head-mounted device 28 according to some embodiments of the present invention.

[0027] In some embodiments, a first medical professional 26 (e.g., a surgeon performing a procedure) wears a first head-mounted device 28. Typically, the head-mounted device includes one or more head-mounted displays 30. In some embodiments, the head-mounted display is generally similar to that described in Benischti's U.S. Patent No. 9,928,629 (Patent Document 1), which is incorporated herein by reference. For example, the head-mounted display may include a combiner controlled by a computer processor (e.g., computer processor 32 and / or computer processor 45 described below) to display, for example, an augmented reality image to the medical professional. In some embodiments, the image is presented on the head-mounted display 30, where (a) the computer-generated image is projected by the projector 31 onto a first portion 33 of the display, and (b) the computer-generated image is aligned with an anatomical portion of the patient undergoing the procedure, and the anatomical portion of the patient can be viewed through a second portion 35 of the display. Typically, the computer-generated image includes a virtual image of a tool superimposed on a virtual image of the patient's anatomical structure. In some embodiments, a portion of the tool that is not visible to the medical professional (e.g., because it is hidden by the patient's anatomical structure) is included in the computer-generated image.

[0028] Some misalignment between the image and the patient's body is acceptable, but for a satisfactory presentation of the image, the misalignment usually needs to be no more than about 2-3 mm. To take such a limitation into account for the misalignment between the presented image and the patient's anatomical structure, the position of the patient's body or a part thereof relative to the head-mounted device is typically tracked. In some cases, an image of a tool used to perform a procedure is incorporated into the image displayed on the head-mounted display. To incorporate an image of the tool into the image, the position of the tool or a part thereof (e.g., a tool marker) is typically tracked so that the position of the tool relative to the patient's anatomical structure is accurately reflected. Typically, it is desirable to determine the placement of the tool relative to the patient's body such that the error in the determined placement of the tool relative to the patient's body is less than 2 mm.

[0029] Typically, the head-mounted device 28 includes a tracking device 34 that is configured to facilitate determination of the placement and orientation of the head-mounted device 28 relative to a part of the patient's body (e.g., the patient's back) and / or relative to the tool 22, and / or the position and orientation of the tool relative to a part of the patient's body. For example, the tracking device can include an image acquisition device 36, such as a camera, configured to image a patient marker 38 and / or a tool marker 40. Typically, a single camera is used for the tracking device of the head-mounted device. In some embodiments, the camera is a high-speed camera. For example, the camera can acquire more than 50 frames per second.

[0030] In some embodiments, the tracking device 34 includes a light source 42 attached to the head-mounted device. The light source is typically configured to illuminate the patient marker and / or the tool marker such that light is reflected from the marker towards the camera. In some embodiments, the image acquisition device 36 is a monochrome camera that includes a filter configured to pass only light of a wavelength similar to the light generated by the light source. For example, the light source can be an infrared light source (e.g., a light source that generates light at a wavelength between 700 nm and 1000 nm (e.g., between 700 nm and 800 nm)), and the camera can include a corresponding infrared filter. In some embodiments, an inertial measurement unit 44 (e.g., an inertial measurement unit configured to measure in six degrees of freedom) is disposed on the head-mounted device, as will be described in more detail below. In some embodiments, the head-mounted device includes an additional camera 43 configured to acquire an image of a scene in the visible spectrum, as described in U.S. Patent No. 9,928,629 to Venisti, which is incorporated herein by reference (Patent Document 1). In some embodiments, the head-mounted device 28 includes additional components, such as those described in U.S. Patent No. 9,928,629 to Venisti, which is incorporated herein by reference.

[0031] Typically, to generate an augmented reality image on the display 30, a computer processor determines the placement and orientation of the head-mounted device 28 relative to a part of the patient's body (e.g., the patient's back), and / or the position and orientation of the tool relative to the patient's body. For example, a computer processor 45 integrated within the head-mounted device can perform the aforementioned functions. Alternatively or additionally, these functions can be performed using a computer processor 32 that is disposed external to the head-mounted device and typically communicates wirelessly with the head-mounted device. The computer processor 32 typically constitutes part of a processing system 50 that is used with the head-mounted device to facilitate image-guided surgery. In some embodiments, the processing system further includes an output device 52 (e.g., a display such as a monitor) for outputting information to an operator of the system, and / or an input device 54 (a pointing device, keyboard, mouse, etc.) configured to allow the operator to input data into the system. Generally, in the context of this specification, when a computer processor is described as performing a particular step, these steps can be performed by the external computer processor 32 and / or the computer processor 45 integrated within the head-mounted device.

[0032] In some embodiments, the patient marker and / or the tool marker includes a reflective element configured to reflect light generated by the light source 42. In some such embodiments, by directing the light from the light source 42 towards the region of interest where the patient marker is located, the placement and orientation of a part of the subject's body with respect to the head-mounted device (e.g., the subject's back) is tracked. Alternatively or additionally, the placement and orientation of the tool with respect to a part of the subject's body is tracked by directing the light from the light source 42 towards the region of interest where the patient marker and / or the tool marker is located. Typically, the image acquisition device 36 is disposed on the head-mounted device in proximity to the light source, such that the image acquisition device is configured to capture the light retro-reflected from the patient marker and / or the tool marker. As described above, in some embodiments, the image acquisition device is a monochrome camera that includes a filter configured to pass only light of a wavelength similar to the light generated by the light source. In such embodiments, the camera typically receives a grayscale image showing the reflective elements of the tool marker and / or the patient marker. Typically, a computer processor determines the placement of a part of the subject's body (e.g., the subject's back) with respect to the head-mounted device by analyzing the image acquired by the image acquisition device. More typically, a computer processor determines the placement and orientation of the tool with respect to a part of the subject's body by analyzing the image acquired by the image acquisition device.

[0033] The above-described techniques for tracking the patient marker and / or the tool marker are presented by way of example, and it should be noted that in some embodiments, alternative techniques may be used to track the patient marker and / or the tool marker. For example, the patient marker and / or the tool marker may include a light-absorbing element and / or a light-generating element, and the image acquisition device may be configured to track the patient marker and / or the tool marker by detecting these elements. Alternatively or additionally, different types of detectors may be used to track the patient marker and / or the tool marker.

[0034] Typically, the patient marker is configured to provide sufficient data for a computer processor to determine the placement and orientation of the head-mounted device relative to a part of the patient's body using data collected from a single tracking device located on the head-mounted display. For example, the patient marker can include an array of elements visible to the tracking device of the head-mounted device, and it is configured such that in any placement and orientation of the head-mounted device relative to the patient marker, the array of elements has an appearance unique to that placement and orientation. In this way, the computer processor can determine the placement and orientation of the head-mounted device relative to a part of the patient's body without using triangulation techniques.

[0035] As described in the above paragraph, generally, the patient marker is configured to provide sufficient data for a computer processor to determine the placement and orientation of the head-mounted device relative to a part of the patient's body using data collected from a single tracking device located on the head-mounted display. However, in some embodiments, at least under certain circumstances, the computer processor is configured to incorporate tracking data received from additional tracking devices (i.e., tracking devices additional to the first tracking device 34) to generate an image on the head-mounted display 30 of the first head-mounted device 28 of the first medical professional 26.

[0036] In some such embodiments, when the tracking device 34 loses line of sight with the patient marker and / or the tool marker and / or a portion thereof, the computer processor is configured to incorporate additional data. This example is shown in FIG. 1, which shows that the right hand of the first medical professional 26 is blocking the line of sight of the tracking device 34 with respect to the patient marker 38. In some embodiments, in such a case the computer processor is configured to receive data from a tracking device 34' of an additional head-mounted device 28' worn by an additional medical professional 26' (e.g., a surgeon or nurse accompanying the procedure) present at the procedure, such as shown in FIG. 1. Typically, the additional head-mounted device 28' is generally similar to the first head-mounted device 28, and the tracking device 34' of the additional head-mounted device is generally similar to that of the first head-mounted device. In some embodiments, when at least a portion of the patient marker and a portion of the tool (e.g., the tool marker) are both within the line of sight of the first tracking device 34, the computer processor generates an augmented reality image on the head-mounted display 30 based on the data received from the first tracking device 34 and without using the data received from the tracking device 34'. When at least a portion of at least the patient marker and a portion of the tool are not necessarily within the line of sight of the first tracking device 34, the computer processor generates an augmented reality image on the head-mounted display 30 based at least in part on the data received from the second tracking device 34'.

[0037] Alternatively or additionally, a tracking device 60 not mounted on a head-mounted device is disposed in the operating room. Typically, the tracking device 60 is disposed at a stationary position within the operating room. For example, the tracking device 60 can be mounted on the ceiling, mounted on the wall, and / or disposed on a stand such as a tripod. In some embodiments, the tracking device 60 includes a light source 62 and an image acquisition device 64, which function in substantially the same manner as those described above with respect to the light source 42 and the image acquisition device 36.

[0038] Now, refer to FIGS. 3A and 3B. FIGS. 3A and 3B are schematic diagrams of examples of displays 30', 30 of head-mounted devices 28', 28 worn by respective medical professionals 26', 26 according to some embodiments of the present invention. FIG. 3A shows an example of the display 30' of the second medical professional 26' shown on the right side of the patient in FIG. 1, and FIG. 3B shows an example of the display 30 of the first medical professional 26 shown on the left side of the patient in FIG. 1. Usually, the images generated by each of the head-mounted display 30 and the head-mounted display 30' are augmented reality views showing the anatomical structure of the virtual patient aligned with the actual patient anatomical structure and the virtual tool aligned with the virtual anatomical structure. As described above, in some embodiments, the virtual tool and the virtual anatomical structure are displayed on the first portion 33 of the head-mounted displays 30, 30', and the anatomical structure of the actual patient can be viewed through the second portion 35 of the head-mounted displays 30, 30'. In some embodiments, the computer processor is configured to generate such views in both 2D and 3D. To generate such views, it is usually necessary to track the placement and orientation of the head-mounted device with respect to the patient in order to correctly align the virtual anatomical structure with the actual patient anatomical structure. FIGS. 3A and 3B show how each head-mounted display normally looks when the tracking device 34 of each medical professional has a clear line of sight to the patient marker.

[0039] Now, refer to FIGS. 4A and 4B. FIGS. 4A and 4B are schematic diagrams of examples of displays 30', 30 of head-mounted devices 28', 28 worn by respective medical professionals 26', 26 when the line of sight of the tracking device 34 of the first medical professional 26 is at least partially blocked with respect to the patient marker 38 according to some embodiments of the present invention. FIG. 4A shows an example of the display 30' of the second medical professional 26' shown on the right side of the patient in FIG. 1, and FIG. 4B shows an example of the display 30 of the first medical professional 26 shown on the left side of the patient in FIG. 1.

[0040] In some such embodiments, the computer processor generates a virtual image on the head-mounted display 30 of the first medical professional 26 that shows a virtual view of the second medical professional 26' (i.e., a view of the second medical professional's virtual anatomical structures and virtual tools) determined based on data received from the second tracking device 34'. For example, the entire view of the second medical professional (including both the view of the virtual anatomical structures and virtual tools and the view of the actual patient's anatomical structures) can be displayed on the head-mounted display 30 of the first medical professional. Such an example is shown in FIGS. 4A and 4B, which show the head-mounted display 30 of the first medical professional 26 displaying the same overall view as that of the second medical professional 26'.

[0041] Now, refer to FIGS. 5A and 5B. FIGS. 5A and 5B are schematic diagrams of examples of the displays 30', 30 of the head-mounted devices 28', 28 worn by the respective medical professionals 26', 26 when the line of sight of the tracking device 34 of the first medical professional 26 is at least partially blocked with respect to the patient marker 38, according to some embodiments of the present invention. FIG. 5A shows an example of the display 30' of the second medical professional 26' shown on the right side of the patient in FIG. 1, and FIG. 5B shows an example of the display 30 of the first medical professional 26 shown on the left side of the patient in FIG. 1. In some embodiments, when the line of sight of the tracking device 34 of the first medical professional 26 is at least partially blocked with respect to the patient marker 38, a virtual image (of the tool and anatomical structure) from the line of sight of the second medical professional is displayed, which substantially fills the entire display 30 attached to the first medical professional's head, and the first medical professional does not see the actual patient's anatomical structure through the transparent portion of the display. An example of such an embodiment is shown in FIGS. 5A and 5B, showing a virtual image from the head-mounted display 30' (shown in FIG. 5A) displayed within the portion 33 of the first medical professional's head-mounted display 30, and the portion 33 fills substantially the entire first medical professional's head-mounted display 30 (shown in FIG. 5B).

[0042] Here, refer to FIGS. 6A and 6B. FIGS. 6A and 6B are schematic diagrams of examples of displays 30', 30 of head-mounted devices 28', 28 worn by respective medical professionals 26', 26 when the line of sight of the tracking device 34 of the first medical professional 26 is at least partially blocked with respect to the patient marker 38. FIG. 6A shows an example of the display 30' of the second medical professional 26' shown on the right side of the patient in FIG. 1, and FIG. 6B shows an example of the display 30 of the first medical professional 26 shown on the left side of the patient in FIG. 1. In some embodiments, in response to detecting that the tracking device 34 has lost sight of the patient marker, and as a result, the tracking device 34 cannot be used to determine the placement and / or orientation of the head-mounted device with respect to the patient with a given level of accuracy, the computer processor generates an image of the virtual tool within the virtual anatomical structure of the subject, without considering aligning the computer-generated image with the anatomical structure of the actual patient. In some such embodiments, the virtual image generated in portion 33 of the display 30 of the first medical professional continues to be displayed from the first medical professional's previous known line of sight, but the position of the tool with respect to the anatomical structure is updated based on the data received from the tracking device 34'. The second portion 35 of the display of the first medical professional is kept transparent so that the first medical professional can see the anatomical structure of the patient from his or her current line of sight. Examples of such embodiments are shown in FIGS. 6A and 6B. As shown in FIG. 6B, since the change in the placement of the head-mounted device 28 with respect to the patient marker is not tracked and not considered, this can cause the virtual image (shown in portion 33) to be slightly misaligned with respect to the patient's body (shown in portion 35). In this regard, it should generally be noted that the first medical professional uses the virtual image of the tool superimposed on the virtual image of the patient's anatomical structure for tool navigation. Therefore, the medical professional can usually continue to navigate the tool even if the virtual images of the tool and the patient's anatomical structure do not match the medical professional's own view of the patient's anatomical structure.

[0043] In some embodiments, a method generally similar to that described in the above paragraphs is performed, but instead of, or in addition to, additional tracking data used to generate an image on the head-mounted display 30 being received from the tracking device 34' of the second head-mounted device 28', it is received from the tracking device 60.

[0044] In some embodiments, in response to the tracking device 34 losing sight of the tool marker and detecting that it cannot determine the placement and / or orientation of the tool relative to the patient with a given level of accuracy, the computer processor uses the data received from the tracking device 34' and / or the tracking device 60 to determine the placement of the tool relative to the patient. Typically, a virtual image including the anatomical structure of the virtual patient and the virtual tool shown in the current placement is displayed on the head-mounted display 30' of the head-mounted device 28, and the current placement of the tool relative to the patient has been determined based on the data received from the tracking device 34' and / or the tracking device 60.

[0045] In some embodiments, the computer processor is configured to incorporate tracking data received from an additional tracking device (i.e., a tracking device added to the tracking device 34) to generate an image on the head-mounted display 30 of the first head-mounted device 28 even when the patient marker and the tool marker are within the line of sight of the tracking device 34. In some embodiments, the computer processor uses a combination of the data received from the tracking device 34' and the data received from the tracking device 34, and / or a combination of the data received from the tracking device 60 and the data received from the tracking device 34, to determine the placement of the tool relative to the patient. For example, the computer processor can use the aforementioned combination of the received data to determine the current average (e.g., mean value) placement of the tool relative to the patient, and the computer processor can generate an image of the virtual tool on the virtual anatomical structure on the head-mounted display 30, where the tool is positioned at the determined current position.

[0046] In some embodiments, even if some of the tracking elements on the patient marker become unclear and are not within the line of sight of the tracking device 34, the computer processor uses a tracking algorithm (e.g., using a Kalman filter) to track the marker (e.g., by tracking the marker) to continue tracking the placement of the head-mounted device relative to the patient. Typically, in such cases, the computer processor does not actively identify the marker while at least some of the patient markers are partially hidden. Rather, the computer processor uses the aforementioned tracking algorithm to continue tracking the already identified marker. In some embodiments, when the patient marker becomes unclear such that at least some of the patient marker is not within the line of sight of the tracking device 34 (e.g., when it becomes partially unclear or completely unclear), the computer processor uses the inertial measurement unit 44 in combination with the last position of the patient marker determined using data from the tracking device 34 to continue to determine the placement of the patient relative to the head-mounted device.

[0047] Embodiments of the invention described herein can take the form of a computer program product accessible from a computer-usable or computer-readable medium (e.g., a non-transitory computer-readable medium) that provides program code used by or connected to a computer or any instruction execution system such as computer processors 32 and / or 45. For the purposes of the description herein, a computer-usable or computer-readable medium can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Typically, a computer-usable or computer-readable medium is a non-transitory computer-usable or computer-readable medium.

[0048] Examples of computer-readable media include semiconductor or solid state memories, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks. Current examples of optical disks include compact disk read-only memory (CD-ROM), compact disk read / write (CD-R / W), and DVD.

[0049] A data processing system suitable for storing and / or executing program code includes at least one processor (e.g., computer processors 32 and / or 45) directly or indirectly coupled to memory elements through a system bus. The memory elements can include local memory used during actual execution of the program code, bulk storage, and cache memory that provides at least some temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution. The system can read instructions of the present invention on a program storage device and perform methodologies of embodiments of the present invention in accordance with these instructions.

[0050] A network adapter can be coupled to the processor to couple the processor to other processors, remote printers, or storage devices through intervening private or public networks. Modems, cable modems, Ethernet cards are but a few of the currently available network adapters.

[0051] Computer program code for carrying out operations of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as the C programming language or similar programming languages.

[0052] It is understood that the algorithms described herein can be executed by computer program instructions. These computer program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, thereby creating means for performing the functions / acts specified by the algorithms described herein through the processor of the computer or other programmable data processing apparatus (e.g., computer processors 32 and / or 45). These computer program instructions can also be stored in a computer-readable medium (e.g., a non-transitory computer-readable medium) that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a manufacture including instruction means for performing the functions / operations specified by the algorithms. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-executed process, whereby the instructions executed on the computer or other programmable apparatus provide a process for performing the functions / acts specified by the algorithms described in this application.

[0053] The computer processor 32 and / or the computer processor 45 are typically hardware devices programmed with computer program instructions to manufacture a special-purpose computer. For example, when programmed to execute the algorithms described with reference to the figures, the computer processors 32 and / or 45 typically function as special-purpose image generation computer processors. Typically, the operations described herein that are performed by the computer processors 32 and / or 45 transform the physical state of a memory, which is a physical article, to have different magnetic polarities, electrical charges, etc. depending on the technology of the memory used. In some embodiments, operations described as being performed by one computer processor are performed in combination with each other by a plurality of computer processors.

[0054] It will be understood by those skilled in the art that the present invention is not limited to what is particularly shown and described herein. Rather, the scope of the present invention includes both the various combinations and sub-combinations of the above features, as well as variations and modifications thereof that are not found in the prior art that would occur to those skilled in the art upon reading the foregoing description.

Claims

Claim 1 An apparatus for use with a tool configured to be placed inside a part of a patient's body, comprising: A first head-mounted device comprising a first head-mounted display and a first tracking device configured to track at least a part of the tool and a patient marker from a first line of sight, wherein the patient marker is configured to be placed on the patient's body; A second tracking device configured to track at least the part of the tool and the patient marker from a second line of sight; and At least one computer processor; Having, The computer processor is configured to: When at least a part of the patient marker and the part of the tool are both within the first line of sight, generate an augmented reality image on the first head-mounted display based on data received from the first tracking device and without using data from the second tracking device, wherein the augmented reality image has a virtual image (a) of the tool and the anatomical structure of the patient superimposed on the patient's body (b); When at least one of at least the part of the patient marker and the part of the tool is within the first line of sight and the other is not within the first line of sight, generate a virtual image of the tool and the anatomical structure of the patient on the first head-mounted display based at least in part on data received from the second tracking device; An apparatus configured to perform the above, characterized in that. Claim 2 The tool has a tool marker, and the first and second tracking devices are configured to track the part of the tool by tracking the tool marker. The apparatus according to claim 1, characterized in that. Claim 3 The apparatus according to claim 1, characterized in that the first tracking device has a first camera and the second tracking device has a second camera. Claim 4 The apparatus according to claim 1, characterized in that the second tracking device has one tracking device arranged in a stationary position. Claim 5 The first head-mounted device is configured to be worn by a first person, and the device is configured to be worn by a second person and further includes a second head-mounted device having a second head-mounted display. The second tracking device is disposed on the second head-mounted device. The device according to claim 1, wherein:

6. The device according to claim 1, further comprising the patient marker.

7. The at least one computer processor is configured to generate the virtual images of the tool and the anatomical structure of the patient on the first head-mounted display based at least in part on data received from the second tracking device by the following steps. The device according to any one of claims 1-6, wherein: In response to a part of the patient marker being within the first line of sight and a part of the tool not being within the first line of sight: Determining the position of the tool relative to the anatomical structure of the patient using data received from the second tracking device; Generating the virtual images of the tool and the anatomical structure of the patient on the first head-mounted display based on the determined position of the tool relative to the anatomical structure of the patient; Determining the position of the patient's body relative to the first head-mounted device based on data received from the first tracking device; and Overlaying the virtual images on the patient's body based on the determined position of the patient's body relative to the first head-mounted device.

8. The at least one computer processor is configured to generate the virtual images of the tool and the anatomical structure of the patient on the first head-mounted display based at least in part on data received from the second tracking device by the following steps. The device according to any one of claims 1-6, wherein: In response to a part of the patient marker not being within the first line of sight: Determining the position of the tool relative to the anatomical structure of the patient using data received from the second tracking device; and Generating, on the first head-mounted display, a virtual image of the tool and the anatomical structure of the patient based on the determined position of the tool relative to the anatomical structure of the patient.

9. The at least one computer processor is configured to generate, on the first head-mounted display, the virtual image of the tool and the anatomical structure of the patient by overlaying the virtual image on the body of the patient based on the position of the patient's body relative to the first head-mounted device, which is determined based on data received from the first tracking device when a portion of the patient marker was within the first line of sight. The apparatus according to claim 8, wherein:

10. The apparatus further comprises an inertial measurement unit disposed on the first head-mounted device, and the at least one computer processor is configured to use data from the inertial measurement unit to track the movement of the first head-mounted device during a time when a portion of the patient marker is within the first line of sight and a time when the portion of the patient marker is not within the first line of sight, so as to overlay the virtual image on the body of the patient. The apparatus according to claim 8, wherein:

11. In response to a portion of the tool being within the first line of sight, the at least one computer processor is configured to generate, on the first head-mounted display, the virtual image of the tool and the anatomical structure of the patient by overlaying the virtual image on the body of the patient based on the position of the first head-mounted device relative to the tool, which is determined based on data received from the first tracking device. The apparatus according to claim 8, wherein:

12. When at least a portion of the patient marker or a portion of the tool is within the first line of sight and the virtual image of the tool and the anatomical structure of the patient is generated on the first head-mounted display, the virtual anatomical structure in the virtual image is aligned with the actual anatomical structure of the patient based on data received from the first tracking device. The apparatus according to claim 1, wherein:

13. The virtual images of the tool and the anatomical structure of the patient are generated on a first part of the first head-mounted display and overlaid on the patient's body, and are aligned with a part of the patient's body visible through a second part of the first head-mounted display. The device according to claim 12, characterized in that.

14. The patient marker has a plurality of tracking elements, and the at least one computer processor is configured to continuously identify the patient marker in an active manner. When some of the plurality of tracking elements are not within the first line of sight, the at least one computer processor is further configured to track the patient marker already identified using a tracking algorithm. The device according to claim 1, characterized in that.

15. When at least a part of the patient marker or a part of the tool is not within the first line of sight, the at least one computer processor is further configured to generate virtual images of the tool and the anatomical structure of the patient on the first head-mounted display from the first line of sight. The tool in the virtual image is aligned with the anatomical structure in the virtual image based on data received from the second tracking device. The device according to claim 1, characterized in that.

16. A computer-implemented method for generating an augmented reality image on a first head-mounted display of a first head-mounted device, comprising: When at least a part of the patient marker and a part of the tool are both within the first line of sight of the first head-mounted device, based on data received from a first tracking device disposed on the first head-mounted device while being worn by a first person, and without using data from a second tracking device, generating an augmented reality image on the first head-mounted display, the augmented reality image having a virtual image (a) of the tool and the anatomical structure of the patient overlaid on the patient's body (b); and When at least one of the part of the patient marker and the part of the tool is within the first line of sight and the other is not within the first line of sight, generating a virtual image of the tool and the anatomical structure of the patient on the first head-mounted display based at least in part on the data received from the second tracking device; having the patient marker is disposed on the patient's body, the tool is configured to be disposed within the patient's body, and the first tracking device and the second tracking device are each configured to track at least a part of the patient marker and the part of the tool from the first line of sight and the second line of sight; A method characterized by this.

17. The method according to claim 16, wherein the first tracking device and the second tracking device are configured to track the part of the tool by tracking a tool marker.

18. The second tracking device is configured to track at least a part of the part of the tool and the patient marker from the second line of sight, and is disposed on a second head-mounted device worn by a second person, and the second head-mounted device has a second head-mounted display, The method further includes generating an additional augmented reality image on the second head-mounted display. The method according to claim 16, characterized by this.

19. Based at least in part on the data received from the second tracking device, the step of generating the virtual image of the tool and the anatomical structure of the patient on the first head-mounted display is: In response to the part of the patient marker being within the first line of sight and the part of the tool not being within the first line of sight: Using the data received from the second tracking device to determine the position of the tool relative to the anatomical structure of the patient; and Based on the determined position of the tool relative to the anatomical structure of the patient, generating the virtual image of the tool and the anatomical structure of the patient on the first head-mounted display; and Determining the position of the patient's body relative to the first head-mounted device based on the data received from the first tracking device; and Based on the determined position of the patient's body relative to the first head-mounted device, superimposing the virtual image on the patient's body; having The method according to any one of claims 16 - 18, characterized in that.

20. The step of generating the virtual image of the tool and the patient's anatomical structure on the first head-mounted display, based at least in part on data received from the second tracking device: In response to a part of the patient marker not being within the first line of sight: Using the data received from the second tracking device to determine the position of the tool relative to the patient's anatomical structure; Based on the determined position of the tool relative to the patient's anatomical structure, generating the virtual image of the tool and the patient's anatomical structure on the first head-mounted display; The method according to any one of claims 16 - 18, characterized in that it has.

21. The step of generating the virtual image of the tool and the patient's anatomical structure on the first head-mounted display Further includes the step of superimposing the virtual image on the patient's body based on the position of the patient's body relative to the first head-mounted device, which is determined based on data received from the first tracking device when a part of the patient marker was within the first line of sight. The method according to claim 20, characterized in that.

22. The step of superimposing the virtual image on the patient's body includes tracking the movement of the first head-mounted device using an inertial measurement unit disposed on the first head-mounted device at a time between when a part of the patient marker was within the first line of sight and when a part of the patient marker was not within the first line of sight. The method according to claim 20, characterized in that.

23. Based on the position of the first head-mounted device with respect to the tool, which is determined based on data received from the first tracking device in response to a part of the tool being within the first line of sight, generating, on the first head-mounted display, the virtual images of the tool and the anatomical structure of the patient by superimposing the virtual image on the patient's body, further comprising, the method according to claim 20, characterized in that.

Images