Patient Visual Recognition System

Abstract

To provide a patient viewing system.SOLUTION: A method of viewing a patient including inserting a catheter is described for health procedure navigation. A CT scan is carried out on a body part of a patient. Raw data from the CT scan is processed to create three-dimensional image data and store the image data in a data storage. Projectors receive generated light in a pattern representative of the image data, and waveguides guide the light to a retina of an eye of a viewer while light from an external surface of the body transmits to the retina of the eye so that the viewer sees the external surface of the body augmented with the processed data rendering of the body part.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority from U.S. Provisional Patent Application No. 62 / 721,516, filed on August 22, 2018, and U.S. Provisional Patent Application No. 62 / 771,534, filed on November 26, 2018, which are hereby incorporated by reference in their entirety for all purposes.

[0002] The present invention relates to a patient visualization system.

Background Art

[0003] Endoscopic catheters are used by surgeons, physicians, and other viewers to capture images of body parts within a patient. A bronchoscope, for example, is an endoscopic catheter used to examine segmental bronchi. The catheter also performs other functions such as other endoscopic visualization functions, treats diseases, or performs surgical procedures.

[0004] Since the tip of the catheter is inserted into the patient's body using minimally invasive or non - invasive surgery, the viewer cannot see the location of the tip of the catheter with the naked eye. To assist the viewer, a screen is provided along with a combination of images. These images generally include preoperative and intraoperative computed tomography (CT) images along with live video data from a catheter camera within the tip of the catheter. These images are usually provided in different quadrants of the display.

[0005] In the case of the bronchi, all bronchial tunnels look the same. As a result, the viewer often has to guess where the bronchus is currently navigating. Such a viewer will often take multiple CT scans as the bronchoscope is pushed through the bronchus to determine a location within the lung.

Summary of the Invention

Means for Solving the Problems

[0006] The present invention provides a patient visualization system having a catheter with a lumen and a tip, a tip tracking device for detecting movement of the tip, left and right projectors, left and right optical waveguides connected to the left and right projectors, a processor, a computer-readable medium connected to the processor, a data storage unit on the computer-readable medium, and a set of instructions stored on the computer-readable medium and executable by the processor. The set of instructions may include: 1) a catheter tracking system connected to the tip tracking device, receiving measurement values based on the movement detected by the tip tracking device, determining the position of the tip based on the measurement values, and storing the position of the tip in the data storage unit; and 2) a stereoscopic analyzer connected to the data storage unit to receive image data, the stereoscopic analyzer determining left and right image data sets, the left and right projectors respectively projecting the left and right image data sets, and the left and right image data sets being different from each other to give the viewer a perception of a three-dimensional rendering.

[0007] The present invention further provides a method of visualizing a patient, including inserting the tip of a catheter into the patient's body, detecting movement of the tip using a tip tracking device, receiving measurement values based on the movement detected by the tip tracking device, determining the position of the tip based on the measurement values, storing the position of the tip, determining left and right image data sets based on the position of the tip, using left and right projectors to project the left and right image data sets as light respectively to generate light in a pattern representing the position of the tip, and guiding the light to the retinas of the viewer's left and right eyes so that the viewer can see the position of the tip, wherein the left and right image data sets are different from each other to give the viewer a perception of a three-dimensional rendering.

[0008] The present invention also provides a patient visualization system including a catheter having a lumen and a tip, a tip tracking device for detecting movement of the tip, a projector, an optical waveguide connected to the projector, a processor, a computer-readable medium connected to the processor, a data storage unit on the computer-readable medium, and a set of instructions stored on the computer-readable medium and executable by the processor. The set of instructions may include: 1) a catheter tracking system connected to the tip tracking device, receiving measurement values based on the movement detected by the tip tracking device, determining the position of the tip based on the measurement values, and storing the position of the tip in the data storage unit; 2) a past path calculator for storing the past path of the tip in the data storage unit; and 3) a catheter display integrator for displaying the past path of the tip together with the position of the tip.

[0009] The present invention further provides a method of visualizing a patient, including inserting the tip of a catheter into the body of a patient, detecting movement of the tip using a tip tracking device, receiving measurement values based on the movement detected by the tip tracking device, determining the position of the tip based on the measurement values, storing the position of the tip, determining left and right image data sets based on the position of the tip, generating light in a pattern representing the position of the tip using left and right projectors that respectively project the left and right image data sets as light, and guiding the light to the retinas of the left and right eyes of an observer so that the position of the tip is visible to the observer, wherein the left and right image data sets are different from each other to give the observer a perception of three-dimensional rendering, storing the past path of the tip in the data storage unit, and displaying the past path of the tip together with the position of the tip.

[0010] The present invention also provides a patient visualization system including a catheter having a lumen and a tip, a tip tracking device for detecting movement of the tip, a projector, an optical waveguide connected to the projector, a processor, a computer-readable medium connected to the processor, a data storage unit on the computer-readable medium, and a set of instructions stored on the computer-readable medium and executable by the processor. The set of instructions may include: 1) a catheter tracking system connected to the tip tracking device, receiving measurement values based on the movement detected by the tip tracking device, determining the position of the tip based on the measurement values, and storing the position of the tip in the data storage unit; 2) a predicted path calculator for calculating a future path of the tip based on the position of the tip, wherein the catheter display integrator displays the future path; and 3) a catheter display integrator for displaying the future path of the tip together with the position of the tip.

[0011] The present invention further provides a method for visualizing a patient, including inserting the tip of a catheter into the body of a patient, detecting movement of the tip using a tip tracking device, receiving measurement values based on the movement detected by the tip tracking device, determining the position of the tip based on the measurement values, storing the position of the tip, determining left and right image data sets based on the position of the tip, generating light in a pattern representing the position of the tip using left and right projectors that respectively project the left and right image data sets as light, and guiding the light to the retinas of the viewer's left and right eyes so that the position of the tip is visible to the viewer, wherein the left and right image data sets are different from each other to give the viewer a perception of three-dimensional rendering, calculating a future path of the tip based on the position of the tip, and displaying the future path of the tip together with the position of the tip.

[0012] The present invention also provides a patient visualization system including a transmitter, an energy source connected to the transmitter to activate the transmitter, wherein the patient's body is positionable relative to the transmitter for the transmitter to generate a forward wave at a body part within the body, a receiver positionable relative to the body to detect a return wave from the body part, the return wave from the body part being responsive to the forward wave generated by the transmitter, a processor, a computer-readable medium connected to the processor, a data storage unit on the computer-readable medium, and a set of instructions stored on the computer-readable medium and executable by the processor. The set of instructions includes: 1) a raw data receiving unit that receives raw data of the return wave detected by the receiver and stores the raw data in the data storage unit; 2) an image generating unit connected to the data storage unit to process the raw data of the return wave to generate image data representing an image and store the image data in the data storage unit; 3) an image data receiving unit that receives the image data from the data storage unit; 4) a projector connected to the image data receiving unit to receive the image data and generate light in a pattern representing the image data; and 5) an optical waveguide connected to the projector to direct light to the retina of the viewer while light from the outer surface of the body is transmitted through the retina of the eye so that the viewer can see the outer surface of the body extended using the rendering of the body part.

[0013] The present invention further includes activating a transmitter to generate a forward wave at a body part within the body, detecting a return wave from the body part using a receiver, where the return wave from the body part responds to the forward wave generated by the transmitter, receiving raw data of the return wave detected by the receiver, storing the raw data in a data storage unit, processing the raw data of the return wave to generate image data representing an image, storing the image data in the data storage unit, receiving the image data from the data storage unit, generating light in a pattern representing the image data, and guiding the light to the retina of the viewer's eye while light from the outer surface of the body is transmitted through the retina of the eye so that the viewer can see the outer surface of the body extended using the rendering of the body part. The present invention provides, for example, the following. (Item 1) A patient viewing system, a catheter having a lumen and a tip, a tip tracking device for detecting the movement of the tip, left and right projectors, left and right optical waveguides connected to the left and right projectors, a processor, a computer-readable medium connected to the processor, a data storage unit on the computer-readable medium, image data stored on the data storage unit, a set of instructions stored on the computer-readable medium and executable by the processor, a catheter tracking system connected to the tip tracking device, where the catheter tracking system receives a measurement value based on the movement detected by the tip tracking device, determines the position of the tip based on the measurement value, and stores the position of the tip in the data storage unit, A stereoscopic analyzer connected to the data storage unit so as to receive the image data, the stereoscopic analyzer determining left and right image data sets, the left and right projectors respectively projecting the left and right image data sets, the left and right image data sets being different from each other so as to give the viewer the perception of three-dimensional rendering, and the stereoscopic analyzer A set of instructions including A patient viewing system comprising (Item 2) A head-mountable frame, wherein the optical waveguide is fixed to the head-mountable frame, the head-mountable frame The patient viewing system according to item 1, further comprising (Item 3) The patient viewing system according to item 2, wherein the optical waveguide is a transparent optical waveguide positioned between the eye and the outer surface of the body. (Item 4) The patient viewing system according to item 2, further comprising a head unit detection device for detecting movement of the head-mountable frame, wherein the set of instructions is A display adjustment algorithm connected to the head unit detection device and receiving a measurement value based on the movement detected by the head unit detection device and calculating an installation value, and A display positioning algorithm for correcting the position of the body part within the view of the eye based on the installation value The patient viewing system according to item 2, comprising (Item 5) The head unit detection device is A head unit inertial measurement unit (IMU) mounted on the head-mountable frame, the head unit IMU including a motion sensor for detecting movement of the head-mountable frame, the head unit IMU The patient viewing system according to item 4, comprising (Item 6) The head unit detection device is A head unit camera mounted on the head mountable frame, wherein the head unit camera detects movement of the head mountable frame by photographing an image of an object within the view of the head unit camera. The patient visual recognition system according to item 4, including the same. (Item 7) The patient visual recognition system according to item 6, wherein the set of commands includes an image processing system that analyzes the image and detects a posture position of the head mountable frame. (Item 8) A transmitter, An energy source connected to the transmitter to activate the transmitter, wherein the patient's body can be positioned relative to the transmitter so that the transmitter generates a forward wave at a body part within the body. A receiver that can be positioned relative to the body to detect a return wave from the body part, wherein the return wave from the body part responds to the forward wave generated by the transmitter. Further comprising, The set of commands includes, An unprocessed data receiving unit that receives unprocessed data of the return wave detected by the receiver and stores the unprocessed data in the data storage unit. An image generation unit connected to the data storage unit to process the unprocessed data of the return wave to generate image data representing an image and store the image data in the data storage unit. An image data receiving unit that receives the image data from the data storage unit. A catheter display integrator that combines the position of the tip with the image data, wherein the pattern of light generated by the projector includes a pattern representing the image data and the position of the tip. The patient visual recognition system according to item 1, including the same. (Item 9) A computed tomography (CT) scanner, A base, a platform for the patient, and a rotor mounted on the base for rotation about the patient, the transmission being an X-ray transmission machine fixed to the rotor for transmitting X-ray waves, the receiver being an X-ray detector fixed to the rotor for detecting the X-ray waves, the movement of the platform relative to the base enabling movement of the patient relative to a plane extending from the X-ray transmission machine to the X-ray detector, and the rotor A computed tomography (CT) scanner including The patient visualization system according to item 8, comprising (Item 10) The set of instructions a past path calculator for storing the past path of the tip in the data storage unit, the catheter display integrator displaying the past path of the tip together with the position of the tip, the past path calculator The patient visualization system according to item 1, comprising (Item 11) The set of instructions a past path calculator for storing the past path of the tip, and a mesh generator for generating a three-dimensional mesh around the past path of the tip and storing the mesh in the data storage unit, the catheter display integrator displaying the mesh together with the position of the tip, the mesh generator The patient visualization system according to item 1, comprising (Item 12) a catheter camera within the tip, the catheter camera capturing video data, the catheter camera further comprising The set of instructions a video data receiving unit connected to the catheter camera within the tip for receiving the video data, the catheter display integrator displaying a live video based on the video data, the video data receiving unit The patient visualization system according to item 11, comprising (Item 13) The set of commands A predicted path calculator that calculates a future path of the tip based on the position of the tip, wherein the catheter display integrator displays the future path, predicted path calculator The patient visualization system according to item 1, comprising: (Item 14) The patient visualization system according to item 13, wherein the future path is calculated based on the movement detected by the tip tracking device. (Item 15) The movement detected by the tip tracking device is a first amount of movement from a first position through a first angle in a selected direction to a second position, and the future path is a second amount of movement from the second position through a second angle in the selected direction to a third position. The patient visualization system according to item 14. (Item 16) The patient visualization system according to item 1, further comprising an invasive surgical tool on the tip. (Item 17) A camera on the tip, At least one display for displaying an image captured by the camera to an operator The patient visualization system according to item 1, further comprising: (Item 18) The first display displays the three-dimensional rendering and the image captured by the camera to a first operator. The patient visualization system according to item 17. (Item 19) The first display displays the three-dimensional rendering to a first operator, A second display that displays the image captured by the camera to a second operator The patient visualization system according to item 17, further comprising: (Item 20) A first display for displaying the three-dimensional rendering to a first operator, A data collection system that collects data without using the catheter, A second display for displaying the data collected using the data collection system to a second operator, and The patient visualization system according to item 1, further comprising. (Item 21) A load detection system connected to the tip to determine the operating load on the tip, and A warning system connected to the load detection system to warn an operator when the load on the tip exceeds a predetermined limit. The patient visualization system according to item 1, further comprising. (Item 22) The patient visualization system according to item 21, wherein the warning is one of a visual warning, an audio warning, and a tactile warning. (Item 23) A motion detection system for observing motion artifacts, and A position adjustment system connected to the motion detection system to perform position adjustment on the tip based on the motion artifacts. The patient visualization system according to item 1, further comprising. (Item 24) The motion detection system collects feedback control images of patient activity at three separate times at times T1 to T2 to T3, the images are analyzed at time T4 to determine changes in patient position, and the adjustment of the control input based on the changes in patient position observed over times T1 to T3 is performed at time T5. The patient visualization system according to item 23. (Item 25) If (T5 - T3 = T3 - T2) ∩ (y3 - y2) < (y2 - y1), the correction at T5 < (y3 - y2). The patient visualization system according to item 24. (Item 26) A method of visualizing a patient, comprising: Inserting the tip of a catheter into a patient's body, and Detecting the movement of the tip using a tip tracking device. Receiving a measurement value based on the movement detected by the tip tracking device; Determining the position of the tip based on the measurement value; Storing the position of the tip; Determining left and right image data sets based on the position of the tip; Generating light in a pattern representing the position of the tip, respectively, using left and right projectors that project the left and right image data sets as light; Guiding the light to the retinas of the viewer's left and right eyes so that the position of the tip is visible to the viewer, wherein the left and right image data sets are different from each other so as to give the viewer a perception of three-dimensional rendering; A method comprising. (Item 27) Mounting a head-mounted frame on the viewer's head, wherein the light waveguide is fixed to the head-mounted frame; The method according to item 26, further comprising. (Item 28) The method according to item 27, wherein the light waveguide is a transparent light waveguide positioned between the eye and the outer surface of the body. (Item 29) Detecting movement of the head-mounted frame; Calculating an installation value based on the detected movement; Correcting the position of the body part within the view of the eye based on the installation value; The method according to item 27, further comprising. (Item 30) The method according to item 29, wherein the movement is detected using a motion sensor of a head unit inertial measurement unit (IMU) mounted on the head-mounted frame. (Item 31) The movement is detected using a head unit camera mounted on the head mountable frame, and the head unit camera detects the movement of the head mountable frame by photographing an image of an object within the view of the head unit camera, the method according to item 29. (Item 32) The method according to item 31, further comprising analyzing the image and detecting a pose position of the head mountable frame. (Item 33) Activating a transmitter to generate a forward wave in a body part within the body, Detecting a return wave from the body part using a receiver, the return wave from the body part being responsive to the forward wave generated by the transmitter, Receiving raw data of the return wave detected by the receiver, Storing the raw data in a data storage unit, Processing the raw data of the return wave to generate image data representing an image, Storing the image data in the data storage unit, Receiving the image data from the data storage unit, Combining the position of the tip with the image data, the generated light pattern including a pattern representing the image data and the position of the tip, The method according to item 26, further comprising. (Item 34) The transmitter is an X-ray transmitter fixed to a rotor and transmitting X-ray waves, the receiver is an X-ray detector fixed to the rotor so as to detect the X-ray waves, and the movement of the platform relative to the base enables the movement of the patient relative to a plane extending from the X-ray transmitter to the X-ray detector, the method according to item 33. (Item 35) Storing a past path of the tip in the data storage unit, Displaying the past path of the tip together with the position of the tip The method according to item 26, further comprising (Item 36) Generating a three-dimensional mesh around the past path of the tip; Storing the mesh in the data storage unit and the catheter display integrator; Displaying the mesh together with the position of the tip The method according to item 26, further comprising (Item 37) Capturing video data using a catheter camera within the tip; Receiving the video data, wherein the catheter display integrator displays a live video based on the video data; Displaying a live video based on the video data The method according to item 36, further comprising (Item 38) The set of instructions Calculating a future path of the tip based on the position of the tip; Displaying the future path The method according to item 26, comprising (Item 39) The method according to item 38, wherein the future path is calculated based on the movement detected by the tip tracking device. (Item 40) The movement detected by the tip tracking device is a first amount of movement from a first position through a first angle in a selected direction to a second position, and the future path is a second amount of movement from the second position through a second angle in the selected direction to a third position. The method according to item 39. (Item 41) The method according to item 26, further comprising performing an invasive procedure using an invasive surgical tool on the tip. (Item 42) Capturing an image using a camera on the tip; Displaying the image captured by the camera to the operator The method according to item 26, further comprising this. (Item 43) The method according to item 42, further comprising displaying the 3D rendering and the image captured by the camera to a first operator. (Item 44) Displaying the 3D rendering to a first operator Displaying the image captured by the camera to a second operator The method according to item 42, further comprising this. (Item 45) Displaying the 3D rendering to a first operator Collecting data without using the catheter Displaying the data collected without using the catheter to a second operator The method according to item 26, further comprising this. (Item 46) Determining the operating load on the tip Warning the operator when the load on the tip exceeds a predetermined limit The method according to item 26, further comprising this. (Item 47) The method according to item 46, wherein the warning is one of a visual warning, an audio warning, and a tactile warning. (Item 48) Observing motion artifacts Performing position adjustment on the tip based on the motion artifacts The method according to item 26, further comprising this. (Item 49) The motion detection system collects feedback control images of patient activity at three separate times at times T1 to T2 to T3, and the images are analyzed at time T4 to determine changes in patient position, and the adjustment of the control input based on the changes in patient position observed over times T1 to T3 is performed at time T5, the method according to item 48. (Item 50) (When (T5 - T3 = T3 - T2) ∩ (y3 - y2) < (y2 - y1), the correction at T5 < (y3 - y2), the method according to item 49. (Item 51) A patient visualization system, A catheter having a lumen and a tip, A tip tracking device for detecting movement of the tip, A projector, An optical waveguide connected to the projector, A processor, A computer-readable medium connected to the processor, A data storage unit on the computer-readable medium, A set of instructions stored on the computer-readable medium and executable by the processor, A catheter tracking system, the catheter tracking system is connected to the tip tracking device, receives measurement values based on the movement detected by the tip tracking device, determines the position of the tip based on the measurement values, and stores the position of the tip in the data storage unit, a catheter tracking system, A past path calculator for storing the past path of the tip in the data storage unit, A catheter display integrator, the catheter display integrator displays the past path of the tip together with the position of the tip, a catheter display integrator including a set of instructions and A patient visualization system comprising. (Item 52) The set of instructions, A mesh generator that generates a three-dimensional mesh around the past path of the tip and stores the mesh in the data storage unit, wherein the catheter display integrator displays the mesh together with the position of the tip. The patient visualization system according to item 51, including (Item 53) A catheter camera within the tip, the catheter camera capturing video data. Further comprising The set of instructions A video data receiving unit connected to the catheter camera within the tip to receive the video data, wherein the catheter display integrator displays a live video based on the video data. The patient visualization system according to item 51, including (Item 54) The set of instructions An expected path calculator that calculates the future path of the tip based on the position of the tip, wherein the catheter display integrator displays the future path. The patient visualization system according to item 51, including (Item 55) The patient visualization system according to item 54, wherein the future path is calculated based on the movement detected by the tip tracking device. (Item 56) The movement detected by the tip tracking device is a first amount of movement from a first position through a first angle in a selected direction to a second position, and the future path is a second amount of movement from the second position through a second angle in the selected direction to a third position. The patient visualization system according to item 55. (Item 57) A head-mountable frame to which the optical waveguide is fixed. The patient visualization system according to item 51, further comprising (Item 58) The patient viewing system according to item 57, wherein the light waveguide tube is a transparent light waveguide tube positioned between the eye and the outer surface of the body. (Item 59) The projector is a left projector, the light waveguide tube is a left light waveguide tube, and the eye is the left eye of the viewer. A right projector connected to the image data receiving unit so as to receive the image data, the right projector generating light in a pattern representing the image data, and a right projector. A right light waveguide tube, wherein the right light waveguide tube is connected to the right projector so as to guide the light from the right projector to the retina of the right eye of the viewer while the light from the outer surface of the body is transmitted through the retina of the right eye so that the outer surface of the body extended using the rendering of the body part with the right eye can be seen by the viewer. The patient viewing system according to item 57, further comprising: (Item 60) The set of instructions A stereoscopic analyzer connected to the data receiving unit so as to receive the image data, the stereoscopic analyzer determining left and right image data sets, the left and right projectors respectively projecting the left and right image data sets, and the left and right image data sets being different from each other so as to give the viewer a perception of three-dimensional rendering. The patient viewing system according to item 59, comprising: (Item 61) Further comprising a head unit detection device for detecting movement of the head mountable frame. The set of instructions A display adjustment algorithm, the display adjustment algorithm being connected to the head unit detection device, receiving a measurement value based on the movement detected by the head unit detection device, and calculating a setting value. A display positioning algorithm for correcting the position of the body part within the view of the eye based on the setting value. The patient visual recognition system according to item 57, including (Item 62) The head unit detection device is A head unit inertial measurement unit (IMU) mounted on the head-mountable frame, wherein the head unit IMU includes a motion sensor for detecting the movement of the head-mountable frame, the head unit IMU The patient visual recognition system according to item 61, including (Item 63) The head unit detection device is A head unit camera mounted on the head-mountable frame, wherein the head unit camera detects the movement of the head-mountable frame by taking an image of an object within the view of the head unit camera, the head unit camera including The set of commands is The patient visual recognition system according to item 61, including an image processing system for analyzing the image and detecting the pose position of the head-mountable frame. (Item 64) A transmitter, and An energy source connected to the transmitter so as to activate the transmitter, wherein the patient's body can be positioned relative to the transmitter for the transmitter to generate a forward wave at a body part within the body, the energy source A receiver that can be positioned relative to the body to detect a return wave from the body part, wherein the return wave from the body part responds to the forward wave generated by the transmitter, the receiver further comprising The set of commands is An unprocessed data receiving unit that receives the unprocessed data of the return wave detected by the receiver and stores the unprocessed data in the data storage unit, and An image generation unit connected to the data storage unit to process the unprocessed data of the return wave to generate image data representing an image and store the image data in the data storage unit, and An image data receiving unit that receives the image data from the data storage unit A catheter display integrator that combines the position of the tip with the image data, wherein the pattern of the light generated by the projector includes a pattern representing the image data and the position of the tip The patient visualization system according to item 31, comprising (Item 65) A computed tomography (CT) scanner A base A platform for the patient A rotor mounted on the base for rotation about the patient, wherein the transmitter is fixed to the rotor and transmits X-ray waves, the receiver is an X-ray detector fixed to the rotor to detect the X-ray waves, and the movement of the platform relative to the base enables the movement of the patient relative to a plane extending from the X-ray transmitter to the X-ray detector A computed tomography (CT) scanner comprising The patient visualization system according to item 64, comprising (Item 66) The patient visualization system according to item 51, further comprising an invasive surgical tool on the tip (Item 67) A camera on the tip At least one display for displaying the image captured by the camera to an operator The patient visualization system according to item 51, further comprising (Item 68) The patient visualization system according to item 67, wherein the first display displays the three-dimensional rendering and the image captured by the camera to a first operator (Item 69) The first display displays the three-dimensional rendering to a first operator A second display that displays the image captured by the camera to a second operator The patient visualization system according to item 67, further comprising (Item 70) A first display for displaying the three-dimensional rendering to a first operator, A data collection system that collects data without using the catheter, And a second display for displaying the data collected using the data collection system to a second operator The patient visualization system according to item 51, further comprising (Item 71) A load detection system connected to the tip to determine the operating load on the tip, A warning system connected to the load detection system to warn the operator when the load on the tip exceeds a predetermined limit The patient visualization system according to item 51, further comprising (Item 72) The patient visualization system according to item 71, wherein the warning is one of a visual warning, an audio warning, and a tactile warning. (Item 73) A motion detection system for observing motion artifacts, And a position adjustment system connected to the motion detection system to perform position adjustment on the tip based on the motion artifacts The patient visualization system according to item 51, further comprising (Item 74) The motion detection system collects feedback control images of patient activity at three separate times T1 to T2 to T3, the images are analyzed at time T4 to determine changes in patient position, and the adjustment of the control input based on the changes in patient position observed over times T1 to T3 is performed at time T5. The patient visualization system according to item 73. (Item 75) If (T5 - T3 = T3 - T2) ∩ (y3 - y2) < (y2 - y1), the correction at T5 < (y3 - y2). The patient visualization system according to item 74. (Item 76) A method for visually recognizing a patient, comprising: inserting the tip of a catheter into the patient's body; detecting the movement of the tip using a tip tracking device; receiving a measurement value based on the movement detected by the tip tracking device; determining the position of the tip based on the measurement value; storing the position of the tip; determining left and right image datasets based on the position of the tip; generating light in a pattern representing the position of the tip using left and right projectors that respectively project the left and right image datasets as light; guiding the light to the retinas of the viewer's left and right eyes so that the position of the tip is visible to the viewer, wherein the left and right image datasets are different from each other so as to give the viewer a perception of three-dimensional rendering; storing the past path of the tip in the data storage unit; displaying the past path of the tip together with the position of the tip A method comprising the above. (Item 77) generating a three-dimensional mesh around the past path of the tip; storing the mesh in the data storage unit and the catheter display integrator; displaying the mesh together with the position of the tip The method according to item 76, further comprising the above. (Item 78) capturing video data using a catheter camera within the tip; receiving the video data, wherein the catheter display integrator displays a live video based on the video data; displaying a live video based on the video data The method according to item 76, further comprising the above. (Item 79) The set of commands is Based on the position of the tip, calculating a future path of the tip; Displaying the future path The method according to item 76, including (Item 80) The method according to item 79, wherein the future path is calculated based on the movement detected by the tip tracking device. (Item 81) The movement detected by the tip tracking device is a first amount of movement from a first position through a first angle in a selected direction to a second position, and the future path is a second amount of movement from the second position through a second angle in the selected direction to a third position. The method according to item 80. (Item 82) Mounting a head-mountable frame on the head of the viewer, wherein the optical waveguide is fixed to the head-mountable frame. The method according to item 76, further including (Item 83) The method according to item 82, wherein the optical waveguide is a transparent optical waveguide positioned between the eye and the outer surface of the body. (Item 84) The eye is the left eye of the viewer, Guiding the light to the retina of the right eye of the viewer while light from the outer surface of the body is transmitted through the retina of the right eye so that the outer surface of the body extended using the rendering of the body part with the right eye is visible to the viewer. The method according to item 82, further including (Item 85) Determining left and right image datasets that are different from each other so as to give the viewer a perception of 3D rendering. The method according to item 84, further including (Item 86) Detecting the movement of the head-mountable frame; Calculating a setting value based on the detected movement; modifying the position of the body part within the view of the eye based on the setting value; The method according to item 82, further comprising: (Item 87) The method according to item 86, wherein the movement is detected using a motion sensor of a head unit inertial measurement unit (IMU) mounted on the head-mountable frame. (Item 88) The movement is detected using a head unit camera mounted on the head-mountable frame, and the head unit camera detects the movement of the head-mountable frame by taking an image of an object within the view of the head unit camera, analyzing the image and detecting the pose position of the head-mountable frame; The method according to item 86, further comprising: (Item 89) activating a transmitter to generate a forward wave at a body part within the body; detecting a return wave from the body part using a receiver, wherein the return wave from the body part is in response to the forward wave generated by the transmitter; receiving raw data of the return wave detected by the receiver; storing the raw data in a data storage unit; processing the raw data of the return wave to generate image data representing an image; storing the image data in the data storage unit; receiving the image data from the data storage unit; combining the position of the tip with the image data, wherein the generated light pattern includes a pattern representing the image data and the position of the tip; The method according to item 76, further comprising: (Item 90) The transmitter is an X-ray transmitter fixed to the rotor for transmitting X-ray waves, the receiver is an X-ray detector fixed to the rotor for detecting the X-ray waves, and the movement of the platform relative to the base enables the movement of the patient relative to the plane extending from the X-ray transmitter to the X-ray detector, the method according to item 89. (Item 91) The method according to item 76, further comprising performing an invasive procedure using an invasive surgical tool on the tip. (Item 92) capturing an image using a camera on the tip, displaying the image captured by the camera to an operator and further comprising the method according to item 76. (Item 93) The method according to item 92, further comprising displaying the 3D rendering and the image captured by the camera to a first operator. (Item 94) displaying the 3D rendering to a first operator, displaying the image captured by the camera to a second operator and further comprising the method according to item 92. (Item 95) displaying the 3D rendering to a first operator, collecting data without using the catheter, displaying the data collected without using the catheter to a second operator and further comprising the method according to item 76. (Item 96) determining an operating load on the tip, warning the operator when the load on the tip exceeds a predetermined limit and further comprising the method according to item 76. (Item 97) The method according to item 96, wherein the warning is one of a visual warning, an audio warning, and a tactile warning. (Item 98) Observing the motion artifact, and Based on the motion artifact, performing position adjustment on the tip The method according to item 76, further comprising. (Item 99) The motion detection system collects feedback control images of patient activities at three separate times T1 to T2 to T3, the images are analyzed at time T4 to determine changes in patient position, and the adjustment of the control input based on the changes in patient position observed over time T1 to T3 is performed at time T5. The method according to item 98. (Item 100) When (T5 - T3 = T3 - T2) ∩ (y3 - y2) < (y2 - y1), the correction at T5 < (y3 - y2). The method according to item 99. (Item 101) A patient visualization system, comprising A catheter having a lumen and a tip, A tip tracking device for detecting movement of the tip, A projector, An optical waveguide tube connected to the projector, A processor, A computer-readable medium connected to the processor, A data storage unit on the computer-readable medium, A set of instructions stored on the computer-readable medium and executable by the processor, A catheter tracking system, the catheter tracking system is connected to the tip tracking device, receives a measurement value based on the movement detected by the tip tracking device, determines the position of the tip based on the measurement value, and stores the position of the tip in the data storage unit. An expected path calculator for calculating an expected future path of the tip based on the position of the tip, and a catheter display integrator for displaying the future path. A catheter display integrator, wherein the catheter display integrator displays the future path of the tip together with the position of the tip, a catheter display integrator and A set of instructions including A patient visualization system comprising (Item 102) The movement detected by the tip tracking device is a first amount of movement from a first position through a first angle in a selected direction to a second position, and the future path is a second amount of movement from the second position through a second angle in the selected direction to a third position. The patient visualization system according to item 101 (Item 103) The future path is calculated based on the movement detected by the tip tracking device. The patient visualization system according to item 101 (Item 104) A catheter camera within the tip, the catheter camera capturing video data, a catheter camera Further comprising The set of instructions is A video data receiving unit connected to the catheter camera within the tip to receive the video data, and the catheter display integrator displays a live video based on the video data. A video data receiving unit The patient visualization system according to item 101 including (Item 105) The set of instructions is A past path calculator that calculates the past path of the tip based on the position of the tip, and the catheter display integrator displays the future path. A past path calculator The patient visualization system according to item 101 including (Item 106) The set of instructions is A mesh generator that generates a three-dimensional mesh around the past path of the tip and stores the mesh in the data storage unit, wherein the catheter display integrator displays the mesh together with the position of the tip. The patient visualization system according to item 105, comprising (Item 107) The movement detected by the tip tracking device is a first amount of movement from a first position through a first angle in a selected direction to a second position, and the future path is a second amount of movement from the second position through a second angle in the selected direction to a third position. The patient visualization system according to item 106. (Item 108) A head-mountable frame, wherein the light waveguide is fixed to the head-mountable frame. The patient visualization system according to item 101, further comprising (Item 109) The light waveguide is a transparent light waveguide positioned between the eye and the outer surface of the body. The patient visualization system according to item 108. (Item 110) The projector is a left projector, the light waveguide is a left light waveguide, and the eye is the left eye of the viewer. A right projector connected to the image data receiving unit to receive the image data, the right projector generating light in a pattern representing the image data. A right light waveguide, the right light waveguide guiding light from the right projector to the retina of the right eye of the viewer while light from the outer surface of the body passes through the right light waveguide and is visible on the retina of the right eye, such that an extended outer surface of the body using rendering of a body part associated with the right eye of the viewer is visible. The patient visualization system according to item 108, further comprising (Item 111) The set of instructions A stereoscopic analyzer connected to the data receiving unit so as to receive the image data, the stereoscopic analyzer determining left and right image data sets, the left and right projectors respectively projecting the left and right image data sets, the left and right image data sets being different from each other so as to give the viewer a perception of three-dimensional rendering, stereoscopic analyzer The patient viewing system according to item 110, including (Item 112) Further comprising a head unit detection device for detecting movement of the head mountable frame, The set of instructions is A display adjustment algorithm, the display adjustment algorithm being connected to the head unit detection device, receiving a measurement value based on the movement detected by the head unit detection device, and calculating a setting value, display adjustment algorithm A display positioning algorithm for correcting the position of the body part within the view of the eye based on the setting value The patient viewing system according to item 108, including (Item 113) The head unit detection device is A head unit inertial measurement unit (IMU) mounted on the head mountable frame, the head unit IMU including a motion sensor for detecting movement of the head mountable frame, head unit IMU The patient viewing system according to item 112, including (Item 114) The head unit detection device is A head unit camera mounted on the head mountable frame, the head unit camera detecting movement of the head mountable frame by photographing an image of an object within the view of the head unit camera, head unit camera Including The set of instructions is The patient viewing system according to item 112, including an image processing system for analyzing the image and detecting the pose position of the head mountable frame. (Item 115) A transmitter, An energy source connected to the transmitter so as to activate the transmitter, wherein the patient's body can be positioned relative to the transmitter for the transmitter to generate a forward wave at a body part within the body, the energy source; A receiver that can be positioned relative to the body so as to detect a return wave from the body part, wherein the return wave from the body part responds to the forward wave generated by the transmitter, the receiver further comprising, The set of instructions, An unprocessed data receiving unit that receives the unprocessed data of the return wave detected by the receiver and stores the unprocessed data in the data storage unit; An image generation unit, wherein the image generation unit processes the unprocessed data of the return wave to generate image data representing an image and is connected to the data storage unit so as to store the image data in the data storage unit, the image generation unit; An image data receiving unit that receives the image data from the data storage unit; A catheter display integrator that combines the position of the tip with the image data, wherein the pattern of light generated by the projector includes a pattern representing the image data and the position of the tip, the catheter display integrator The patient visualization system according to item 1011, comprising. (Item 116) A computed tomography (CT) scanner, A base, A platform for the patient, A rotor mounted on the base for rotation about the patient, wherein the transmitter is an X-ray transmitter fixed to the rotor and transmitting X-ray waves, the receiver is an X-ray detector fixed to the rotor so as to detect the X-ray waves, and the movement of the platform relative to the base enables the movement of the patient relative to a plane extending from the X-ray transmitter to the X-ray detector, the rotor A computed tomography (CT) scanner including The patient visualization system according to item 115, comprising. (Item 118) The camera on the tip, and At least one display for displaying an image captured by the camera to an operator The patient visualization system according to item 101, further comprising. (Item 119) The first display displays the three-dimensional rendering and the image captured by the camera to a first operator. The patient visualization system according to item 118. (Item 120) The first display displays the three-dimensional rendering to a first operator, A second display that displays the image captured by the camera to a second operator The patient visualization system according to item 118, further comprising. (Item 121) A first display for displaying the three-dimensional rendering to a first operator, and A data collection system that collects data without using the catheter, A second display for displaying the data collected using the data collection system to a second operator The patient visualization system according to item 101, further comprising. (Item 122) A load detection system connected to the tip to determine the operating load on the tip, and A warning system connected to the load detection system to warn an operator when the load on the tip exceeds a predetermined limit The patient visualization system according to item 101, further comprising. (Item 123) The warning is one of a visual warning, an audio warning, and a tactile warning. The patient visualization system according to item 122. (Item 124) A motion detection system for observing motion artifacts, and a position adjustment system connected to the motion detection system to perform position adjustment at the tip based on the motion artifact The patient visualization system according to item 101, further comprising. (Item 125) The motion detection system collects feedback control images of patient activities at three separate times at times T1 to T2 to T3, the images are analyzed at time T4 to determine changes in patient position, and the adjustment of the control input based on the changes in patient position observed over times T1 to T3 is performed at time T5. The patient visualization system according to item 124. (Item 126) If (T5 - T3 = T3 - T2) ∩ (y3 - y2) < (y2 - y1), the correction at T5 < (y3 - y2). The patient visualization system according to item 125. (Item 127) A method of visualizing a patient, comprising: inserting the tip of a catheter into the patient's body; detecting the movement of the tip using a tip tracking device; receiving a measurement value based on the movement detected by the tip tracking device; determining the position of the tip based on the measurement value; storing the position of the tip; determining left and right image datasets based on the position of the tip; generating light in a pattern representing the position of the tip using left and right projectors that project the left and right image datasets as light, respectively; guiding the light to the retinas of the viewer's left and right eyes so that the position of the tip is visible to the viewer, wherein the left and right image datasets are different from each other so as to give the viewer a perception of 3D rendering; calculating a future path of the tip based on the position of the tip displaying a future path of the tip along with the position of the tip A method comprising: (Item 128) The movement detected by the tip tracking device is a first amount of movement from a first position through a first angle in a selected direction to a second position, and the future path is a second amount of movement from the second position through a second angle in the selected direction to a third position. The method according to item 127 (Item 129) The method according to item 127, wherein the future path is calculated based on the movement detected by the tip tracking device (Item 130) capturing video data using a catheter camera within the tip; receiving the video data, wherein the catheter display integrator displays a live video based on the video data; displaying a live video based on the video data The method according to item 127, further comprising: (Item 131) The set of instructions includes: storing a past path of the tip in the data storage unit; displaying the past path of the tip along with the position of the tip The method according to item 127, comprising: (Item 132) generating a three-dimensional mesh around the past path of the tip; storing the mesh in the data storage unit and the catheter display integrator; displaying the mesh along with the position of the tip The method according to item 131, further comprising: (Item 133) The movement detected by the tip tracking device is a first amount of movement from a first position through a first angle in a selected direction to a second position, and the future path is a second amount of movement from the second position through a second angle in the selected direction to a third position, the method according to item 132. (Item 134) Mounting a head-mountable frame on the viewer's head, wherein the light waveguide is fixed to the head-mountable frame. The method according to item 127, further comprising. (Item 135) The method according to item 134, wherein the light waveguide is a transparent light waveguide positioned between the eye and the outer surface of the body. (Item 136) The eye is the viewer's left eye, Guiding light to the retina of the viewer's right eye while light from the outer surface of the body passes through the retina of the right eye so that the outer surface of the body extended using rendering of the body part with the right eye is visible to the viewer. The method according to item 134, further comprising. (Item 137) Determining left and right image data sets, the left and right image data sets being different from each other so as to give the viewer a perception of a three-dimensional rendering. The method according to item 136, further comprising. (Item 138) Detecting movement of the head-mountable frame, Calculating an installation value based on the detected movement, Modifying the position of the body part within the view of the eye based on the installation value. The method according to item 134, further comprising. (Item 139) The method according to item 138, wherein the movement is detected using a motion sensor of a head unit inertial measurement unit (IMU) mounted on the head-mountable frame. (Item 140) The movement is detected using a head unit camera mounted on the head mountable frame, and the head unit camera detects the movement of the head mountable frame by photographing an image of an object within the view of the head unit camera, analyzes the image, and detects the attitude position of the head mountable frame The method according to item 138, further comprising. (Item 141) activating a transmitter to generate a forward wave in a body part within the body, and detecting a return wave from the body part using a receiver, wherein the return wave from the body part responds to the forward wave generated by the transmitter, receiving the raw data of the return wave detected by the receiver, and storing the raw data in a data storage unit, and processing the raw data of the return wave to generate image data representing an image, and storing the image data in the data storage unit, and receiving the image data from the data storage unit, and combining the position of the tip with the image data, wherein the generated light pattern includes a pattern representing the image data and the position of the tip, The method according to item 127, further comprising. (Item 142) The transmitter is an X-ray transmitter fixed to a rotor and transmitting X-ray waves, the receiver is an X-ray detector fixed to the rotor so as to detect the X-ray waves, and the movement of the platform relative to the base enables the movement of the patient relative to a plane extending from the X-ray transmitter to the X-ray detector. The method according to item 141. (Item 143) The method according to item 127, further comprising performing an invasive procedure using an invasive surgical tool on the tip. (Item 144) Capturing an image using the camera on the tip; Displaying the image captured by the camera to an operator; The method according to item 127, further comprising: (Item 145) Displaying the 3D rendering and the image captured by the camera to a first operator; The method according to item 144, further comprising: (Item 146) Displaying the 3D rendering to a first operator; Displaying the image captured by the camera to a second operator; The method according to item 144, further comprising: (Item 147) Displaying the 3D rendering to a first operator; Collecting data without using the catheter; Displaying the data collected without using the catheter to a second operator; The method according to item 127, further comprising: (Item 148) Determining an operating load on the tip; Warning the operator when the load on the tip exceeds a predetermined limit; The method according to item 127, further comprising: (Item 149) The method according to item 148, wherein the warning is one of a visual warning, an audio warning, and a tactile warning. (Item 150) Observing motion artifacts; Performing position adjustment on the tip based on the motion artifacts; The method according to item 127, further comprising: (Item 151) The motion detection system collects feedback control images of patient activities at three separate times, namely time T1, T2, and T3. The images are analyzed at time T4 to determine changes in the patient's position. The adjustment of the control input based on the changes in the patient's position observed over time T1 to T3 is performed at time T5, as described in item 150. (Item 152) (If (T5 - T3 = T3 - T2) ∩ (y3 - y2) < (y2 - y1), the correction at T5 < (y3 - y2), as described in item 151. (Item 153) A transmitter, An energy source connected to the transmitter to activate the transmitter, where the patient's body can be positioned relative to the transmitter such that the transmitter generates a forward wave at a body part within the body. A receiver that can be positioned relative to the body to detect a feedback wave from the body part, where the feedback wave from the body part responds to the forward wave generated by the transmitter. A processor, A computer-readable medium connected to the processor, A data storage unit on the computer-readable medium, A set of instructions stored on the computer-readable medium and executable by the processor, An unprocessed data receiving unit that receives the unprocessed data of the feedback wave detected by the receiver and stores the unprocessed data in the data storage unit. An image generation unit that processes the unprocessed data of the feedback wave to generate image data representing an image and is connected to the data storage unit to store the image data in the data storage unit. An image data receiving unit that receives the image data from the data storage unit. A projector connected to the image data receiving unit so as to receive the image data, the projector generating light in a pattern representing the image data, the projector and, An optical waveguide, the optical waveguide being connected to the projector so as to guide light to the retina of the viewer's eye while light from the outer surface of the body is transmitted through the retina of the eye so that an outer surface of the body extended using a rendering of a body part is visible to the viewer, the optical waveguide Including a set of instructions and A patient viewing system comprising. (Item 154) A head-mountable frame, the optical waveguide being fixed to the head-mountable frame, the head-mountable frame The patient viewing system according to item 153, further comprising. (Item 155) The patient viewing system according to item 154, wherein the optical waveguide is a transparent optical waveguide positioned between the eye and the outer surface of the body. (Item 156) The projector is a left projector, the optical waveguide is a left optical waveguide, the eye is the left eye of the viewer, A right projector connected to the image data receiving unit so as to receive the image data, the right projector generating light in a pattern representing the image data, the right projector and, A right optical waveguide, the right optical waveguide being connected to the right projector so as to guide light from the right projector to the retina of the viewer's right eye while light from the outer surface of the body is transmitted through the retina of the right eye so that an outer surface of the body extended using a rendering of a body part with the right eye is visible to the viewer, the right optical waveguide The patient viewing system according to item 154, further comprising. (Item 157) The set of instructions is A stereoscopic analyzer connected to the data receiving unit to receive the image data, the stereoscopic analyzer determining left and right image data sets, the left and right projectors respectively projecting the left and right image data sets, the left and right image data sets being different from each other so as to give the viewer a perception of 3D rendering, stereoscopic analyzer The patient viewing system according to item 156, including (Item 158) Further comprising a head unit detection device for detecting movement of the head-mountable frame, the set of instructions A display adjustment algorithm, the display adjustment algorithm being connected to the head unit detection device, receiving measurement values based on the movement detected by the head unit detection device, and calculating installation values, display adjustment algorithm A display positioning algorithm for correcting the position of the body part within the view of the eye based on the installation value The patient viewing system according to item 154, including (Item 159) The head unit detection device A head unit inertial measurement unit (IMU) mounted on the head-mountable frame, the head unit IMU including a motion sensor for detecting movement of the head-mountable frame, head unit IMU The patient viewing system according to item 158, including (Item 160) The head unit detection device A head unit camera mounted on the head-mountable frame, the head unit camera detecting movement of the head-mountable frame by taking an image of an object within the view of the head unit camera, head unit camera Including The set of instructions An image processing system for analyzing the image and detecting the pose position of the head-mountable frame The patient viewing system according to item 158, including (Item 161) A computed tomography (CT) scanner, a base, a platform for the patient, a rotor mounted on the base for rotation about the patient, wherein the transmitter is fixed to the rotor and transmits X-ray waves, the receiver is an X-ray detector fixed to the rotor to detect the X-ray waves, and movement of the platform relative to the base enables movement of the patient relative to a plane extending from the X-ray transmitter to the X-ray detector, the rotor comprising a computed tomography (CT) scanner The patient visualization system according to item 158, comprising the same. (Item 162) a catheter having a lumen and a tip, a tip tracking device for detecting movement of the tip further comprising, the set of instructions is a catheter tracking system, the catheter tracking system being connected to the tip tracking device, receiving measurement values based on the movement detected by the tip tracking device, and determining the position of the tip based on the measurement values, the catheter tracking system a catheter display integrator for combining the position of the tip with the image data, the pattern of light generated by the projector including a pattern representing the image data and the position of the tip, the catheter display integrator The patient visualization system according to item 153, comprising the same. (Item 163) the set of instructions is a past path computer for storing the past path of the tip in the data storage unit, the catheter display integrator displaying the past path of the tip together with the position of the tip, the past path computer The patient visualization system according to item 162, comprising the same. (Item 164) the set of instructions is A past path calculator that stores the past path of the tip A mesh generator that generates a three-dimensional mesh around the past path of the tip and stores the mesh in the data storage unit, and the catheter display integrator displays the mesh together with the position of the tip. The patient visualization system according to item 162, comprising: (Item 165) A catheter camera inside the tip, the catheter camera capturing video data further comprising: The set of instructions A video data receiving unit connected to the catheter camera inside the tip to receive the video data, and the catheter display integrator displays a live video based on the video data. The patient visualization system according to item 164, comprising: (Item 166) The set of instructions A predicted path calculator that calculates a future path of the tip based on the position of the tip, and the catheter display integrator displays the future path. The patient visualization system according to item 162, comprising: (Item 167) The patient visualization system according to item 166, wherein the future path is calculated based on the movement detected by the tip tracking device. (Item 168) The movement detected by the tip tracking device is a first amount of movement from a first position through a first angle in a selected direction to a second position, and the future path is a second amount of movement from the second position through a second angle in the selected direction to a third position. The patient visualization system according to item 167. (Item 169) A method for visualizing a patient, comprising: Activating a transmitter to generate a forward wave at a body part within the body, Detecting a feedback wave from the body part using a receiver, wherein the feedback wave from the body part responds to the forward wave generated by the transmitter, Receiving raw data of the feedback wave detected by the receiver, Storing the raw data in a data storage unit, Processing the raw data of the feedback wave to generate image data representing an image, Storing the image data in the data storage unit, Receiving the image data from the data storage unit, Generating light in a pattern representing the image data, Inducing the light onto the retina of the viewer's eye while light from the outer surface of the body is passing through to the retina of the eye, such that the viewer can see the outer surface of the body extended using the rendering of the body part, A method comprising the above. (Item 170) Mounting a head-mounted frame on the viewer's head, wherein the light waveguide is fixed to the head-mounted frame, The method according to item 169, further comprising the above. (Item 171) The method according to item 170, wherein the light waveguide is a transparent light waveguide positioned between the eye and the outer surface of the body. (Item 172) The eye is the viewer's left eye, Inducing the light onto the retina of the viewer's right eye while light from the outer surface of the body is passing through to the retina of the right eye, such that the viewer can see the outer surface of the body extended using the rendering of the body part with the right eye, The method according to item 170, further comprising the above. (Item 173) Determining left and right image datasets, the left and right image datasets being different from each other so as to give the viewer a perception of 3D rendering The method according to item 172, further comprising (Item 174) Detecting movement of the head-mountable frame; Calculating an installation value based on the detected movement; Modifying the position of the body part within the view of the eye based on the installation value The method according to item 170, further comprising (Item 175) The method according to item 174, wherein the movement is detected using a motion sensor of a head unit inertial measurement unit (IMU) mounted on the head-mountable frame. (Item 176) The movement is detected using a head unit camera mounted on the head-mountable frame, the head unit camera detecting the movement of the head-mountable frame by taking an image of an object within the view of the head unit camera, analyzing the image, and detecting the pose position of the head-mountable frame The method according to item 174, further comprising (Item 177) The transmitter is an X-ray transmitter fixed to the rotor and transmitting X-ray waves, the receiver is an X-ray detector fixed to the rotor so as to detect the X-ray waves, and the movement of the platform relative to the base enables the movement of the patient relative to a plane extending from the X-ray transmitter to the X-ray detector. The method according to item 174. (Item 178) Inserting the tip of the catheter into the patient's body; Detecting the movement of the tip using a tip tracking device; Receiving a measurement value based on the movement detected by the tip tracking device; Determining the position of the tip based on the measurement value; Combining the position of the tip with the image data, wherein the generated light pattern includes a pattern representing the image data and the position of the tip The method according to item 179, further comprising (Item 179) Storing the past path of the tip in the data storage unit Displaying the past path of the tip together with the position of the tip The method according to item 178, further comprising (Item 180) Generating a three-dimensional mesh around the past path of the tip Storing the mesh in the data storage unit and the catheter display integrator Displaying the mesh together with the position of the tip The method according to item 178, further comprising (Item 181) Capturing video data using a catheter camera within the tip Receiving the video data, wherein the catheter display integrator displays a live video based on the video data Displaying a live video based on the video data The method according to item 180, further comprising (Item 182) The set of instructions Calculating a future path of the tip based on the position of the tip Displaying the future path The method according to item 178, comprising (Item 183) The method according to item 182, wherein the future path is calculated based on the movement detected by the tip tracking device (Item 184) The movement detected by the tip tracking device is a first amount of movement from a first position through a first angle in a selected direction to a second position, and the future path is a second amount of movement from the second position through a second angle in the selected direction to a third position, the method according to item 183.

Brief Description of the Drawings

[0014] The present invention will be further described by way of example with reference to the accompanying drawings.

[0015]

Figure 1

[0016]

Figure 2

[0017]

Figure 3

[0018]

Figure 4

[0019]

Figure 5

[0020]

Figure 6

[0021]

Figure 7

[0022]

Figure 8

[0023]

Figure 9

[0024]

Figure 10

[0025]

Figure 11

[0026]

Figure 12

[0027]

Figure 13

[0028]

Figure 14

[0029]

Figure 15

[0030]

Figure 16

[0031]

Figure 17

[0032]

Figure 18

[0033]

Figure 19

[0034] FIG. 1 of the accompanying drawings illustrates a patient viewing system 20, according to an embodiment of the present invention, including a CT scanner 22, a data storage unit 24, a catheter 26, and a display system 28.

[0035] The data storage unit 24 is connected to the CT scanner 22. Raw data from the CT scanner 22 may be stored in the data storage unit 24. The data storage unit 24 also stores image data based on the raw data.

[0036] The display system 28 is connected to the data storage unit 24 so as to be able to read out image data from the data storage unit 24. The catheter 26 is connected to the display system 28 so that the display system 28 can read out measurement values and video data from the catheter 26 for further processing or for display to the viewer.

[0037] During use, the patient is positioned at the station 32 in the CT scanner 22. The patient's body 30 is scanned using the CT scanner 22 so that the CT scanner 22 acquires raw data stored in the data storage unit 24. The raw data is then processed to acquire 3D image data.

[0038] The patient is transferred from the station 32 in the CT scanner 22 to the station 34 in the display system 28. The viewer uses the display system 28 to view the patient's body 30. The display system 28 also reads out image data from the data storage unit 24. The viewer uses the display system 28 to view an image in the form of a 3D rendering of the patient's body 30. The viewer inserts the catheter 26 into the body 30. The display system 28 reads out data from the tip of the catheter 26 for further processing or for display to the viewer.

[0039] FIG. 2 illustrates components of a patient viewing system 20 including a CT scanner 22, a data storage unit 24, an energy source 36, a data receiving unit 38, and an image generating unit 40, according to some embodiments.

[0040] The CT scanner 22 includes a base 42, a platform 44, a rotor 46, an X-ray transmitter 48, and a plurality of X-ray detectors 50.

[0041] The platform 44 is fixed to the base 42 through a mechanism (not shown) that allows the platform 44 to translate parallel to the base 42. An actuator such as a stepping motor (not shown) is operable to cause the platform 44 to translate parallel to the base 42.

[0042] The rotor 46 has an opening 52. The X-ray transmitter 48 is fixed to the rotor 46 on one side of the opening 52, and the X-ray detector 50 is fixed to the rotor 46 on the opposite side of the opening 52. The rotor 46 is mounted on the base 42 around the platform 44. The platform 44 moves relative to the opening 52 during its translation. A motor (not shown) is connected between the base 42 and the rotor 46 and is operable to rotate the rotor 46 around the platform 44.

[0043] The energy source 36 may be connected to the X-ray transmitter 48 through a switch 54. The X-ray detector 50 is connected to the data receiving unit 38. The data receiving unit 38 may be a software unit resident on a computer-readable medium of a computer. The data storage unit 24 is resident on a computer-readable medium. The computer-readable medium may be a single computer-readable medium, or may be separated within one personal computer, or several personal computers interconnected on a network. The data receiving unit 38 is connected to the data storage unit 24 either directly or via a network.

[0044] The image generation unit 40 may be a computer program resident on a computer-readable medium. The image generation unit 40 is connected to the data storage unit 24 either directly or via a network.

[0045] During use, an operator of the CT scanner 22 positions the patient by lying their body 30 across the platform 44. A motor connected between the base 42 and the rotor 46 is then switched on so that the rotor 46 rotates in a direction 58 about the platform 44 and the patient's body 30. The operator also switches on a motor to move the platform 44 in a translational direction relative to the base 42 such that the platform 44 moves in a direction 60 relative to the rotor 46. The operator then connects a switch 54 between the energy source 36 and the X-ray transmitter 48 to activate the X-ray transmitter 48. The X-ray transmitter then generates a forward X-ray wave 62.

[0046] The patient's body 30 is positioned relative to the X-ray transmitter 48 such that the forward X-ray wave 62 penetrates the body 30 to a body part (not shown) within the body 30. For the purposes of this example, the body part to be scanned is the patient's lungs. The lungs have many bronchioles through which a catheter can proceed. It is also possible to consider that a catheter could proceed through hollow passages within the heart, arteries, and veins of the blood circulatory system, etc. The system described herein may also find use in visualizing internal body parts without using a catheter for vision, surgery, or intervention, for example, visualizing growths within the abdomen, analyzing internal knee functions, etc. The body part reduces the energy of the forward X-ray wave 62. Different substances within the body part reduce the energy by different amounts. One of the X-ray detectors 50 is positioned relative to the body 30 to detect a feedback X-ray wave 64 from the body part. The feedback X-ray wave 64 from the body part is detected in response to the forward X-ray wave 62 and is essentially the forward X-ray wave 62 with a reduced output due to the reduction by the body part. Additional forward X-ray waves 66 are also shown. The additional X-ray waves are generated between the forward X-ray waves 62 and 66 and are detected by individual ones of the X-ray detectors 50. In this way, feedback X-ray waves are received from different parts of the body part.

[0047] The X-ray transmitter 48 and the X-ray detector 50 rotate together with the rotor 46 around a body part within the patient's body 30. In this way, the body part can be scanned from different angles so as to generate a two-dimensional "slice" of the biological structure. The CT scan can show bones, organs, and soft tissues. Subsequent slices are taken by moving the platform 44 in the direction 60. Each slice thus represents two-dimensional data, and together the slices represent the data of the body part in three dimensions.

[0048] The data receiving unit 38 receives the raw data of the feedback X-ray wave 64 from the X-ray detector 50. The raw data includes the time-series correlation among the angle of the X-ray transmitter 48 with respect to the body part within the patient's body 30, the energy detected by each of the X-ray detectors 50, the location of each of the X-ray detectors 50, and the position of the platform 44. The data receiving unit 38 stores the raw data as the raw data 68 of the feedback X-ray wave detected by the X-ray detector 50.

[0049] When sufficient raw data 68 of the body part is collected, the operator disconnects the switch 54 and stops the platform 44. The operator then stops the rotor 46 and removes the patient from the platform 44.

[0050] The image generating unit 40 reads the raw data 68 from the data storage unit 24. The image generating unit 40 generates image data based on the raw data 68. The image data includes a three-dimensional rendering of the body part. The image generating unit 40 then stores the image data as image data 70 in the data storage unit 24. The data storage unit 24 may be a single data storage unit or may be distributed among platforms. Thus, the raw data 68 and the image data 70 can be located within a single data storage unit in a personal computer or within several data storage units in several personal computers.

[0051] Figure 3 illustrates in more detail the components of the patient visualization system 20, showing, according to some embodiments, a data storage unit 24 (which holds image data 70), a catheter 26, and a display system 28.

[0052] The catheter 26 includes a lumen 76 and a tip 78 attached to an end of the lumen 76. The lumen is an elongate member (e.g., the cavity of a tubular part) that forms a majority of the length of the catheter 26. The lumen 76 includes a mechanism (not shown) operable to move the tip 78 in at least four orthogonal directions and all directions between the orthogonal directions. The tip 78 is thus steerable using the mechanism within the lumen 76. The lumen has a hollow bore large enough to hold the mechanism used to steer the tip, along with any electrical cables and / or optical fibers that may be required to relay signals from the tip through the lumen 76 to the display system 28.

[0053] The catheter 26 further includes a catheter inertial motion unit (IMU) 80 and a catheter camera 82 affixed to the tip 78. The catheter IMU 80 may be, for example, a semiconductor chip having several measurement devices formed therein. The measurement devices include one or more gyroscopes and one or more accelerometers. Measurements from the gyroscopes and accelerometers provide data indicative of the movement of the tip 78, either individually or in combination. Such movement can be tracked in six degrees of freedom, e.g., translational movement in the x, y, and z directions, and rotation about the x, y, and z axes.

[0054] The catheter camera 82 has a lens (not shown) on the side of the tip 78 facing the lumen 76. The catheter camera 82 is positioned to capture an image in the form of live video data within the area in front of the tip 78, i.e., on the side facing the lumen 76. While there may be multiple light sources and multiple cameras on different sides of the tip of the camera, for ease of discussion, it will be assumed that only a single camera, e.g., a built-in camera and light source on the distal end of the catheter, is present.

[0055] The display system 28 includes a head-mountable frame 86, left and right projectors 88A and 88B, left and right waveguides 90A and 90B, a detection device 92, and a vision algorithm 94. The left and right projectors 88A and 88B, the left and right waveguides 90A and 90B, and the detection device 92 are fixed to the head-mountable frame 86. The head-mountable frame 86 is shaped to be mounted on the head of a viewer. The components of the head-mountable frame 86 may include, for example, a strap (not shown) that wraps around the rear of the viewer's head.

[0056] The left and right projectors 88A and 88B are connected to a power supply. Each projector 88A or 88B has an individual input for image data to be provided to the individual projector 88A or 88B. When powered, the individual projector 88A or 88B generates light in a two-dimensional pattern and emits light therefrom. The left and right waveguides 90A and 90B are positioned to receive light from the left and right projectors 88A and 88B, respectively. The left and right waveguides 90A and 90B are transparent waveguides.

[0057] The detection device 92 includes a head unit IMU 100 and one or more head unit cameras 102. The head unit IMU 100 includes one or more gyroscopes and one or more accelerometers. The gyroscopes and accelerometers are typically formed within a semiconductor chip and are capable of detecting movement of the head unit IMU 100 and the head-mountable frame 86, including movement along three orthogonal axes and rotation about three orthogonal axes.

[0058] The head unit camera 102 continuously captures images from the environment around the head-mountable frame 86. The images can be compared with each other to detect movement of the head-mountable frame 86 and the viewer's head.

[0059] The visual algorithm 94 includes an image data receiving unit 106, a display positioning algorithm 108, a catheter integration system 110, a display adjustment algorithm 112, an image processing system 114, and a stereo analyzer 116. The image data receiving unit 106 is connected to the data storage unit 24 either through a direct connection or via a network. The components of the visual algorithm 94 are linked to each other through subroutines or calls. Through such subroutines or calls, the image data receiving unit 106 is linked to the stereo analyzer 116 via the display positioning algorithm 108.

[0060] The catheter integration system 110 may be connected to the catheter IMU 80 and the catheter camera 82 through conductors within the lumen 76. Those skilled in the art will understand that the visual algorithm 94 resides on a computing system, the catheter integration system 110 receives signals from the catheter camera 82 and the catheter IMU 80, and such signals can be converted from analog or digital data to computer software data. The catheter integration system 110 may be connected to the stereo analyzer 116 through subroutines or calls.

[0061] The display adjustment algorithm 112 and the image processing system 114 are each connected to the head unit IMU 100 and the head unit camera 102. Such connections pass through conductors and, where applicable, through an inverter that converts analog or digital data to computer software data. The display adjustment algorithm 112 may be connected to the display positioning algorithm 108 through subroutines and calls. The image processing system 114 may be connected to the display adjustment algorithm 112 through calls and subroutines.

[0062] In use, the viewer mounts the head - mountable frame 86 on their head. The left and right waveguides 90A and 90B are then positioned in front of the viewer's left and right eyes 120A and 120B.

[0063] The image data receiving unit 106 reads the image data 70 from the data storage unit 24 and provides the image data 70 to the display positioning algorithm 108. The display positioning algorithm 108 inputs the image data 70 into the stereo analyzer 116. The image data 70 is three-dimensional image data of a body part as described above. The stereo analyzer 116 analyzes the image data 70 and determines left and right image data sets based on the image data 70. The left and right image data sets are data sets representing two-dimensional images that are slightly different from each other for the purpose of giving the viewer the perception of three-dimensional rendering. The image data 70 is a static data set that does not change over time.

[0064] The stereo analyzer 116 inputs the left and right image data sets into the left and right projectors 88A and 88B. The left and right projectors 88A and 88B then generate left and right light patterns 122A and 122B. The components of the display system 28 are shown in a plan view, and the left and right light patterns 122A and 122B are shown in a front elevation view. Each light pattern 122A and 122B includes a plurality of pixels. For illustrative purposes, light rays 124A and 126A from two of the pixels are shown exiting the left projector 88A and entering the left light guide 90A. The light rays 124A and 126A are reflected from the side surface of the left light guide 90A. The light rays 124A and 126A are shown to propagate from left to right within the left light guide 90A through internal reflection, but it should be understood that the light rays 124A and 126A also propagate in the direction into the paper using a heat-resistant and reflective system. The light rays 124A and 126A exit the left light guide 90A through the pupil 128A and enter the left eye 120A through the pupil 130A of the left eye. The light rays 124A and 126A then impinge on the retina 132A of the left eye 120A. Thus, the left light pattern 122A impinges on the retina 132A of the left eye 120A. The viewer is given the perception that the pixels formed on the retina 132A are pixels 134A and 136A, which the viewer perceives as being at a distance on the side surface of the left light guide 90A facing the left eye 120A.

[0065] Similarly, the stereo analyzer 116 inputs the right image data set into the right projector 88B. The right projector 88B transmits a right light pattern 122B represented by pixels in the form of light rays 124B and 126B. The light rays 124B and 126B are reflected within the right waveguide 90B and exit through the pupil 128B. The light rays 124B and 126B then enter through the pupil 130B of the right eye 120B and impinge on the retina 132B of the right eye 120B. The pixels of the light rays 124B and 126B are perceived as pixels 134B and 136B behind the right optical waveguide 90B.

[0066] The patterns generated on the retinas 132A and 132B are individually perceived as the left and right images 140A and 140B shown in the front elevation view. The left and right images 140A and 140B are slightly different from each other due to the function of the stereo analyzer 116. The left and right images 140A and 140B are perceived as a 3D rendering in the viewer's brain.

[0067] As described, the left and right waveguides 90A and 90B are transparent. Light from actual objects on the sides of the left and right waveguides 90A and 90B facing the eyes 120A and 120B can project through the left and right waveguides 90A and 90B and impinge on the retinas 132A and 132B. In particular, light from the surface of the patient's body 30 projects onto the retinas 132A and 132B so that the viewer can see the surface of the patient's body 30. An augmented reality is generated in which the surface of the patient's body 30 visible to the viewer is extended using the 3D rendering perceived by the viewer, due to the left and right images 140A and 140B combined and perceived by the viewer.

[0068] The head unit IMU 100 detects all movements of the viewer's head. If the viewer moves, for example, counterclockwise around the patient's body 30 while simultaneously rotating their head counterclockwise and continuing to look at the patient's body 30, such movements will be detected by the gyroscope and accelerometer within the head unit IMU 100. The head unit IMU 100 provides the measurement values from the gyroscope and accelerometer to the display adjustment algorithm 112. The display adjustment algorithm 112 calculates the installation value and provides the installation value to the display positioning algorithm 108. The display positioning algorithm 108 corrects the image data 70 to compensate for the movement of the viewer's head. The display positioning algorithm 108 provides the corrected image data 70 to the stereoscopic analyzer 116 for display to the viewer.

[0069] The head unit camera 102 continuously captures images as the viewer moves their head. The image processing system 114 analyzes the images by identifying the images of objects within the images. The image processing system 114 analyzes the movement of the objects and determines the pose position of the head-mountable frame 86. The image processing system 114 provides the pose position to the display adjustment algorithm 112. The display adjustment algorithm 112 uses the pose position to further refine the installation value that the display adjustment algorithm 112 provides to the display positioning algorithm 108. The display positioning algorithm 108 thus corrects the image data 70 based on the combination of the motion sensors within the head unit IMU 100 and the images captured by the head unit camera 102.

[0070] The catheter integration system 110 may detect the location of the tip 78 of the catheter 26 before a viewer inserts the tip 78 into the patient's body 30. The viewer then inserts the tip 78 into the patient's body 30. The tip 78 then becomes invisible to the viewer. The catheter IMU 80 provides a signal indicating the overall movement of the tip 78 to the catheter integration system 110. The catheter integration system 110 can thus track the position of the tip 78 using the motion sensors within the catheter IMU 80. Unlike the static image data 70, the position of the tip 78 changes over time. The catheter integration system 110 provides the position of the tip 78 to the stereoscopic analyzer 116. The position of the tip 78 can be dynamic in that it changes over time and moves in three dimensions. The stereoscopic analyzer 116 locates the tip 78 within the left and right image datasets that are inserted into the left and right projectors 88A and 88B. The viewer can thus see the location of the tip 78 within the left and right images 140A and 140B. The location of the tip 78 varies slightly within the left and right images 140A and 140B so that the viewer perceives the location of the tip 78 in three dimensions. The rendering of the location of the tip 78 as provided by the left and right images 140A and 140B changes over time as the tip 78 passes through the patient's body 30. Such movement of the location of the tip 78 as a rendering changes in three dimensions, i.e., as it moves left, right, up, down, forward, backward, etc., so that the viewer perceives the rendering of the tip 78 in three dimensions.

[0071] The catheter camera 82 continues to capture video data and provides the video data to the catheter integration system 110. The catheter integration system 110 provides the video data to the stereoscopic analyzer 116. The stereoscopic analyzer 116 places the video data at a fixed location within the viewer's view until or unless a user interaction event indicating that the location should change is detected. The video data changes over time as different images are captured by the catheter camera 82.

[0072] The visual algorithm 94 is a set of instructions stored together with the data storage unit 24 on a computer-readable medium. The set of instructions is executable by a processor to perform the method described above. The computer-readable medium storing the visual algorithm 94 may be located on a belt pack worn by the viewer.

[0073] FIG. 4 illustrates in more detail the components of the patient viewing system 20, particularly the components of the catheter integration system 110 and their relationships with the catheter IMU 80, the catheter camera 82, and the stereo analyzer 116 within the tip 78.

[0074] The catheter integration system 110 includes a catheter tracking system 150, a past path calculator 152, a mesh generator 154, a predicted path calculator 156, a video data receiving unit 158, and a catheter display integrator 160. The catheter tracking system 150 is connected to the catheter IMU 80. The catheter tracking system 150 calculates the position of the tip 78 based on the movement detected by the catheter IMU 80. The catheter IMU 80 includes several tip tracking devices including several gyroscopes and accelerometers for tracking its movement in six degrees of freedom. The catheter tracking system 150 stores the current position of the tip 78 as the position 162 in the data storage unit 24. The catheter tracking system 150 continues to monitor the catheter IMU 80, calculate the current position of the tip 78, and store the current position of the tip 78 as the current position 162 in the data storage unit 24.

[0075] The catheter display integrator 160 receives the current position 162 from the data storage unit 24 and provides the current position 162 to the stereo analyzer 116. The stereo analyzer 116 displays the current position 162 of the tip 78 to the viewer as a rendering such that the position of the tip 78 can be seen by the viewer in three dimensions.

[0076] The past path calculator 152 reads all positions 162 at all instants from the data storage unit 24. The past path calculator 152 calculates the past path of the tip 78 in three dimensions and stores the past path as the past path 164 in the data storage unit 24. The catheter display integrator 160 receives the past path 164 from the data storage unit 24 and provides the past path 164 to the stereoscopic analyzer 116. The stereoscopic analyzer 116 displays the past path 164 to the viewer as a three-dimensional rendering.

[0077] The mesh generator 154 reads the past path 164 from the data storage unit and generates a three-dimensional mesh around the past path 164. The mesh generator 154 then stores the mesh as the mesh 166 in the data storage unit 24. The catheter display integrator 160 reads the mesh 166 from the data storage unit 24 and provides the mesh 166 to the stereoscopic analyzer 116. The stereoscopic analyzer 116 displays the mesh 166 to the viewer. In some embodiments, the stereoscopic analyzer 116 generates a three-dimensional rendering of the mesh 166 that overlays the past path 164.

[0078] The predicted path calculator 156 reads all positions 162 of the tip 78 from the data storage unit 24 and calculates the future path of the tip 78 based on the positions 162 read from the data storage unit 24 and past positions. The predicted path calculator 156 then stores the future path as the future path 168 in the data storage unit 24. The catheter display integrator 160 reads the future path 168 from the data storage unit 24 and provides the future path 168 to the stereoscopic analyzer 116. The stereoscopic analyzer 116 displays the future path 168 to the viewer as a three-dimensional rendering.

[0079] The video data receiving unit 158 receives live video from the catheter camera 82. The video data receiving unit 158 provides the live video data to the catheter display integrator 160. The catheter display integrator 160 provides the live video data to the stereoscopic analyzer 116. The stereoscopic analyzer 116 displays the live video data to the viewer. The live video data is a two-dimensional display that is displayed to the viewer at a predetermined distance within a three-dimensional space. The catheter display integrator also integrates the mesh 166 with the video data from the video data receiving unit 158 so that the mesh 166 is displayed on the video data. As the video data changes with the changing position of the catheter 26 within the patient's body 30, the mesh 166 also changes accordingly.

[0080] FIG. 5 illustrates the use of the patient viewing system 20 as described above by a viewer 172 in the form of a surgeon using the catheter 26 as a bronchoscope for the purpose of examining a body part 174 including segmental bronchi within a patient's lung, according to some embodiments.

[0081] The viewer 172 can view the patient's body 30 through the left and right waveguides 90A and 90B. The body part 174 is inside the patient's body 30, and thus the viewer cannot see the actual (i.e., physical) body part 174.

[0082] The viewer 172 can also see a three-dimensional rendering 176 based on the image data 70 as described above. In certain embodiments, the rendering 176 is located adjacent to the patient's body 30. The rendering 176 is included in the figure to show where the viewer 172 perceives the rendering 176 relative to the patient's body 30, but it should be understood that from the perspective of the reader of this document, the rendering 176 does not exist in the real world. The inset 180 shows that the viewer 172 can see a three-dimensional rendering 182 of the body part 174 as part of the rendering 176.

[0083] The viewer 172 inserts the tip 78 of the catheter 26 into the patient's mouth. The viewer 172 then advances the tip 78 into the body part 174. The location of the tip 78 is monitored in cases where the times are closely spaced as described above, and its past path is stored in three dimensions. The sampling time can vary depending on the use case, and optimizations such as capturing data only while the endoscope is inside the patient's body or only after the user activates the "start recording / sampling" feature are possible considerations. The inset 184 shows that the rendering 176 includes a rendering 186 of the location of the tip 78 in three dimensions and a rendering 188 of the past path of the tip 78 in three dimensions. The renderings 182, 186, and 188 may be simultaneously displayed to the viewer 172 such that the renderings 186 and 188 within the rendering 182 are visible to the viewer.

[0084] FIG. 6 is a top plan view showing the location of the viewer 172 relative to the patient's body 30 according to some embodiments, further illustrating the location of the rendering 176 within the view of the viewer 172. The rendering 176 may be placed at any position relative to the patient's body 30 based on user preference, pre-programmed default settings, or any other suitable means. The specific relative location of the patient's body 30 to the rendering 176 in FIG. 6 is for illustrative purposes only and should not be considered limiting in any way.

[0085] FIG. 7 illustrates a view 192 as seen by the viewer 172 of FIG. 6 according to some embodiments. The viewer 172 can see the patient's actual body 30 and the rendering 176. The view 192 further includes a live video based on the video data captured by the catheter camera 82 of FIG. 4. The view 192 further shows a mesh 196 overlaid on the video 194. The mesh 196 is the display of the mesh 166 of FIG. 4.

[0086] In FIG. 8, according to some embodiments, viewer 172 moved counterclockwise around patient's body 30 and rotated his head counterclockwise to keep looking at patient's body 30. Display adjustment algorithm 112 detected the movement of viewer 172's head and accordingly adjusted the position of rendering 176 so that rendering 176 appeared to remain stationary with respect to patient's body 30 within viewer 172's view.

[0087] In FIG. 9, according to some embodiments, patient's body 30 rotated clockwise with respect to FIG. 7. Rendering 176 also rotated clockwise so as to remain stationary with respect to patient's body 30. However, the location of live video 194 did not change from view 192 of FIG. 7 to view 192 of FIG. 9. Viewer 172 thus sees live video 194 and mesh 196 in the same location, and these components do not move in response to the movement of viewer 172's head. Viewer 172 can thus view patient's body 30 and rendering 176 from different sides and angles without losing sight of live video 194 and mesh 196. The purpose of mesh 196 may be to assist the viewer in guiding tip 78 of catheter 26 when viewer 172 inserts tip 78 into the passage within body part 174a for the second time after the mesh has been generated or during removal of the catheter as the catheter moves through the same path in the opposite direction. Some embodiments may have different viewing configurations (e.g., mesh 196, live video 194, rendering 176) for virtual content where some or all of the virtual content is fixed with respect to real-world coordinates or fixed with respect to the viewer.

[0088] Figure 10 shows components of the rendering 176 presented to a viewer that are too small to be seen in the views of FIGS. 7 and 9 according to some embodiments. The viewer 172 can see renderings 182, 186, and 188 of the body part 174, the tip 78, and the past path of the tip. The viewer can also see a 3D rendering of the mesh 196. The mesh 196 is shown separately from the renderings 182, 186, and 188 for illustrative purposes, but it should be understood that the mesh 196 can overlay the rendering 182 of the body part 174.

[0089] As shown in FIGS. 11 and 12, in some embodiments, the viewer 172 can see the mesh 196 in two places, namely, as part of the rendering 176 (FIG. 11), and overlaid on the live video 194 (FIG. 12).

[0090] Figure 13 illustrates the functionality of the predicted path calculator 156 of FIG. 4 according to some embodiments. The graph illustrates the rotation of the tip 78 of the catheter 26 about the x, y, and z axes over time. The rotation about each axis may be analyzed over a short amount of time to determine a first amount 200 of movement from a first position 202 to a second position 204. The first amount 200 of movement may be used to calculate a prediction of the future movement of the tip 78.

[0091] Figure 14 illustrates the tip 78 at the first position 202 according to some embodiments. In FIG. 15, the tip 78 has moved from the first position 202 to the second position 204 by the first amount 200 according to some embodiments. The first amount 200 shown in FIG. 15 is the sum of all vectors of all movements about all axes and in all translational directions.

[0092] In FIG. 14, when the tip 78 is further inserted into the body portion 174, it can be assumed that the direction the tip 78 would follow is along the extension 206 of the lumen 76. FIG. 16 shows the extension 208 of the lumen 76 after the lumen 76 has moved by a first amount 200, according to some embodiments. The viewer will typically not advance the tip 78 along the path of the extension 208 because the tip 78 would contact the body portion 174 and cause injury. Instead, the viewer 172 will preferably follow a path 210 to avoid injury. The path 210 is displaced by a second amount 212 from a second position 204 to a third position 214. The first amount 200 and the second amount 212 are measured in the same direction for ease of reference.

[0093] FIG. 17 illustrates the actual path 218 that the tip 78 will likely follow, according to some embodiments. The path 218 exits the path 208 and approaches the path 214 as the viewer further inserts the tip 78 into the body portion 174. The stereoscopic analyzer 116 displays the path 218 in three dimensions to the viewer 172.

[0094] FIG. 18 shows a graphical representation of a machine in an exemplary form of a computer system 900, within which a set of instructions for causing the machine to execute any one or more of the methodologies discussed herein can be executed. In alternative embodiments, the machine may operate as a stand-alone device or may be connected (e.g., networked) to other machines. Further, although only a single machine is illustrated, the term "machine" shall also be construed to include any collection of machines that individually or jointly execute a set of instructions (or multiple sets) for performing any one or more of the methodologies discussed herein.

[0095] The exemplary computer system 900 includes a processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both) that communicates with each other via a bus 908, a main memory 904 (e.g., a dynamic random access memory (DRAM) such as a read-only memory (ROM), a flash memory, a synchronous DRAM (SDRAM), or a Rambus DRAM (RDRAM)), and a static memory 906 (e.g., a flash memory, a static random access memory (SRAM), etc.).

[0096] The computer system 900 may further include a disk drive unit 916 and a network interface device 920.

[0097] The disk drive unit 916 includes a machine-readable medium 922 in which one or more sets of instructions 924 (e.g., software) that implement one or more of the methodologies or functions described herein are stored. The software may also reside, in whole or in part, within the main memory 904 and / or within the processor 902 during execution thereof by the computer system 900, the main memory 904, and the processor 902, which also constitutes a machine-readable medium.

[0098] The software may also be transmitted or received via the network 928 via the network interface device 920.

[0099] The computer system 900 includes a laser driver chip 950 that is used to drive a projector to generate a laser beam. The laser driver chip 950 includes its own data storage and its own processor 962.

[0100] The machine-readable medium 922, although shown as a single medium in the exemplary embodiment, the term "machine-readable medium" should be construed to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable medium" should also be construed to include any medium that is capable of storing, encoding, or carrying a set of instructions for machine execution and causing a machine to perform any one or more of the methodologies of the present invention. The term "machine-readable medium" shall, therefore, be construed to include, without limitation, solid-state memory, optical and magnetic media, and carrier wave signals.

[0101] The implementation described above uses the CT scanner 22 to scan the body part 174. The CT scanner has a transmitter in the form of an X-ray transmitter and a receiver in the form of an X-ray detector, and transmits and receives waves in the form of X-ray waves. It is conceivable that other scanning devices that use other transmitters and receivers to transmit and detect different waves could be used. For example, an ultrasound system uses an acoustic transmitter for transmitting acoustic waves and an acoustic receiver for receiving acoustic waves. A vision system may include a light source inserted into the body part for transmitting light waves and a camera located within the body part for capturing the light waves reflected from the body part.

[0102] However, because a CT scanner provides extremely highly detailed raw data of a body part in three dimensions and such data can be easily converted using an image generation unit to generate three-dimensional image data, a CT scanner is preferred over other scanning devices. CT data also has the advantage that it can include data regarding specific substances, materials, and the density of materials. The implementation being described shows a rendering 176 placed adjacent to the patient's body 30 within the viewer 172's view 192. It is also possible to consider aligning the rendering with the patient's body 30 such that the rendering of the body part is where the actual body part is and the rendering of the tip of the catheter is where the actual position of the tip of the catheter is.

[0103] Aspects of the present invention can also be implemented without a catheter. For example, it is possible to consider scanning a patient's body to determine growth and a viewer using a display system to overlay a three-dimensional rendering of the growth onto the patient's actual body. In this way, the viewer can "see" the growth "within" the patient's actual body.

[0104] In certain aspects and embodiments, invasive surgical instruments generally are utilized in place of, or in addition to, the catheter described. For example, referring to FIGS. 14 - 17, tip 78 may indicate the leading point of a thermal imaging camera that is coupled to the catheter to collect temperature data through the anatomical channel through which it passes. The thermal data may then be converted from the black and white thermal image and displayed to the operator as a variable color overlay through image receiving unit 106. It should be understood that not all displays need to require a common image for all users. While a first operator or observer (local or remote) may desire a display of certain information such as temperature data indicating arterial blood versus venous blood or frozen tissue, a second operator or observer may desire a display of certain information such as that indicating the position of the surgical instrument. In other words, the display of information need not be limited to an image of the position of the instrument within the patient, but may also be an image collected from the instrument. In certain thermal image embodiments, the temperature gradient can be selected per user, where a first operator may desire to distinguish organic tissue from inorganic material and set the base temperature to 98.6 degrees Fahrenheit, while a second operator may specifically intend to track the surgical instrument and set the base temperature for imaging to a pre - set temperature rather than an absolute one.

[0105] In some embodiments, an operating envelope is implemented. Returning to FIGS. 14 - 17, while navigating an anatomical channel, a particular placement of the patient can modify the operating load of the instrument. For example, if a patient image shows sensitive tissue in proximity to the instrument, the load may be adjusted proportionally. To further illustrate, if an image of a patient with atrial fibrillation shows a thin left atrial wall, an ablation catheter with a standard axial load of 0.1 N may be tented up to a range of 0.01 or 0.05 N while operating in proximity to the left atrial wall. In some embodiments, a load paradigm as a function of position, both absolute and relative to anatomical markers such as sensitive tissue, may be displayed to the operator in conjunction with the position of the instrument or an instrument image such as a collected image. Visual feedback indicating an upper limit defined for the load on the instrument as a function of position may thus be provided to inform the operator of the device's capabilities and limitations at that instant. Alternatively, an additional operator or observer may be able to immediately identify when the instrument is not in the correct position or is exceeding patient parameters for a given procedure. Such feedback has been described as visual, but the feedback may take other forms such as audio or tactile (increased resistance or friction in instrument control).

[0106] In some embodiments, observable motion artifacts provide position adjustment to the instrument. The head unit camera 102 may image patient position data and provide real-time position adjustment to the instrument. In some embodiments, the patient's breathing rhythm is observed and the instrument control is modified to adapt to the respiratory state. The head unit camera 102 may align position data by reference markers in the operating environment or in a fixed machine of known dimensions, compare chest positions during exhalation and inhalation, and correlate dimensional changes. In some embodiments, as the lungs expand during inhalation and the surrounding biological structures react and compress, the instrument motion may correspondingly decelerate, and then, as the lungs contract during exhalation, the normal operating parameters may be restored, or the instrument may move absolutely within the anatomical channel but be stable relative to that anatomical reference, such that the x-y-z adjustment may be made to match the up and down of the thoracic cavity. Other observed stimuli may also provide motion adjustment, and the heart rate and the expected contraction / expansion of blood vessels may provide position updates to the instrument.

[0107] In some embodiments, feedback control of patient activity (e.g., breathing rhythm, heart rate) is collected at three separate times by the head unit camera 102 at times T1 to T2 to T3, and the images are analyzed at time T4 to determine changes in patient position (due to patient movement, their breathing rhythm, heart rate, etc.). An instrument operably coupled to the head unit camera 102 is given a control input at time T0 (i.e., before or at the same time as T1), and an adjustment of the control input based on the observed changes in patient position over time from T1 to T3 is made at time T5. In some embodiments, the change in the control input is the measured rate of change in patient position.

[0108] Figure 19 illustrates an exemplary relationship between the measured patient position data and the control feedback adjustment. As depicted, the theoretical y-direction change in patient position due to chest expansion arising from respiration is represented by curve 1901. The observed patient position data is collected at times T1, T2, and T3, corresponding to chest positions y1, y2, and y3, respectively. Preferably, the three collection times are created to provide a trend analysis for target measurement. For example, the average human respiration rate is 15 breaths per minute, allocating approximately 2 seconds to any given inhalation or exhalation. Based on the measured inhalation, at least three measurements are taken to provide a trend analysis to avoid corrections to the instrument position applied during exhalation. Considering the average human respiration rate, a time interval between measurements of at least 0.667 seconds or 1.5 Hz is preferred. Such a frequency need not be the time between each of T0 and T5, and it is preferred to apply the associated actions at T4 and T5 as quickly as possible.

[0109] Returning to FIG. 19, the change in positions y1 - y2 - y3 indicates to the system during the analysis at T4 that the inhalation is complete near T3. Thus, the feedback control at T5 may not provide an adjustment to avoid applying an "inhalation correction" during exhalation, or may provide a negative y-position adjustment based on the time relationship of T5 to T3 and the estimated y-position change. For example, if the time between T3 and T5 is similar to the time between T2 and T3 and the system recognizes from a change such as being between y1 and y2 compared to y2 and y3 that the patient is undergoing a change in the y direction, the adjustment may be less than or equal to the change measured from T2 to T3. This can be depicted as the following logical relationship. If (T5 - T3 = T3 - T2) ∩ (y3 - y2) < (y2 - y1), then the correction at T5 < (y3 - y2)

[0110] Although an exemplary embodiment has been described and shown in the accompanying drawings, such embodiments are merely illustrative and not limitations of the present invention, and it should be understood that the present invention is not limited to the specific structures and arrangements shown and described, as modifications may occur to those skilled in the art.

Claims

1. A patient visualization system, wherein the patient visualization system comprises: A catheter having a tip for insertion into a patient's body; Means for detecting movement of the tip; Means for receiving a measurement value output from the means for detecting movement of the tip based on the movement of the tip; Means for determining the position of the tip based on the measurement value; Means for storing the position of the tip; Means for determining a left image dataset and a right image dataset based on the position of the tip; Means for generating the light in a pattern representing the position of the tip by projecting the left image dataset and the right image dataset as light respectively; Means for guiding the light to the retina of the viewer's left eye and the retina of the viewer's right eye so that the viewer can see the position of the tip, wherein the left image dataset and the right image dataset are different from each other so as to give the viewer a perception of 3D rendering; Means for generating a 3D mesh; Means for storing the 3D mesh; And comprising; The means for guiding the light to the retina of the viewer's left eye and the retina of the viewer's right eye so that the viewer can see the position of the tip further guides the light to display the 3D mesh on the retina of the viewer's left eye and the retina of the viewer's right eye together with the past path of the tip and the body part in the body and the position of the tip, whereby the viewer visually recognizes that the rendering of the 3D mesh overlays the rendering of the past path of the tip and the 3D rendering of the body part in the body, and the 3D rendering of the body part includes the rendering of the past path of the tip. A patient visualization system.

2. The patient visualization system further comprises a head-mountable frame mounted on the viewer's head; The means for guiding the light to the retina of the viewer's left eye and the retina of the viewer's right eye comprises a tubular optical waveguide fixed to the head-mountable frame. The patient visualization system according to claim 1.

3. The tubular optical waveguide is a transparent optical waveguide positioned in front of the viewer's left and right eyes. The patient visualization system according to claim 2.

4. The patient visualization system comprises: Means for detecting movement of the head-mountable frame; means for calculating installation values used to place a patient's body part within the views of the viewer's left and right eyes based on the movement of the head-mountable frame; means for correcting the position of the body part within the views of the left and right eyes based on the installation values; The patient viewing system according to claim 2, further comprising: **Claim 5** The patient viewing system according to claim 4, wherein the means for detecting movement of the head-mountable frame comprises a motion sensor of a head unit inertial measurement unit (IMU) mounted on the head-mountable frame. **Claim 6** The means for detecting movement of the head-mountable frame comprises a head unit camera mounted on the head-mountable frame, The head unit camera detects movement of the head-mountable frame by taking an image of an object within the view of the head unit camera. The patient viewing system according to claim 4. **Claim 7** The patient viewing system according to claim 6, further comprising means for detecting the pose position of the head-mountable frame by analyzing the image. **Claim 8** The patient viewing system is means for activating a transmitter to generate a forward wave in a body part within the body, the body part reducing the output of the forward wave; means for detecting a return wave from the body part, the return wave being the forward wave with the output reduced by the body part, and the return wave from the body part being detected in response to the forward wave generated by the transmitter; means for receiving raw data of the return wave; means for storing the raw data in a data storage unit; means for generating image data representing an image by processing the raw data of the return wave; means for storing the image data in the data storage unit; means for receiving the image data from the data storage unit; means for combining the position of the tip and the image data, the pattern including a pattern representing the image data and the position of the tip; The patient viewing system according to claim 1, further comprising: **Claim 9** The transmitter is fixed to the rotor and is an X-ray transmitter that transmits X-ray waves. The means for detecting the feedback wave is an X-ray detector fixed to the rotor so as to detect X-ray waves. The movement of the platform relative to the base enables the movement of the patient relative to the plane extending from the X-ray transmitter to the X-ray detector. The patient visualization system according to claim 8.

10. The patient visualization system further includes means for storing the past path of the tip in the data storage unit and the means for guiding the light to the retina of the viewer's left eye and the retina of the viewer's right eye so that the viewer can see the position of the tip further guides the light to display the past path of the tip together with the position of the tip on the retina of the viewer's left eye and the retina of the viewer's right eye. The patient visualization system according to claim 1.

11. The patient visualization system further includes means for capturing video data of the area in front of the tip and means for receiving the video data and the means for guiding the light to the retina of the viewer's left eye and the retina of the viewer's right eye so that the viewer can see the position of the tip further guides the light to display a live video on the retina of the viewer's left eye and the retina of the viewer's right eye based on the video data. The patient visualization system according to claim 1.

12. The patient visualization system further includes means for calculating the future path of the tip based on the position of the tip and the means for guiding the light to the retina of the viewer's left eye and the retina of the viewer's right eye so that the viewer can see the position of the tip further guides the light to display the future path on the retina of the viewer's left eye and the retina of the viewer's right eye. The patient visualization system according to claim 1.

13. The future path is calculated based on the movement of the tip. The patient visualization system according to claim 12.

14. The movement of the tip is a movement of a first amount from a first position to a second position in a selected direction, and the future path is a path that moves a second amount from the second position to a third position in the selected direction. The patient visualization system according to claim 13.

15. The patient visualization system further includes means for performing an invasive procedure. The patient visualization system according to claim 1.

16. The patient visualization system means for observing patient activity including at least one of respiratory rhythm and heartbeat, and means for enabling the viewer to perform position adjustment with respect to the tip based on the patient activity The patient viewing system according to claim 1, further comprising.

17. The patient viewing system further comprises means for collecting images of the patient activity at three separate times at times T1 to T2 to T3, the images being analyzed at time T4 to determine changes in patient position, and adjustment of the control input of the catheter based on the changes in patient position observed over times T1 to T3 is performed at time T5. The patient viewing system according to claim 16.

18. When (T5 - T3 = T3 - T2) ∩ (y3 - y2) < (y2 - y1), the correction at T5 < (y3 - y2), where y1 represents the patient position corresponding to time T1, y2 represents the patient position corresponding to time T2, and y3 represents the patient position corresponding to time T3. The patient viewing system according to claim 17.

Images