Patient Observation System

Abstract

A method for observing a patient is described, the method comprising inserting a catheter for health process navigation. A CT scan is performed on a body part of the patient. Raw data from the CT scan is processed to create three-dimensional image data, and the image data is stored in a data storage device. A projector receives light generated in a pattern representing the image data, and a waveguide guides the light to a retina of an eye of an observer, and light from an outer surface of the body is sent to the retina of the eye so that the observer sees the outer surface of the body enhanced with the body part rendered with the processed data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 62 / 721,516 filed on August 22, 2018 and U.S. Provisional Patent Application Serial No. 62 / 771,534 filed on November 26, 2018, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to patient observation systems. Background Art

[0004] Endoscopic catheters are used by surgeons, physicians, and other observers to capture images of body parts within a patient's body. For example, a bronchoscope is an endoscopic catheter used to examine segmental bronchi. The catheter also has other functions, such as other endoscopic viewing functions, to treat disease or perform surgical procedures.

[0005] Because the tip of the catheter is inserted into the patient's body with minimal or no invasive surgery, the position of the tip of the catheter cannot be seen by the naked eye by the observer. To assist the observer, the screen provides a combination of images. These images typically include computed tomography (CT) images generated before and during surgery, and live video data from a catheter camera in the catheter tip. These images are typically provided in different quadrants of the display.

[0006] As far as the bronchi are concerned, all bronchial tunnels look identical. As a result, observers often guess where the bronchus is currently navigating. Such observers often perform multiple CT scans as they push the bronchoscope through the bronchi to determine their location in the lungs. Summary of the invention

[0007] The present invention provides a patient observation system, which includes: a catheter having an inner cavity and a tip, a tip tracking device that detects movement of the tip, a left projector and a right projector, a left optical waveguide and a right optical waveguide connected to the left projector and the right projector, a processor, a computer-readable medium connected to the processor, a data storage on the computer-readable medium, and an instruction set stored on the computer-readable medium and executable by the processor. The instruction set may include: 1) a catheter tracking system that is connected to the tip tracking device and receives a measurement based on the movement detected by the tip tracking device, and determines the position of the tip based on the measurement and stores the position of the tip in the data storage, and 2) a stereo analyzer connected to the data storage to receive image data, the stereo analyzer determines a left image data set and a right image data set, the left projector and the right projector project the left image data set and the right image data set, respectively, the left image data set and the right image data set are different from each other to provide a three-dimensional rendering perception to the observer.

[0008] The present invention also provides a method for observing a patient, the method comprising: inserting the tip of a catheter into the patient's body; detecting the movement of the tip with a tip tracking device; receiving measurements based on the movement detected by the tip tracking device; determining the position of the tip based on the measurements; storing the position of the tip; determining a left image dataset and a right image dataset based on the position of the tip; using a left projector and a right projector that project the left image dataset and the right image dataset, respectively, to generate light in a pattern representing the position of the tip as light; and directing the light to the retinas of the left and right eyes of an observer so that the observer sees the position of the tip, the left image dataset and the right image dataset being different from each other to provide the observer with a perception of three-dimensional rendering.

[0009] The present invention also provides a patient observation system, which includes: a catheter having an inner cavity and a tip, a tip tracking device that detects the movement of the tip, a projector, an optical waveguide connected to the projector, a processor, a computer-readable medium connected to the processor, a data storage on the computer-readable medium, and an instruction set stored on the computer-readable medium and executable by the processor. The instruction set may include: 1) a catheter tracking system that is connected to the tip tracking device and receives a measurement based on the movement detected by the tip tracking device, and determines the position of the tip based on the measurement, and stores the position of the tip in the data storage; 2) a past path calculator that stores the past path of the tip in the data storage; and 3) a catheter display integrator that displays the past path of the tip together with the position of the tip.

[0010] The present invention also provides a method for observing a patient, the method comprising: inserting the tip of a catheter into the patient's body; detecting the movement of the tip with a tip tracking device; receiving measurements based on the movement detected by the tip tracking device; determining the position of the tip based on the measurements; storing the position of the tip; determining a left image dataset and a right image dataset based on the position of the tip; using a left projector and a right projector that project the left image dataset and the right image dataset, respectively, to generate light in a pattern representing the position of the tip as light; and directing the light to the retinas of the left and right eyes of an observer so that the observer sees the position of the tip, the left image dataset and the right image dataset being different from each other to provide the observer with a perception of three-dimensional rendering; storing the past path of the tip in a data storage device; and displaying the past path of the tip together with the position of the tip.

[0011] The present invention also provides a patient observation system, which includes: a catheter having an inner cavity and a tip, a tip tracking device that detects the movement of the tip, a projector, an optical waveguide connected to the projector, a processor, a computer-readable medium connected to the processor, a data storage on the computer-readable medium, and an instruction set stored on the computer-readable medium and executable by the processor. The instruction set may include: 1) a catheter tracking system that is connected to the tip tracking device and receives a measurement based on the movement detected by the tip tracking device, and determines the position of the tip based on the measurement and stores the position of the tip in the data storage; 2) an expected path calculator that calculates the future path of the tip based on the position of the tip, and the catheter display integrator displays the future path, and 3) a catheter display integrator that displays the future path of the tip together with the position of the tip.

[0012] The present invention also provides a method for observing a patient, the method comprising: inserting the tip of a catheter into the patient's body; detecting the movement of the tip with a tip tracking device; receiving measurements based on the movement detected by the tip tracking device; determining the position of the tip based on the measurements; storing the position of the tip; determining a left image dataset and a right image dataset based on the position of the tip; using a left projector and a right projector that project the left image dataset and the right image dataset, respectively, to generate light in a pattern representing the position of the tip as light; guiding the light to the retinas of the left and right eyes of an observer so that the observer sees the position of the tip, the left image dataset and the right image dataset being different from each other to provide the observer with a perception of three-dimensional rendering; calculating the future path of the tip based on the position of the tip; and displaying the future path of the tip together with the position of the tip.

[0013] The present invention also provides a patient observation system, which includes: a transmitter; an energy source, which is connected to the transmitter to activate the transmitter, the patient's body is positionable relative to the transmitter so that the transmitter generates a forward wave at a body part inside the body; a receiver, which is positionable relative to the body to detect a return wave from the body part, the return wave from the body part is a response to the forward wave generated by the transmitter, a processor; a computer-readable medium, which is connected to the processor; a data storage, which is located on the computer-readable medium; and an instruction set, which is stored on the computer-readable medium and can be executed by the processor. The instruction set may include: 1) a raw data receiving unit that receives raw data of a return wave detected by a receiver and stores the raw data in a data storage device; 2) an image generating unit that is connected to the data storage device to process the raw data of the return wave to create image data representing an image, and stores the image data in the data storage device; 3) an image data receiving unit that receives the image data from the data storage device; 4) a projector that is connected to the image data receiving unit to receive the image data, the projector generating light in a pattern representing the image data, and 5) an optical waveguide connected to the projector that guides light to the retina of an observer's eye, while light from the outer surface of the body is sent to the retina of the eye, so that the observer sees the outer surface of the body enhanced by rendering the body part.

[0014] The present invention also provides a method for observing a patient, the method comprising: activating a transmitter to generate a forward wave at a body part within the body; detecting a return wave from the body part with a receiver, the return wave from the body part being a response to the forward wave generated by the transmitter; receiving raw data of the return wave detected by the receiver; storing the raw data in a data storage device; processing the raw data of the return wave to create image data representing an image; storing the image data in a data storage device; receiving the image data from the data storage device; generating light in a pattern representing the image data; and guiding the light to the retina of an observer's eye, while light from the outer surface of the body is sent to the retina of the eye, so that the observer sees the outer surface of the body enhanced by rendering the body part. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention is also described by way of example with reference to the accompanying drawings, in which:

[0016] Figure 1 is a block diagram of a patient observation system according to an embodiment of the present invention;

[0017] Figure 2 is a partial perspective view and partial block diagram of a CT scanner, a data receiving unit, an image generating unit, and a data storage device forming part of a patient viewing system according to some embodiments;

[0018] Figure 3 is a partial perspective view and partial block diagram of a display system, catheter, and data storage device forming part of a patient viewing system according to some embodiments;

[0019] Figure 4 According to some embodiments, Figure 3 A block diagram of a catheter integrated system showing a portion of the system, and also showing a catheter;

[0020] Figure 5 is a perspective view showing a viewer in the form of a surgeon seeing a rendering of the patient's body and body parts within the patient, and also seeing a rendering of the catheter tip and the past path of the tip, according to some embodiments;

[0021] Figure 6 According to some embodiments Figure 5 Top view of;

[0022] Figure 7 is the view seen by an observer according to some embodiments;

[0023] Figure 8 is similar to Figure 6 's view;

[0024] Fig. 9 is a diagram showing how to modify the patient's body within a view and render similar to Figure 7 's view;

[0025] Fig.10 According to some embodiments, the Figure 7 and Fig. 9 A rendering shown to the viewer in ;

[0026] Fig.11 According to some embodiments, Figure 7 and Fig. 9 an enlarged view of a portion of the view as seen by an observer in;

[0027] Fig.12 According to some embodiments, Figure 7 and Fig. 9 A magnified view of the live video data and the grid as seen by an observer in the view;

[0028] Fig.13 is a graph illustrating movement of a tip of a catheter in three degrees of freedom and movement by a first amount from a first position to a second position according to some embodiments;

[0029] Fig.14is a view of a tip of a catheter in a first position according to some embodiments;

[0030] Fig.15 is a view of a catheter in a second position according to some embodiments;

[0031] Fig.16 is a diagram similar to that of FIG. 1 showing a path extending through a lumen and a path not extending through a lumen to avoid injury according to some embodiments. Fig.16 's view;

[0032] Fig.17 is a diagram similar to that showing the calculated future path of a tip according to some embodiments. Fig.16 a view in which the future path of the tip is displayed as a three-dimensional rendering to the viewer;

[0033] Fig.18 is a block diagram of a machine in the form of a computer that may find application in the system of the present invention according to one embodiment of the present invention; and

[0034] Fig.19 The timing relationship of adjusting the input of a medical device based on the position change observed by the patient according to one embodiment of the present invention is shown. DETAILED DESCRIPTION

[0035] In the attached figure Figure 1 A patient viewing system 20 is shown including a CT scanner 22 , a data storage 24 , a catheter 26 , and a display system 28 , in accordance with an embodiment of the present invention.

[0036] The data storage 24 is connected to the CT scanner 22. Raw data from the CT scanner 22 may be stored in the data storage 24. The data storage 24 also stores image data based on the raw data.

[0037] The display system 28 is connected to the data storage 24 to be able to retrieve image data from the data storage 24. The catheter 26 is connected to the display system 28 so that the display system 28 can retrieve measurement and video data from the catheter 26 for further processing or display to an observer.

[0038] In use, a patient is positioned at a station 32 of a CT scanner 22. The patient's body 30 is scanned with the CT scanner 22 to obtain raw data which the CT scanner 22 stores in a data memory 24. The raw data is then processed to obtain 3D image data.

[0039] The patient is transferred from a workstation 32 at the CT scanner 22 to a workstation 34 at the display system 28. An observer uses the display system 28 to observe the patient's body 30. The display system 28 also retrieves image data from the data storage 24. The observer uses the display system 28 to observe images in the form of a 3D rendering of the patient's body 30. The observer inserts the catheter 26 into the body 30. The display system 28 retrieves data from the tip of the catheter 26 for further processing or display to the observer.

[0040] Figure 2 Components of a patient viewing system 20 are shown, including a CT scanner 22 , a data storage 24 , an energy source 36 , a data receiving unit 38 , and an image generating unit 40 , according to some embodiments.

[0041] CT scanner 22 includes a base 42 , a platform 44 , a rotor 46 , an x-ray transmitter 48 , and a plurality of x-ray detectors 50 .

[0042] The platform 44 is secured to the base 42 by a mechanism (not shown) that allows translational movement of the platform 44 relative to the base 42. An actuator such as a stepper motor (not shown) is operable to cause translational movement of the platform 44 relative to the base 42.

[0043] The rotor 46 has an opening 52. The x-ray transmitter 48 is fixed to the rotor 46 on one side of the opening 52, and the x-ray detector 50 is fixed to the rotor 46 on the opposite side of the opening 52. The rotor 46 is mounted to the base 42 about the platform 44. The platform 44 moves relative to the opening 52 during its translational movement. A motor (not shown) is connected between the base 42 and the rotor 46 and is operable to rotate the rotor 46 about the platform 44.

[0044] The energy source 36 may be connected to the x-ray transmitter 48 via a switch 54. The x-ray detector 50 may be connected to the data receiving unit 38. The data receiving unit 38 may be a software unit residing on a computer readable medium of a computer. The data storage 24 resides on a computer readable medium. The computer readable medium may be a single computer readable medium or may be divided within a personal computer or multiple personal computers connected to each other on a network. The data receiving unit 38 is connected to the data storage 24 directly or through a network.

[0045] The image generation unit 40 may be a computer program residing on a computer readable medium.The image generation unit 40 is connected to the data storage 24 directly or through a network.

[0046] In use, an operator of the CT scanner 22 places the patient's body 30 on the platform 44. The motor connected between the base 42 and the rotor 46 is then turned on, causing the rotor 46 to rotate about the platform 44 and the patient's body 30 in a direction 58. The operator also turns on the motor, which moves the platform 44 in a translational direction relative to the base 42, causing the platform 44 to move relative to the rotor 46 in a direction 60. The operator then connects the switch 54 between the energy source 36 and the x-ray transmitter 48 to activate the x-ray transmitter 48. The x-ray transmitter then generates a forward x-ray wave 62.

[0047] The patient's body 30 is positioned relative to the x-ray transmitter 48 so that the forward x-ray wave 62 penetrates the body 30 to a body part (not shown) within the body 30. For the purposes of this example, the body part being scanned is the patient's lungs. The lungs have many bronchial tubes through which a catheter can be advanced. The catheter may also be advanced through hollow passages in the heart, arteries, and veins of the blood circulation system, etc. The system described herein may also be used to observe internal body parts without the need for visual, surgical, or interventional use of a catheter, such as observing growths within the abdomen, analyzing the internal function of a knee, etc. The body part reduces the energy of the forward x-ray wave 62. Different materials within the body part reduce the energy by different amounts. One of the x-ray detectors 50 is positioned relative to the body 30 to detect a return x-ray wave 64 from the body part. The return x-ray wave 64 from the body part is detected in response to the forward x-ray wave 62 and is essentially the forward x-ray wave 62 having a reduced power due to being reduced in power by the body part. A further forward x-ray wave 66 is also shown. Further x-ray waves are generated between forward x-ray waves 62 and 66 and are detected by corresponding ones of x-ray detectors 50. In this way, return x-ray waves are received from different portions of the body part.

[0048] The x-ray transmitter 48 and x-ray detector 50 rotate with the rotor 46 around the body part within the patient's body 30. In this way, the body part can be scanned from different angles to create two-dimensional "slices" of the anatomical structure. A CT scan can show bones, organs, soft tissue. Subsequent slices are acquired by moving the platform 44 in the direction 60. Thus, each slice represents two-dimensional data, and the slices together represent three-dimensional data of the body part.

[0049] The data receiving unit 38 receives raw data of the return x-ray waves 64 from the x-ray detectors 50. The raw data includes the angle of the x-ray transmitter 48 relative to the body part within the patient's body 30, the energy detected by each x-ray detector 50, the position of each x-ray detector 50, and the time-sequential correlation between the position of the platform 44. The data receiving unit 38 stores the raw data as raw data 68 of the return x-ray waves detected by the x-ray detectors 50.

[0050] When enough raw data 68 of the body part is collected, the operator turns off the switch 54 and stops the platform 44. Then, the operator stops the rotor 46 and removes the patient from the platform 44.

[0051] The image generation unit 40 retrieves raw data 68 from the data storage 24. The image generation unit 40 generates image data based on the raw data 68. The image data includes a three-dimensional rendering of the body part. The image generation unit 40 then stores the image data as image data 70 in the data storage 24. The data storage 24 may be a single data storage or may be distributed between platforms, and thus, the raw data 68 and the image data 70 may be located in a single data storage within a personal computer, or in multiple data storages within multiple personal computers.

[0052] Figure 3 Components of patient viewing system 20 are shown in greater detail according to some embodiments, and data storage 24 (holding image data 70 ), catheter 26 , and display system 28 are shown.

[0053] The catheter 26 includes a lumen 76 and a tip 78 attached to the end of the lumen 76. The lumen is an elongated member (e.g., a cavity of a tubular portion) that forms most of the length of the catheter 26. The lumen 76 includes a mechanism (not shown) that is operable to move the tip 78 in at least four orthogonal directions and in all directions between the orthogonal directions. Thus, the tip 78 can be manipulated using the mechanism in the lumen 76. The lumen has a sufficiently large bore to accommodate the mechanism for manipulating the tip along with any cables and / or optical fibers that may be required to relay signals from the tip through the lumen 76 to the display system 28.

[0054] The catheter 26 also includes a catheter inertial motion unit (IMU) 80 and a catheter camera 82 fixed to the tip 78. The catheter IMU 80 can be, for example, a semiconductor chip having a plurality of measurement devices formed therein. The measurement devices include one or more gyroscopes and one or more accelerometers. Measurements from the gyroscopes and accelerometers provide data indicating the movement of the tip 78, either alone or in combination. Such movement can be tracked in six degrees of freedom, for example, translation in the x, y, and z directions and rotation around the x, y, and z axes.

[0055] The catheter camera 82 has a lens (not shown) on the side of the tip 78 opposite the lumen 76. The catheter camera 82 is positioned to capture images in the form of live video data in the area in front of the tip 78, i.e., on the side opposite the lumen 76. There may be multiple light sources and multiple cameras on different sides of the camera tip, but for ease of discussion, we assume only a single camera, such as a built-in camera and light source on the distal end of the catheter.

[0056] The display system 28 includes a head mounted frame 86, left and right projectors 88A and 88B, left and right waveguides 90A and 90B, a detection device 92, and a visual algorithm 94. The left and right projectors 88A and 88B, the left and right waveguides 90A and 90B, and the detection device 92 are fixed to the head mounted frame 86. The head mounted frame 86 is shaped to be mounted to the head of the observer. The components of the head mounted frame 86 may, for example, include a band (not shown) wrapped around the back of the observer's head.

[0057] The left projector 88A and the right projector 88B are connected to a power source. Each projector 88A or 88B has a corresponding input for providing image data to the corresponding projector 88A or 88B. The corresponding projector 88A or 88B generates light in a two-dimensional pattern and emits light therefrom when powered on. The left waveguide 90A and the right waveguide 90B are positioned to receive light from the left projector 88A and the right projector 88B, respectively. The left waveguide 90A and the right waveguide 90B are transparent waveguides.

[0058] The detection device 92 includes a head unit IMU 100 and one or more head unit cameras 102. The head unit IMU 100 includes one or more gyroscopes and one or more accelerometers. The gyroscopes and accelerometers are typically formed in a semiconductor chip and are capable of detecting movement of the head unit IMU 100 and the head mounted frame 86, including movement along three orthogonal axes and rotation around three orthogonal axes.

[0059] The head unit camera 102 continuously captures images from the environment around the head mounted frame 86. The images may be compared to each other to detect movement of the head mounted frame 86 and the viewer's head.

[0060] The visual algorithm 94 includes an image data receiving unit 106, a display positioning algorithm 108, a catheter integration system 110, a display adjustment algorithm 112, an image processing system 114, and a stereo analyzer 116. The image data receiving unit 106 is connected to the data storage 24 by a direct connection or through a network. The components of the visual algorithm 94 are linked to each other through subroutines or calls. Through such subroutines and calls, the image data receiving unit 106 is linked to the stereo analyzer 116 via the display positioning algorithm 108.

[0061] The catheter integration system 110 can be connected to the catheter IMU 80 and the catheter camera 82 through conductors in the lumen 76. One of ordinary skill in the art will appreciate that the vision algorithm 94 resides on the computing system and the catheter integration system 110 receives signals from the catheter camera 82 and the catheter IMU 80 and that the signals can be converted from analog or digital data to computer software data. The catheter integration system 110 can be connected to the stereo analyzer 116 through a subroutine or call.

[0062] The display adjustment algorithm 112 and the image processing system 114 are connected to the head unit IMU 100 and the head unit camera 102, respectively. Such connections are made through conductors and, if applicable, through converters (inverters) that convert analog or digital data to computer software data. The display adjustment algorithm 112 can be connected to the display positioning algorithm 108 through subroutines and calls. The image processing system 114 can be connected to the display adjustment algorithm 112 through calls and subroutines.

[0063] In use, the viewer mounts the head mounted frame 86 to their head. The left waveguide 90A and the right waveguide 90B are then located in front of the viewer's left eye 120A and right eye 120B.

[0064] The image data receiving unit 106 retrieves the image data 70 from the data storage 24 and provides the image data 70 to the display positioning algorithm 108. The display positioning algorithm 108 inputs the image data 70 into the stereo analyzer 116. The image data 70 is three-dimensional image data of the body part as described above. The stereo analyzer 116 analyzes the image data 70 to determine a left image data set and a right image data set based on the image data 70. The left image data set and the right image data set are data sets representing two-dimensional images that are slightly different from each other to provide a perception of three-dimensional rendering to an observer. The image data 70 is a static data set that does not change over time.

[0065] The stereo analyzer 116 inputs the left image data set and the right image data set to the left projector 88A and the right projector 88B. Then, the left projector 88A and the right projector 88B create the left light pattern 122A and the right light pattern 122B. The components of the display system 28 are shown in plan view, and the left light pattern 122A and the right light pattern 122B are shown in front view. Each light pattern 122A and 122B includes a plurality of pixels. For the purpose of illustration, it is shown that the light rays 124A and 126A from two pixels in the pixel leave the left projector 88A and enter the left waveguide 90A. The light rays 124A and 126A are reflected from the side of the left waveguide 90A. It is shown that the light rays 124A and 126A propagate from left to right in the left waveguide 90A by internal reflection, but it should be understood that the light rays 124A and 126A also propagate into the paper in a certain direction using the refraction and reflection system. Light rays 124A and 126A leave the left optical waveguide 90A through a pupil 128A and then enter the left eye 120A through the pupil 130A of the left eye. Then, the light rays 124A and 126A fall on the retina 132A of the left eye 120A. In this way, the left light pattern 122A falls on the retina 132A of the left eye 120A. The perception given to the observer is that the pixels formed on the retina 132A are the pixels 134A and 136A perceived by the observer at a certain distance on the side of the left waveguide 90A opposite to the left eye 120A.

[0066] In a similar manner, the stereo analyzer 116 inputs the right image data set into the right projector 88B. The right projector 88B sends a right light pattern 122B, which is represented by pixels in the form of light rays 124B and 126B. The light rays 124B and 126B reflect within the right waveguide 90B and exit through the pupil 128B. The light rays 124B and 126B then enter through the pupil 130B of the right eye 120B and fall on the retina 132B of the right eye 120B. The pixels of light rays 124B and 126B are perceived as pixels 134B and 136B behind the right optical waveguide 90B.

[0067] The patterns created on the retinas 132A and 132B are perceived as left image 140A and right image 140B, respectively, shown in front of the front view. The left image 140A and right image 140B are slightly different from each other due to the function of the stereo analyzer 116. In the mind of the observer, the left image 140A and right image 140B are perceived as three-dimensional renderings.

[0068] As described above, the left waveguide 90A and the right waveguide 90B are transparent. Light from a real object on the side opposite to the eyes 120A and 120B from the left waveguide 90A and the right waveguide 90B can be projected through the left waveguide 90A and the right waveguide 90B and fall on the retinas 132A and 132B. In particular, light from the surface of the patient's body 30 falls on the retinas 132A and 132B, so that the observer can see the surface of the patient's body 30. An augmented reality is created in which the surface of the patient's body 30 seen by the observer is enhanced by the three-dimensional rendering perceived by the observer due to the left image 140A and the right image 140B perceived by the observer in combination.

[0069] The head unit IMU 100 detects every movement of the observer's head. For example, if the observer moves counterclockwise around the patient's body 30 and simultaneously rotates his head counterclockwise to continue observing the patient's body 30, such movement will be detected by the gyroscope and accelerometer in the head unit IMU 100. The head unit IMU 100 provides the measurements from the gyroscope and accelerometer to the display adjustment algorithm 112. The display adjustment algorithm 112 calculates the placement value and provides the placement value to the display positioning algorithm 108. The display positioning algorithm 108 modifies the image data 70 to compensate for the movement of the observer's head. The display positioning algorithm 108 provides the modified image data 70 to the stereo analyzer 116 for display to the observer.

[0070] The head unit camera 102 continuously captures images as the viewer moves their head. The image processing system 114 analyzes the images by identifying the images of objects within the images. The image processing system 114 analyzes the motion of the objects to determine the pose position of the head mounted frame 86. The image processing system 114 provides the pose position to the display adjustment algorithm 112. The display adjustment algorithm 112 uses the pose position to further refine the placement value that the display adjustment algorithm 112 provides to the display positioning algorithm 108. The display positioning algorithm 108 thus modifies the image data 70 based on a combination of the motion sensors in the head unit IMU 100 and the images captured by the head unit camera 102.

[0071] Before the observer inserts the tip 78 into the patient's body 30, the catheter integration system 110 can detect the position of the tip 78 of the catheter 26. The observer then inserts the tip 78 into the patient's body 30. Then, the tip 78 is invisible to the observer. The catheter IMU 80 provides a signal indicating each movement of the tip 78 to the catheter integration system 110. The catheter integration system 110 can therefore use the motion sensor in the catheter IMU 80 to track the position of the tip 78. Unlike the static image data 70, the position of the tip 78 changes over time. The catheter integration system 110 provides the position of the tip 78 to the stereo analyzer 116. The position of the tip 78 can be dynamic because it changes over time and moves in three dimensions. The stereo analyzer 116 locates the tip 78 in the left image data set and the right image data set inserted into the left projector 88A and the right projector 88B. The observer can therefore see the position of the tip 78 in the left image 140A and the right image 140B. The position of the tip 78 varies slightly within the left image 140A and the right image 140B, causing the observer to perceive the position of the tip 78 in three dimensions. The rendering of the position of the tip 78 provided by the left image 140A and the right image 140B changes over time as the tip 78 passes through the patient's body 30. Such movement of the position of the tip 78 changes with the rendering in three dimensions, causing the observer to perceive the rendering of the tip 78 as moving in three dimensions, i.e., left, right, up, down, forward, backward, etc.

[0072] The catheter camera 82 continues to capture video data and provides the video data to the catheter integration system 110. The catheter integration system 110 provides the video data to the stereo analyzer 116. The stereo analyzer 116 places the video data at a fixed position within the viewer's field of view unless or until a user interaction event is detected indicating that the position should be changed. The video data changes over time as the catheter camera 82 captures different images.

[0073] The vision algorithm 94 is a set of instructions stored on a computer readable medium along with the data storage 24. The set of instructions can be executed by a processor to perform the above method. The computer readable medium storing the vision algorithm 94 can be located on a waist bag worn by the observer.

[0074] Figure 4 The components of the patient viewing system 20 , particularly the components of the catheter integrated system 110 , and their relationship to the catheter IMU 80 and catheter camera 82 in the tip 78 and the stereo analyzer 116 are shown in greater detail.

[0075] The catheter integrated system 110 includes a catheter tracking system 150, a past path calculator 152, a grid generator 154, an expected path calculator 156, a video data receiving unit 158, and a catheter display integrator 160. The catheter tracking system 150 is connected to the catheter IMU 80. The catheter tracking system 150 calculates the position of the tip 78 based on the motion detected by the catheter IMU 80. The catheter IMU 80 includes a plurality of tip tracking devices, including a plurality of gyroscopes and accelerometers, to track its motion in six degrees of freedom. The catheter tracking system 150 stores the current position of the tip 78 in the data memory 24 as the position 162. The catheter tracking system 150 continues to monitor the catheter IMU 80, continues to calculate the current position of the tip 78, and continues to store the current position of the tip 78 in the data memory 24 as the current position 162.

[0076] Catheter display integrator 160 receives current position 162 from data store 24 and provides current position 162 to stereo analyzer 116. Stereo analyzer 116 displays current position 162 of tip 78 to the viewer as a rendering so that the viewer can see the position of tip 78 as a three-dimensional rendering.

[0077] The past path calculator 152 retrieves each position 162 from the data store 24 at each point in time. The past path calculator 152 calculates the past path of the tip 78 in three dimensions and stores the past path as a past path 164 in the data store 24. The catheter display integrator 160 receives the past path 164 from the data store 24 and provides the past path 164 to the stereo analyzer 116. The stereo analyzer 116 displays the past path 164 as a three-dimensional rendering to the viewer.

[0078] The grid generator 154 retrieves the past path 164 from the data store and generates a three-dimensional grid around the past path 164. The grid generator 154 then stores the grid as a grid 166 in the data store 24. The catheter display integrator 160 retrieves the grid 166 from the data store 24 and provides the grid 166 to the stereo analyzer 116. The stereo analyzer 116 displays the grid 166 to the viewer. The stereo analyzer 116 creates a three-dimensional rendering of the grid 166, which in some embodiments overlays the past path 164.

[0079] The expected path calculator 156 retrieves each position 162 of the tip 78 from the data memory 24 and calculates a future path of the tip 78 based on the positions 162 retrieved from the data memory 24 and the past positions. The expected path calculator 156 then stores the future path as a future path 168 in the data memory 24. The catheter display integrator 160 retrieves the future path 168 from the data memory 24 and provides the future path 168 to the stereo analyzer 116. The stereo analyzer 116 displays the future path 168 to the viewer as a three-dimensional rendering.

[0080] The video data receiving unit 158 ​​receives live video from the catheter camera 82. The video data receiving unit 158 ​​provides the live video data to the catheter display integrator 160. The catheter display integrator 160 provides the live video data to the stereo analyzer 116. The stereo analyzer 116 displays the live video data to the observer. The live video data is a two-dimensional display that is displayed to the observer at certain predetermined distances in three-dimensional space. The catheter display integrator also integrates a grid 166 with the video data from the video data receiving unit 158 ​​so that the grid 166 is displayed on the video data. As the video data changes, as the position of the catheter 26 within the patient's body 30 changes, the grid 166 changes accordingly.

[0081] Figure 5 The use of the patient viewing system 20 described above by a viewer 172 in the form of a surgeon using a catheter 26 as a bronchoscope for the purpose of examining a body part 174 including a segmental bronchus of a patient's lung is shown in accordance with some embodiments.

[0082] The observer 172 can see the patient's body 30 through the left waveguide 90A and the right waveguide 90B. The body part 174 is inside the patient's body 30, so the observer cannot see the real (ie, physical) body part 174.

[0083] The observer 172 also sees a three-dimensional rendering 176 based on the image data 70 as described above. In a particular embodiment, the rendering 176 is located next to the patient's body 30. The rendering 176 is included in the figure to show the position of the observer 172 perceiving the rendering 176 relative to the patient's body 30, but it should be understood that from the perspective of a reader of this document, the rendering 176 does not really exist in the real world. The inset 180 shows that the observer 172 can see a three-dimensional rendering 182 of the body part 174 as part of the rendering 176.

[0084] Observer 172 inserts tip 78 of catheter 26 into the patient's mouth. Observer 172 then advances tip 78 into body part 174. As described above, the position of tip 78 is monitored at close intervals in time and its past path is stored in three dimensions. The sampling time may vary depending on the use case and may be optimized, such as capturing data only when the endoscope is inside the patient's body or after the user activates a "start recording / sampling" function. Inset 184 shows that rendering 176 includes rendering 186 of the position of tip 78 in three dimensions and rendering 188 of the past path of tip 78 in three dimensions. Renderings 182, 186, and 188 may be displayed to observer 172 simultaneously so that the observer sees renderings 186 and 188 within rendering 182.

[0085] Figure 6 1 is a top view showing the position of an observer 172 relative to the patient's body 30 according to some embodiments, and also showing the position of a rendering 176 within the field of view of the observer 172. The rendering 176 can be placed at any position relative to the patient's body 30 based on user preferences, pre-programmed default settings, or any other suitable means. Figure 6 The specific relative position of with respect to the rendering 176 is for illustrative purposes only and should not be considered limiting in any way.

[0086] Figure 7 According to some embodiments, Figure 6 The view 192 is viewed by the observer 172 in FIG. The observer 172 can see the patient's actual body 30 and the rendering 176. The view 192 also includes Figure 4 The view 192 also shows a grid 196 overlaying the video 194. The grid 196 is Figure 4 The display of the grid 166 is shown in FIG.

[0087] According to some embodiments, Figure 8 , the observer 172 has moved counterclockwise around the patient's body 30 and has also rotated his head counterclockwise to continue viewing the patient's body 30. The display adjustment algorithm 112 detects the movement of the observer's 172 head and adjusts the position of the rendering 176 accordingly so that the rendering 176 appears to remain stationary relative to the patient's body 30 within the observer's 172 field of view.

[0088] According to some embodiments, Fig. 9 In the patient's body 30 has been relative to Figure 7 The rendering 176 has also been rotated clockwise so that it remains stationary relative to the patient's body 30. However, the position of the live video 194 has not changed. Figure 7 View 192 in the image is changed to Fig. 9 192 in the view 192 in the viewer 172. Thus, the observer 172 sees the live video 194 and the grid 196 in the same position, and these components do not move when the observer 172's head moves. Therefore, the observer 172 can observe the patient's body 30 and the rendering 176 from different sides and angles without losing sight of the live video 194 and the grid 196. The purpose of the grid 196 may be to assist the observer in guiding the tip 78 of the catheter 26 when the observer 172 inserts the tip 78 into the passage of the body part 174 a second time after the grid is generated, or during the removal of the catheter as the catheter moves through the same path in the opposite direction. Some embodiments may have different viewing configurations for virtual content (e.g., the grid 196, the live video 194, the rendering 176), in which some or all of the virtual content is fixed relative to real-world coordinates, or fixed relative to the observer.

[0089] Fig.10 According to some embodiments, the components of the rendering 176 displayed to the viewer are too small to be displayed on the Figure 7 and Fig. 9 182, 186, and 188 of the past path of the tip. The viewer 172 sees the body part 174, the tip 78, and renderings 182, 186, and 188 of the past path of the tip. The viewer also sees a three-dimensional rendering of a mesh 196. For purposes of illustration, the mesh 196 is shown as being separate from the renderings 182, 186, and 188, but it should be understood that the mesh 196 can overlay the rendering 182 of the body part 174.

[0090] like Fig.11 and Fig.12 As shown in FIG. 1 , in some embodiments, the viewer 172 sees the grid 196 in two locations, namely as a rendering 176 ( Fig.11 ) and covers the live video 194( Fig.12 ).

[0091] Fig.13 According to some embodiments Figure 4 1. The graph shows the tip 78 of the catheter 26 rotating about the xy axis and the x axis over time. The rotation about each axis can be analyzed over a short amount of time to determine a first amount of motion 200 from a first position 202 to a second position 204. The first amount of motion 200 can be used to calculate a prediction of the movement of the tip 78.

[0092] Fig.14 The tip 78 is shown in a first position 202 according to some embodiments. Fig.15 , the tip 78 has moved a first amount 200 from a first position 202 to a second position 204 . Fig.15The first quantity 200 shown in is the sum of all vectors of all motions around all axes and in all translation directions.

[0093] exist Fig.14 , it can be assumed that if the tip 78 is inserted further into the body part 174, the direction that the tip 78 will follow is along the extension 206 of the inner cavity 76. Fig.16 Extension 208 of lumen 76 is shown after lumen 76 has moved first amount 200 according to some embodiments. An observer will generally not advance tip 78 along the path of extension 208 because it will contact body part 174 and cause injury. Instead, observer 172 will prefer to follow path 210 in order to avoid injury. Path 210 is displaced second amount 212 from second position 204 to third position 214. For ease of reference, first amount 200 and second amount 212 are measured in the same direction.

[0094] Fig.17 An actual path 218 is shown that tip 78 will likely follow according to some embodiments. As the viewer inserts tip 78 further into body part 174, path 218 moves away from path 208 and approaches path 214. Stereo analyzer 116 displays path 218 to viewer 172 in three dimensions.

[0095] Fig.18 A schematic diagram of a machine in the exemplary form of a computer system 900 is shown according to some embodiments, within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. Furthermore, while only a single machine is shown, the term "machine" shall also be construed to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0096] The exemplary computer system 900 includes a processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), and a static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.) that communicate with each other via a bus 908.

[0097] The computer system 900 may also include a disk drive unit 916 and a network interface device 920 .

[0098] The disk drive unit 916 includes a machine-readable medium 922 on which is stored one or more sets of instructions 924 (e.g., software) embodying any one or more of the methods or functions described herein. During execution of the software by the computer system 900, the software may also reside, in whole or in part, within the main memory 904 and / or within the processor 902, the main memory 904 and the processor 902 also constituting machine-readable media.

[0099] The software may further be transmitted or received over a network 928 via the network interface device 920 .

[0100] The computer system 900 includes a laser driver chip 950 for driving a projector to generate laser light. The laser driver chip 950 includes its own data memory and its own processor 962.

[0101] Although the machine-readable medium 922 is shown as a single medium in the exemplary embodiment, the term "machine-readable medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable medium" should also be considered to include any medium that can store, encode, or carry a set of instructions for execution by a machine and cause the machine to perform any one or more of the following methods of the present invention. Therefore, the term "machine-readable medium" should be considered to include, but is not limited to, solid-state memories, optical and magnetic media, and carrier signals.

[0102] The implementation described above uses a CT scanner 22 to scan a body part 174. The CT scanner has a transmitter in the form of an x-ray transmitter, a receiver in the form of an x-ray detector, and sends and receives waves in the form of x-ray waves. Other scanning devices may be used that use other transmitters and receivers and send and detect different waves. For example, a sonar system uses a sound transmitter to send sound waves and uses a sound receiver to receive sound waves. A visual system may include a light source that is inserted into the body part and sends light waves and has a camera located within the body part that captures the light waves reflected from the body part.

[0103] However, a CT scanner is preferable to other scanning devices because a CT scanner can provide very detailed raw data of a body part in three dimensions, and such data can be easily converted with an image generation unit to create three-dimensional image data. Another advantage of CT data is that it can contain data about specific substances, materials, and material densities. The described implementation shows a rendering 176 placed next to the patient's body 30 in the view 192 of the observer 172. The rendering can also be matched to the patient's body 30 so that the rendering of the body part is located where the actual body part is, and the rendering of the catheter tip is located where the actual position of the catheter tip is.

[0104] Aspects of the invention may also be implemented without a catheter. For example, a patient's body may be scanned to determine a growth, and an observer may use a display system to overlay a rendering of the growth in three dimensions over the patient's actual body. In this manner, the observer may "see" the growth "inside" the patient's actual body.

[0105] In certain aspects and embodiments, instead of or in addition to the described catheters, typically invasive surgical tools are used. For example, reference Figures 14 to 17 , the tip 78 can indicate the front end of a thermal imaging camera that is coupled to the catheter to collect temperature data through the anatomical channel through which it passes. The thermal data can then be converted from the black and white thermal image and displayed to the operator as a variable color overlay by the image receiving unit 106. It will be appreciated that not every display requires an image that is common to every user. A first operator or observer (local or remote) may want certain information to be displayed, such as temperature data indicating arterial blood versus venous blood or frostbitten tissue, while a second operator or observer may want certain information, such as information indicating the location of a surgical tool. In other words, the display of information need not be limited to an image of the location of the tool within the patient's body, but can also be an image collected from the tool. In some thermal imaging embodiments, a temperature gradient can be selected for each user; a first operator may want to distinguish organic tissue from inorganic material and set the base temperature to 98.6 degrees Fahrenheit, while a second operator may aim to track surgical tools specifically and set the base temperature of the imaging to a preset temperature rather than an absolute temperature.

[0106] In some embodiments, an operational envelope is implemented. Figure 14-17 , while navigating anatomical pathways, the patient's particular disposition may change the instrument's operating load. For example, if patient imaging indicates sensitive tissue near the instrument, the load may be adjusted proportionally. To further illustrate, if imaging of a patient with atrial fibrillation indicates a thin left atrium wall, an ablation catheter with a standard axial load of 0.1 N may be pulled to within 0.01 or 0.05 N when operating near the left atrium wall. In some embodiments, a loading paradigm as a function of position may be displayed to the operator in conjunction with an instrument image such as position or collected images, which is absolute and relative to anatomical landmarks (such as sensitive tissue). Thus, visual feedback may be provided indicating the upper limit of the load on the instrument as a function of position to inform the operator of the device's capabilities or limitations at that moment. Additionally, for a given procedure, an additional operator or observer may immediately recognize when the instrument is not in the correct position or exceeds patient parameters. Although such feedback is described as visual feedback, feedback may take other forms, such as audio or tactile (increased resistance or friction on instrument controls).

[0107] In some embodiments, observable motion artifacts provide position adjustments for the instrument. The head unit camera 102 can image the patient's position data and provide real-time position adjustments to the instrument. In some embodiments, the patient's respiratory rhythm is observed and the instrument controls are modified to adapt to the respiratory state. The head unit camera 102 can record position data, for example, through fiducial markers within the operating environment or a fixed machine of known size, to compare chest position and associate size changes during exhalation and inspiration. In some embodiments, as the lungs expand during inspiration and the surrounding anatomical structures compress in response, the instrument movement can be slowed down accordingly and then restored to normal operating parameters as the lungs shrink during exhalation, or xyz adjustments are made to match the rise and fall of the chest cavity so that the instrument can be moved absolutely in the anatomical channel, but is stable relative to the anatomical reference. Other observed triggers can also provide operational adjustments, and heart rate and expected vasoconstriction / dilation can provide position updates for the instrument.

[0108] In some embodiments, at time T l Feedback control of patient activity (e.g., breathing rhythm, heartbeat) is collected by the head unit camera 102 at three separate times from T2 to T3, and the images are analyzed at time T4 to determine changes in patient position (due to patient movement, their breathing rhythm, heartbeat, etc.). Control inputs are provided to instruments operably coupled to the head unit camera 102 at time T0 (i.e., before or simultaneously with T1), and the control inputs are adjusted at time T5 based on changes in patient position observed during time T1 to T3. In some embodiments, the changes in control inputs are part of the changes in patient position that are measured.

[0109] Fig.19 An exemplary relationship between the measured patient position data and the control feedback adjustment is shown. As shown, curve 1901 represents the theoretical change in the y direction of the patient position caused by chest expansion due to breathing. The observed patient position data is collected at times T1, T2 and T3, corresponding to chest positions y1, y2 and y3, respectively. Preferably, three collection times are performed to provide trend analysis for target measurement. For example, the average respiratory rate of a person is 15 breaths per minute, and about 2 seconds are allocated for any given inhalation or exhalation. In order to avoid correction of the instrument position applied during exhalation based on the measured inhalation, at least three measurements are performed to provide trend analysis. Given the average respiratory rate of humans, the time interval between two measurements is preferably at least 0.667 seconds or 1.5Hz. Such a frequency does not have to be the time of each time between T0 and T5, and it is preferred that the relevant actions at T4 and T5 are applied as quickly as possible.

[0110] Back to Fig.19 , the change in position y1 to y2 to y3 during the analysis at T4 indicates to the system that inspiration ended near T3. Thus, feedback control at T5 may provide no adjustment to avoid applying an "inhalation correction" during exhalation, or may provide a negative y-position adjustment based on the temporal relationship of T5 to T3 and the extrapolated y-position change. For example, if the time between T3 and T5 is similar to the time between T2 and T3, and the system identifies that the patient is experiencing a change in the y direction based on the change between y1 and y2 (compared to y2 and y3), then the adjustment may be less than or equal to the change as measured from T2 to T3. This can be described as the following logical relationship:

[0111] If (T5-T3=T3-T2)∩(y3-y2)<(y2-y1), correction is performed at T5<(y3-y2).

[0112] Although certain exemplary embodiments have been described and shown in the drawings, it should be understood that these embodiments are merely illustrative and not restrictive of the invention, and that the invention is not limited to the specific construction and arrangements shown and described, as modifications may be made by those of ordinary skill in the art.

Claims

1. A patient observation system, comprising: a catheter having a lumen and a tip; a tip tracking device that detects movement of the tip; left projector and right projector; a left optical waveguide and a right optical waveguide, the left optical waveguide being connected to the left projector and the right optical waveguide being connected to the right projector; processor; a computer readable medium coupled to said processor; a data storage device located on said computer readable medium; image data stored on the data storage device; A set of instructions, stored on the computer-readable medium and executed by the processor, comprising: a catheter tracking system connected to the tip tracking device, wherein the catheter tracking system receives measurements based on motion detected by the tip tracking device, determines a position of the tip based on the measurements, and stores the position of the tip in the data storage; and A stereo analyzer connected to the data storage to receive the image data, the stereo analyzer determining a left image data set and a right image data set, the left projector and the right projector projecting the left image data set and the right image data set, respectively, the left image data set and the right image data set being different from each other to provide a perception of three-dimensional rendering to an observer.

2. The patient observation system of claim 1, further comprising: A head-mounted frame to which the optical waveguide is fixed.

3. The patient observation system according to claim 2, wherein: The optical waveguide is a transparent optical waveguide positioned between the eye and the outer surface of the body.

4. The patient observation system of claim 2, further comprising: a head unit detection device that detects movement of the head mounted frame, The instruction set comprises: a display adjustment algorithm connected to the head unit detection device and receiving measurements based on motion detected by the head unit detection device and calculating a placement value; and A positioning algorithm is displayed that modifies the position of the body part within the field of view of the eye based on the placement value.

5. The patient observation system according to claim 4, wherein: The head unit detection device comprises: A head unit inertial measurement unit (IMU) is mounted to the head mounted frame, the head unit IMU including a motion sensor for detecting motion of the head mounted frame.

6. The patient observation system according to claim 4, wherein: The head unit detection device comprises: A head unit camera is mounted to the head mounted frame, the head unit camera detecting movement of the head mounted frame by capturing images of objects within the field of view of the head unit camera.

7. The patient observation system according to claim 6, wherein: The instruction set includes: An image processing system analyzes the image to detect a posture position of the head mounted frame.

8. The patient observation system of claim 1, further comprising: transmitter; an energy source connected to the transmitter to activate the transmitter, the patient's body being positionable relative to the transmitter so that the transmitter generates a forward wave at a body site within the body; a receiver positionable relative to the body to detect a return wave from the body part in response to the forward wave generated by the transmitter, The instruction set comprises: a raw data receiving unit that receives raw data of the return wave detected by the receiver and stores the raw data in the data memory; an image generation unit connected to the data memory to process the raw data of the return wave to create image data representing an image, and store the image data in the data memory; an image data receiving unit that receives the image data from the data storage; and A catheter display integrator combines the position of the tip with the image data, the light pattern produced by the projector including a pattern representing the image data and the position of the tip.

9. The patient observation system according to claim 8, comprising: Computed tomography (CT) scanner, which includes: Pedestal; a platform for said patient; and a rotor mounted to the base for rotation about the patient, wherein the transmitter is an x-ray transmitter fixed to the rotor and emitting x-ray waves, the receiver is an x-ray detector fixed to the rotor and detecting the x-ray waves, and movement of the platform relative to the base allows movement of the patient relative to a plane extending from the x-ray transmitter to the x-ray detector.

10. The patient observation system according to claim 1, wherein: The instruction set includes: A past path calculator stores a past path of the tip in the data memory, and the catheter display integrator displays the past path of the tip together with the position of the tip.

11. The patient observation system according to claim 1, wherein: The instruction set includes: a past path calculator that stores a past path of the tip; and A grid generator generates a three-dimensional grid around the past path of the tip and stores the grid in the data memory, and the catheter display integrator displays the grid together with the position of the tip.

12. The patient observation system of claim 11, further comprising: a catheter camera in the tip, the catheter camera capturing video data, The instruction set comprises: A video data receiving unit is connected to the catheter camera in the tip to receive the video data, and the catheter display integrator displays a live video based on the video data.

13. The patient observation system of claim 1, wherein: The instruction set includes: An expected path calculator calculates a future path of the tip based on the position of the tip, the catheter display integrator displays the future path.

14. The patient observation system according to claim 13, wherein: The future path is calculated based on the motion detected by the tip tracking device.

15. The patient observation system of claim 14, wherein: The movement detected by the tip tracking device is a first amount of movement from a first position to a second position in a selected direction through a first angle, and the future path is a second amount of movement from the second position to a third position in the selected direction through a second angle.

16. The patient observation system of claim 1, further comprising: An invasive surgical tool is provided on the tip.

17. The patient observation system of claim 1, further comprising: a camera on said tip; as well as At least one display that displays images captured by the camera to an operator.

18. The patient observation system of claim 17, wherein: A first display displays the three-dimensional rendering and the image captured by the camera to a first operator.

19. The patient observation system of claim 17, wherein: The first display displays the three-dimensional rendering to the first operator, further comprising: A second display displays images captured by the camera to a second operator.

20. The patient observation system of claim 1, further comprising: a first display that displays the three-dimensional rendering to a first operator; a data collection system that collects data in the absence of the catheter; as well as A second display displays data collected by the data collection system to a second operator.

21. The patient observation system of claim 1, further comprising: a load detection system connected to the tip to determine an operational load on the tip; as well as A warning system is connected to the load detection system to warn an operator if the load on the tip exceeds a predetermined limit.

22. The patient observation system of claim 21, wherein: The warning is one of: a visual warning, an audio warning, and a tactile warning.

23. The patient observation system of claim 1, further comprising: a motion detection system, which observes motion artifacts; as well as A position adjustment system is connected to the motion detection system to adjust the position of the tip based on the motion artifact.

24. The patient observation system of claim 23, wherein: The motion detection system collects feedback control images of the patient's activity at three separate times from time T1 to time T2 to time T3, the images are analyzed at time T4 to determine changes in the patient's position, and the control input is adjusted at time T5 based on the changes in the patient's position observed from time T1 to time T3.

25. The patient observation system of claim 24, wherein: If (T5-T3=T3-T2)∩(y3-y2)<(y2-y1), then make a correction at T5<(y3-y2). Therein, the patient positions observed at times T1, T2, and T3 correspond to chest positions y1, y2, and y3, respectively.

Images