Augmented-reality endoscopic vessel harvesting

Abstract

To provide an endoscopic vessel harvesting system for surgical removal of a blood vessel to be used for a coronary artery bypass, which uses an endoscopic instrument for isolating and severing the vessel.SOLUTION: An endoscope camera in an endoscopic instrument captures an image from a distal side tip of an instrument in an anatomical tunnel around a blood vessel. An image processor assembles a three-dimensional model of the tunnel from a series of images captured by the endoscope camera. An augmented-reality display connected to the image processor expresses an integrated map indicating the three-dimensional model with a marker in relation to a map showing a present place of the distal side tip (displayed to a user visually in a field of view of the user, for example).SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] Cross - reference to related applications Not applicable. Description of research funded by federal government Not applicable.

[0002] The present invention generally relates to devices and methods for endoscopic dissection of blood vessels within a patient's limb, and more particularly to integrating an endoscopic vessel harvesting system with an augmented reality device to improve ease of use and patient prognosis.

Background Art

[0003] In relation to coronary artery bypass grafting (CABG), a blood vessel or vascular section, such as an artery or vein, is "harvested" (i.e., removed) from its natural location within the patient's body for use elsewhere in the body. In CABG surgery, for example, this blood vessel is used to form a bypass between an arterial blood source and a coronary artery to be bypassed. Suitable sources for the vasculature to be used as bypass grafts include, in particular, the saphenous vein in the leg and the radial artery in the arm.

[0004] To avoid the disadvantages and potential complications of harvesting via open incision, endoscopic surgical techniques for harvesting sections of veins (e.g., saphenous veins) subcutaneously have been developed. One such minimally invasive technique employs a small incision to locate the desired vasculature and introduce one or more endoscopic harvesting devices. The initial dissection is performed by introducing dissecting instruments through the incision to create a working space and separate the vasculature from the surrounding tissue. Next, a cutting instrument is introduced into the working space to detach the blood vessel from connective tissue and vascular branches. The branches can be cauterized using the cutting instrument.

[0005] In one typical procedure, the endoscopic entry point is located near the midpoint of the vessel to be harvested, where the dissection and cutting of the branch advances bidirectionally along the vessel from the entry point. To remove the desired section of the vessel, a second small incision or puncture is made at one end of this desired section, and this vessel section is ligated. A third small incision is made at the other end of the thus ligated vessel section, thereby allowing the complete removal of this desired vessel section through the first incision. Alternatively, if the length of the endoscopic device is sufficient to obtain the desired length of vessel while working in only one direction along the vessel from the entry point, only the first two incisions may be necessary.

[0006] An example of a commercially available product for performing the endoscopic venous sampling described above is the VirtuoSaph Plus® endoscopic vascular sampling system from Terumo Cardiovascular Systems Corporation of Ann Arbor, Michigan. This type of endoscopic vascular sampling system is also shown in U.S. Patent Nos. 7,331,971 and 8,048,100, as well as U.S. Patent Application Publication Nos. 2010 / 0292533 and 2012 / 0035606, which are incorporated herein by reference in their entirety.

[0007] Dissection tools typically consist of a longitudinal rod made of stainless steel or plastic, with a tip at one end and a surgeon's handle at the other. The tip tapers to a non-sharp end and is made of clear plastic. Dissection is performed by advancing along the outer circumference of the harvested vessel to separate it from the surrounding tissue and expose its side branches, so that the vessel can be cut using a cutting tool. In the VirtuoSaph®Plus® system, the cutting tool for cutting and cauterizing branches has the form of a V-cutter, where the V-shaped tip is extendable from the distal end of the unit to guide the branch to be cut into a longitudinal slit. An electrode adjacent to the slit is electrically excited with a high-frequency voltage to cauterize and decompose the branch by coagulation. In addition, a V-keeper extends from the distal end to capture the vessel and guide the tool along the vessel.

[0008] An internal endoscopic view is provided to the user via an optical system having a camera and video display. The camera may be installed inside the distal tip of the collection device. Alternatively, a lens and optical fiber mounted inside the collection device can transmit images to a camera located at the remote end of an optical fiber outside the collection device or located inside the handle of the device. The view is illuminated by a light source, such as an LED, installed at the tip of the collection device (dissection instrument or cutter), or by a remote source that inputs light into an optical fiber extending through the collection device to emit light from the tip. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] U.S. Patent No. 7,331,971 [Patent Document 2] U.S. Patent No. 8,048,100 [Patent Document 3] U.S. Patent Application Publication No. 2010 / 0292533 [Patent Document 4] U.S. Patent Application Publication No. 2012 / 0035606 [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] The endoscopic camera view during the dissection or cutting phase is displayed on a computer monitor. Significant training may be required for the user to become proficient in appropriately adjusting their movements with dissection / cutting instruments while looking at the monitor and taking their eyes off the patient.

[0011] Augmented reality (AR) is an interactive experience of the real world environment in which objects present in the real world are enhanced by computer-generated perceptual information, sometimes through multiple types of senses, including sight, hearing, touch, somatosensory perception, and smell. Specifically, an augmented reality display may have eyewear comprising an open view field for viewing physical objects and an augmented viewing portion configured to represent glyphs and / or video content. [Means for solving the problem]

[0012] In one aspect of the present invention, a vascular sampling system comprises an endoscopic camera of an endoscopic instrument that captures images from the distal tip of the instrument within a dissection tunnel around a vascular vascular sample to be taken. An image processor assembles a three-dimensional model of the tunnel from a series of images captured by the endoscopic camera. An augmented reality display connected to the image processor displays an integrated map representing the three-dimensional model with markers in relation to a map indicating the current location of the distal tip (for example, visually displayed to the user in the user's field of view). [Brief explanation of the drawing]

[0013] [Figure 1] This is an external view showing the saphenous vein taken from the leg. [Figure 2] A side view showing a prior art dissection instrument unit. [Figure 3] A side view showing a prior art cutting unit. [Figure 4] A plan view showing a prior art non-sharp dissection instrument with an endoscope and a trocar. [Figure 5] A partial cross-sectional view showing the dissection of a blood vessel. [Figure 6] An endoscope camera view showing the V-keeper and V-cutter of a sampling device unit deployed in a working tunnel around a target vasculature. [Figure 7] A block diagram showing one embodiment of a vasculature sampling system using augmented reality. [Figure 8] A schematic diagram showing an augmented reality display representing a three-dimensional model of the tunnel / vessel during sampling. [Figure 9] An endoscope camera view showing changing characteristic portions when a cutting tool advances along a dissection tunnel. [Figure 10] An endoscope camera view showing changing characteristic portions when a cutting tool advances along a dissection tunnel. [Figure 11] A block diagram showing components for aggregating a three-dimensional model and for generating a map for representation on an augmented reality display. [Figure 12] A schematic diagram showing an augmented reality display representing a real-size indicator. [Figure 13] A block diagram showing components for identifying the types of structures visible within an endoscope camera image. [Figure 14] A schematic diagram showing an augmented reality display representing a structure identification indicator. [Figure 15] A perspective view showing the distal tip of a dissector instrument having a sensor for detecting the depth of a structure visible within an endoscope camera image. [Figure 16] A schematic diagram showing an augmented reality display representing a relative depth indicator. [Figure 17] A schematic diagram showing an augmented reality display representing an actual size indicator for a recognized feature portion. [Figure 18] A schematic diagram showing an augmented reality display having an image of an endoscopic instrument with size markings. [Figure 19] A schematic diagram showing an augmented reality display representing a width indicator for a side branch. [Figure 20] A diagram showing pre - mapping of a vascular location on a patient's skin. [Figure 21] A diagram showing tracking of the position of an endoscopic instrument with respect to a pre - mapping vascular path. [Figure 22] An endoscopic camera view visible through an anatomical instrument. [Figure 23] A schematic diagram showing an augmented reality display representing a message generated when the dissection of a tunnel does not match the pre - mapping path and is impaired. [Figure 24] A flowchart showing one method of dissecting a tunnel using augmented reality.

Mode for Carrying Out the Invention

[0014] Referring to FIG. 1, patient 10 has a certain superficial vein 11 within the lower limb portion 12. An incision 13 is made just above the vein 11, and tissue is peeled away from the incision 13 to access the vein. An endoscopic instrument is inserted through the incision 13 to separate the vein 11 from the connective tissue and then to cut off and cauterize the side branches extending from the vein 11. A second incision or puncture 14 is made at a second position on the limb portion 12, so that the second end of the vein 11 can be cut off. Then, the vein 11 is pulled out through one of the incisions. The entry point and / or the second incision or puncture can be placed at various locations along the vein 11 as shown in FIG. 15.

[0015] Figure 2 shows a known dissection instrument unit 16 for endoscopic dissection of a saphenous vein or other blood vessel, which is inserted through the initial incision and pressed into the fat along the direction of the blood vessel to separate it from adjacent tissue. The dissection instrument unit 16 has a handle 18 connected to a longitudinal rod 19 having a dissection instrument tip 17 at its distal end. A receiver 20 at the end of the handle 18 receives an endoscope and optical cable (not shown) extending through the rod 19 to the dissection instrument tip 17, and the dissection instrument tip 17 is transparent to allow visualization of the blood vessel and surrounding tissue. An inhalation tube 21 is part of an inhalation gas channel that passes through the handle 12 and extends to a discharge port located within or near the tip 17. The tube 21 is connected to a source of CO2 or other inhalation gas to fill cavities adjacent to the blood vessel when cavities are formed.

[0016] Following an initial blunt dissection around the blood vessel, a harvesting device cutting unit 22, as shown in Figure 3, is used subcutaneously to grasp the dissected blood vessel and to sever any branches or connective tissue connected to the blood vessel. The harvesting device 22 has an elongated sleeve member 24 and a handle 23 connected to an endoscope receiver 25. At the distal end of the sleeve 24, there is a vascular keeper (V-keeper) 26 for holding the dissected blood vessel and a vascular cutter (V-cutter) 27 for severing branches. The V-keeper 26 is operated by a V-keeper button 28 located on the handle 23. The V-cutter 27 is extended or retracted by operating a V-cutter extension button 29 located on the handle 23. An inhalation device tube 30 is adapted to be connected to an inhalation source to deliver gas to the distal end of the sleeve 24 via a gas channel extending between the handle 23 at the proximal end and the discharge port at the distal end. A bipolar cord or integrated bipolar cord 31 is connected to a source of high-frequency voltage and has conductors for supplying voltage to electrodes on a V-cutter 27 for cutting and cauterizing branches and connective tissue.

[0017] In some embodiments, cutting and cauterization can be achieved using a scissor-like instrument, such as ordinary scissors, instead of a V-cutter. The scissor-like instrument may have electrodes or other excitation devices on its inner surface, which is pressed against the side branches to be cut.

[0018] Figures 4 and 5 show another vascular sampling system comprising an endoscope unit 32 for performing intracellular monitoring of a patient, a dissection instrument unit 36 ​​for dissecting blood vessels within the body, and a trocar 40 for assisting in the insertion of the endoscope 32 and the dissection instrument unit 36 ​​into the body. The optical system is shown as a rigid endoscope 32 having an elongated rod-shaped insertion portion 33. The proximal end of the insertion portion 33 is connected to an end adapter 34 for transmitting endoscopic images. A light guide port 35 protrudes from the end adapter 34 to be connected to a light guide cable that supplies illumination light to the endoscope 32. In other embodiments, the optical system may employ a camera and LED light source mounted at the distal end of the endoscope 32, connected via an electrical cable for power supply, and a video image processor.

[0019] The dissection instrument unit 36 ​​has a tubular main body portion with a hollow longitudinal rod 37 into which the endoscope 32 is inserted. The endoscope 32 is inserted into or removed from the longitudinal rod 37 through the handle portion 38. The material of the longitudinal rod 37 may be a fluoropolymer. The most suitable material for forming the outer surface of the longitudinal rod 37 is polytetrafluoroethylene (PTFE). By using a fluoropolymer, the friction generated by moving the rod 37 through connective tissue is reduced, thereby reducing the force required to perform the dissection.

[0020] A non-sharp dissection instrument tip 39 is positioned at the distal end of the longitudinal rod 37. The tip 39 has a conical shape and includes a transparent synthetic resin material to assist in visualizing the direction along the tip 39 using the endoscope 32. A trocar 40 guides the dissection instrument unit 36 ​​into the incision site. The outer surface of the trocar 40 has projections for engaging with biological tissue and a retaining portion 41 for holding the trocar 40 against biological tissue 43 (e.g., the patient's skin). Because the insertion direction of the dissection instrument 36 is aligned with the direction of the target blood vessel 45 being dissected, in order to dissect peripheral tissue 46 from the blood vessel 45 (to create a working tunnel 44), the surgeon will gradually insert the dissection instrument while viewing the endoscopic image on a display 48 connected to the endoscope 32 by a cable 47.

[0021] After dissecting the working tunnel along the target vessel, the dissection instruments may be removed, and cutting instruments may be inserted into the working tunnel to separate the target vessel from any side branches and any undissected connective tissue. Figure 6 is an endoscopic view (in other words, field of view) as seen during vascular harvesting, where the target vessel 45 (e.g., saphenous vein) is held within the V-keeper 26. Side branches 50 extend from the vessel 45 into the previously made tunnel during blunt dissection. The V-cutter 27 is positioned to extend toward the side branches 50 to cauterize and separate them in order to prepare the section of the vessel 45 for removal. Because side branches such as side branches 50 extend radially away from the vessel 45, the harvesting device must be rotated around the vessel 45 to directly access all different side branches along the length of the vessel 45 being harvested. For several reasons, it may be difficult for the user to maintain a mental picture of the subcutaneous tissue, and it may also be difficult for the user to maintain the location of the instrument head within this mental picture when monitoring the progress on a computer.

[0022] Figure 7 shows a first embodiment of an augmented reality vascular harvesting system to assist the user in maintaining a supportive position of the object while working during the dissection and amputation phases. The patient 51 has an incision 52 into which a harvesting instrument (e.g., a dissection instrument or cutter) 53 is inserted. The instrument 53 is coupled to an inhalation gas source 54, a power supply 55, and a light source 56. Camera images from the instrument 53 are sent (coupled) to an image processor 57, and the processed images may be sent to a conventional display 58 and / or an augmented reality display 60. A selector 59 is coupled to the image processor 57, and the selector 59 may be used by the user to initiate commands to update the augmented reality content on the display 60. Preferably, the selector 59 is configured to generate commands in a hands-free manner, as will be described later.

[0023] The augmented reality display 60 may consist of a head-mounted display, sometimes referred to as “smart glasses.” For example, the display 60 may take the form of glasses, a visor, an open area, or a face shield that can be worn on the head or face by a user (e.g., a surgical technician or a medical assistant). The display 60 may have a view field through which a user can see physical objects within their field of vision, and may be sometimes referred to as “non-obstructive” or “non-obstructive head-up display (HUD).” For example, there may be a clear portion of glass, plastic, or a similar transparent material through which light emitted from a physical object passes into the user’s eye. In some embodiments, the display 60 may have a solid or opaque portion that completely or partially obstructs the user’s view, and may be sometimes referred to as “obstructive” or “obstructive HUD.” This view field may have one or more screens (e.g., light-emitting diode screens, i.e., LED screens) along with one or more cameras that capture video data of the user’s viewpoint. Therefore, the video is displayed on the screen, thereby providing the user with a view field that is similar to a clear view of the physical environment.

[0024] In another embodiment, the display 60 may have a retinal projection device configured to project an image directly onto the wearer's eyes. In some cases, the retinal projection device may have a clear portion of glass, plastic, or similar transparent material through which light emitted from a physical object passes into the user's eyes. In some cases, the display 60 with the retinal projection device may have one or more cameras that capture video data of the user's viewpoint. The video is then represented and projected onto the user's eyes, thereby providing the user with a view field similar to a clear view of the physical environment. In some implementations, the display 60 may be configured to take into account the user's visual impairment. For example, the retinal projection device may be configured to provide projected images to users with corneal opacity or cataracts in a manner that is clear to such users.

[0025] In yet another embodiment, the display 60 may have a half-mirror portion made of glass, plastic, or a similar transparent material, through which light emitted from a physical object passes to the user's eyes, during which time light is emitted onto the half-mirror view field to represent glyphs and the like.

[0026] The augmented reality display 60 is configured to represent glyphs (e.g., characters, symbols, colored overlays, etc.) and to represent video within a view field. For example, a light emitter may emit light into a transparent view field, resulting in the user seeing a reflection of light. In another embodiment, if a screen is used to display video from the user's viewpoint, glyphs and video may be displayed superimposed on the viewpoint video. In either case, the display 60 displays the glyphs and video as superimposed displays relative to a view of physical objects.

[0027] The display 60 may have other feature components. For example, it may include a microphone or earphone for connecting to an internal communication device, a mobile phone, or other telecommunications device. This may allow the surgeon to communicate with people in the same facility or with people further away via the microphone or earphone.

[0028] As will be discussed in more detail later, many different types of glyphs and video images may be displayed to the user. Selector 59 allows the user to generate screen update commands to modify the content of display 60 (e.g., changing the characteristics of the displayed items by selecting a different glyph, scrolling through and examining the patient's monitored physiological parameters, selecting a different image source, or zooming in on a region of an image). Because it is desired that the user (e.g., the wearer of display 60) maintain their handgrip over the collection instrument, selector 59 is configured to receive commands while the user continues to hold the instrument. Selector 59 may consist of a manual control device installed within its gripping area on the instrument. Alternatively, selector 59 may consist of a hands-free device that senses other actions by the user. For example, selector 59 may have an eye-tracking camera that detects specific eye movements of the user designated to trigger a corresponding update command. Alternatively, the selector 59 may have 1) a microphone and voice recognition system for enabling the user to generate screen update commands via voice commands, 2) a motion sensor that responds to predetermined movements of the user, or 3) a foot pedal (e.g., connected to the image processor 57 via a Bluetooth® connection) equipped with one or more switches for generating desired update commands.

[0029] Figure 8 shows an embodiment of the view field 61 on and through the augmented reality display. The transparent portion of the view field 61 without the overlay display provides an actual live view of the patient's limbs 62 and the endoscopic instrument 63. The overlay display 64 provides a copy of the instantaneous endoscopic image received by the image processor from the endoscopic camera (e.g., displayed on the lateral side of the view field). The overlay display 65 (e.g., displayed on the upper side of the view field) represents an integrated map representing a three-dimensional model of the surgical anatomy (e.g., target vessels, their side branches, and other aspects of the anatomical tunnel), and a marker positioned relative to the 3D map to indicate the current location of the distal tip of the head of the endoscopic instrument relative to the 3D map.

[0030] In some embodiments, an image processor (e.g., a local or remote computer-based unit in communication with an endoscope instrument that performs at least part of an image processing task using images captured by an endoscope camera) assembles a three-dimensional model of the tunnel of a vascular structure from a series of images captured over time by the endoscope camera. The image processor may be configured to extract feature point points by utilizing a comparison of the changing positions of detected feature points in the series of images to determine the estimated distance between detected feature points in the three-dimensional model. The process for assembling the 3D model may include automatically stitching together image data from overlapping and / or adjacent view fields, as done to create a panoramic type image.

[0031] Figure 9 shows a first capture image acquired at a specific location along the anatomical tunnel. The image may contain one or more recognizable feature segments suitable for tracking to establish reference points along the tunnel, such as side branches, sections of connective tissue with distinctive shapes, pockets of adipose tissue, or other optically stable markings. As the head of the endoscopic instrument moves along the tunnel, the expected positions of the traceable feature segments change. Figures 9 and 10 show virtual grids 70 and 71 that remain fixed relative to the endoscopic camera, so that as the imaging position changes, the traceable feature segments move relative to grids 70 and 71. For example, feature segment 72 corresponds to a bifurcation point where a particular side branch intersects with the main target vessel. In Figure 9, feature segment 72 appears within a grid square 73. The endoscopic camera view in Figure 10 corresponds to the camera position advanced along the tunnel, thereby causing feature segment 72 to appear within another grid square 74. This image processing estimates the relative distances between various traceable feature segments to compile a 3D model of the tunnel and the structures within it. Based on this model, a projection image can be determined to create an effective representation (e.g., a map) for simulating the appearance of a tunnel / vascular system for the user through augmented reality overlay.

[0032] Figure 11 provides a diagram illustrating the process for aggregating 3D models and projection map images. Images 1 through 6 of a series of images 75 are shown along with the changing positions of a pair of feature parts, indicated as x and y. The positions of feature parts x and y move from one image to the next. Once a feature part is identified, it is added to the feature part list 76. The locations of the listed feature parts (e.g., compared to the location of an instrument head or distal tip) are tracked and stored in the track list 77. As more and more images are captured along the tunnel (e.g., as the tunnel is created by dissection or while traversing the tunnel for subsequent cutting operations), a more complete picture of the size and shape of the tunnel and structure is recorded by combining the changing positions of numerous different feature parts. A 3D spatial model 78 is obtained by linking the feature parts and estimating their relative distances, and this 3D spatial model 78 mathematically defines the three-dimensional surface within the tunnel. A simulated view of the 3D spatial model is generated as a map 79 for display on an augmented reality display. This map could include projected images of blood vessels and their collateral branches from a viewpoint that would be seen, for example, from the location of an augmented reality display.

[0033] Figure 12 shows an exemplary view field 80 comprising: 1) a map overlay display 81 representing a map projection of a 3D model of the target blood vessel; and 2) a marker 82 indicating the current location of the distal tip portion of the head of an endoscopic vascular sampling instrument. In addition, a real-size / distance marker 83 is superimposed on the view field 80 in relation to the overlay display 81 to indicate the corresponding length of a portion of the 3D model. For example, the marker 83 is a character glyph that allows estimation of the length of the distance from the skin entry incision (e.g., the starting point of dissection) to the current instrument location of the marker 82.

[0034] In some embodiments, the user may be assisted in identifying specific anatomical objects (e.g., specific blood vessels, collateral branches, or connective tissue) based on pattern recognition. As shown in Figure 13, the matrix 85 stores a series of captured images along with the estimated location of the camera that captured each image and / or the estimated location of the detected feature portion. The matrix 85 can further aggregate other sensor data (such as blood flow-related ranging information and effects, such as fractional flow reserve (FFR) or speckle interference) to assist in object recognition. The data from the matrix 85 is applied to the pattern recognition engine 86, which outputs the uniqueness and location of the classified objects to the 3D model 78. For classified detected objects visible in the endoscope camera view at that time, corresponding labels are generated and / or relayed by the 3D model 78 to be presented as glyphs on an augmented reality display in relation to the classified objects that are visible. For example, Figure 14 shows a view field 84 comprising a map overlay display 85 that represents a map projection of a 3D model of the target blood vessel, and an instantaneous endoscopic camera image 86. Structure identification indicators 87 (which indicate the saphenous vein) and 88 (which indicate a side branch) are overlaid on the view field 84, and may consist of letter glyphs along with leader lines or arrows that point to specific structures referenced in the image.

[0035] In some embodiments, the depth of a visible structure or object (e.g., the distance from the distal tip of an endoscopic instrument to the object) is included as an overlay display. The depth can be estimated or measured. When measuring the depth, a distance sensor may be provided at the distal tip of the endoscopic instrument. Figure 15 shows a dissection instrument unit 88 having a transparent, non-sharp tip 89 having a cone shape for forming a tunnel by separating a target blood vessel from the surrounding tissue. Discontinuities 90, such as grooves or protrusions on the inner surface of the tip 89, create a visible circle around the end of the tip 89 when viewed in an endoscopic image, for the user to track the position of the most distal end of the tip 89. The rod portion 91 of the dissection instrument 88 carries one or more distance sensors, such as a time-of-flight (TOF) sensor 92. Furthermore, depth may be sensed using a stereoscopic imager with laterally spaced image sensors 93 and 94 to obtain a stereoscopic image from which depth can be estimated. An active sensor 95, such as a LiDAR, laser, or radar sensor, may be used to measure the range to the object or surface to be identified. A light source 96, such as an LED, is provided at the end of the rod portion 91 to provide illumination for obtaining an endoscopic image.

[0036] Using depth information, an overlay display can be created to assist the user in approaching and treating target tissue (e.g., connective tissue and collateral branches). Figure 16 shows a view field 100 with a map overlay display 101 representing a map projection image of a 3D model of the target blood vessel and an endoscopic video image 102. The video image 102 is enhanced by representing a foreground overlay display 103 and a background overlay display 104 that highlight a pair of visible objects or surfaces and show the relative depth relationship between the visible structures. Two visible structures may be automatically selected by the image processor based on pattern recognition of significant objects (e.g., collateral branches), or the user may identify two points in the video image for evaluation. As shown in Figure 17, a depth indicator may be provided to show the estimated real distance from the distal end of the endoscopic camera lens to at least one visible structure in the instantaneous endoscopic image. Therefore, the view field 105 has an endoscopic video image 106 that includes a depth overlay display 107 (showing a distance of 1.1 cm to a indicated spot on the target blood vessel) and a depth overlay display 108 (showing a distance of 1.6 cm to a side branch).

[0037] To further assist the user in understanding the spatial arrangement and size of various feature portions within an endoscopic image, the sampling instrument of the present invention may have absolute reference markings that can be visualized by the user via the endoscopic camera view. Figure 18 shows an endoscopic video image 110 represented on an augmented reality display, where the endoscopic instrument (e.g., a V-cutter) has a visible surface engraved with a linear distance measuring scale. A series of tick marks 111 are provided on the vascular maintenance portion, and a series of tick marks 112 are provided on the cutter / electrode portion. The tick marks 111 and 112 follow any convenient distance sequence identified by the user. The tick marks 111 and 112 can be visually compared by the user using nearby structures in the endoscopic video image, thereby enabling the user to estimate the size of any feature portion in the image.

[0038] Furthermore, depth information helps to automatically determine the size of selected (and / or automatically recognized) structures in the image. As shown in Figure 19, instantaneous endoscopic video images 115 represented on an augmented reality display can be augmented using a real-size indicator 116. It may be particularly beneficial for the user to know the diameter of the side branches in order to enable the application of the appropriate amount of cutting / cauterizing energy to the side branches. The real-size indicator 116 is associated with the side branch identified by the pointer. The accompanying letters indicate the diameter measurement ("D") and the actual diameter distance (e.g., "2.2 mm"), which can be automatically estimated by the image processor based on the measured depth from i) the endoscopic lens and ii) the apparent size in the endoscopic image.

[0039] In some embodiments, a pre-mapping representation of the target vessel to be harvested is obtained before dissecting the endoscopic tunnel. The pre-mapping representation can be used in assembling a three-dimensional model and / or in guiding dissection instruments during tunnel dissection. The pre-mapping representation may be obtained by percutaneous sensing of the location of the target vessel. Percutaneous sensing may consist of ultrasound imaging. The pre-mapping representation may be used to define the dissection route. During dissection, an image processor can compare the current location of the distal tip with the dissection route. Whenever the discrepancy between the current location of the distal tip and the dissection route exceeds a predetermined threshold, the augmented reality display can display a warning to the user. Sound or other warnings may also be generated. Warnings displayed on the display may further include instructions for correcting the discrepancy.

[0040] Figure 20 shows a patient's lower limb 120 being examined using an ultrasound probe 121 connected to an ultrasound control unit / display 122. The user examines the limb 120 to identify a target vessel (e.g., saphenous vein). A marker 123 is used to draw a trace 124 on the skin of the limb 120 to follow the path of the target vessel when the target vessel is detected by ultrasound. The ultrasound control unit 122 may be electrically connected to a processor of a vascular sampling system to provide data characterizing the location of the target vessel, which should be used by the vascular sampling system to generate a 3D model.

[0041] A pre-mapping representation may be used to guide the dissection as shown in Figure 21. An incision 125 is made for inserting the distal tip of a dissection instrument or dissection instrument 126. The user wears an augmented reality display 127 carrying a video camera 128 configured to capture images including the lower limb 120, trace 124, and dissection instrument 126. The captured images are transmitted to an image processor / control device (not shown), which is programmed to estimate the subcutaneous location of the distal tip of the dissection instrument 126 by utilizing the orientation of the dissection instrument 126 in the image and by calculating the ratio of the length of the dissection instrument 126 that has passed through the incision 125 to the total length of the dissection instrument 126. Figure 22 shows an endoscopic view 130 in an example during dissection of a tunnel around a target vessel 131 with a side branch 132. The dissection instrument has a transparent tip with markings or discontinuities within the inner surface shape of the transparent tip, which make an "eye" marking 133 in the image. Figure 23 shows the view field 135 of the augmented reality display during dissection, showing an overlay display 136 of the 3D map projection image and an endoscope camera view 137. When the movement of the distal tip of a dissection instrument is tracked, a warning overlay display 140 is generated whenever the direction or movement of the dissection instrument deviates from the path of the target blood vessel. The overlay display 140 may indicate that the direction of dissection is deviating from the desired path. Based on the direction of the deviation, the overlay display 140 may include a correction icon 141 to inform the user of the direction to be taken to reduce the deviation.

[0042] Figure 24 illustrates the method of the present invention, in which pre-mapping of the target vascular pathway is obtained in step 150 using remote sensing (e.g., percutaneous sensing such as ultrasound). In step 151, the pathway is traced on the patient's skin and / or the coordinates of the three-dimensional pathway geometry of the vascular are stored in a processor (e.g., computer memory). During tunnel dissection, external images showing endoscopic instruments (e.g., dissection instruments), the patient / subject, and the traced pathway are captured in step 152.

[0043] Using the captured image, the control device estimates the location of the head (distal end) of the dissection instrument in step 153, for example, based on the part of the dissection instrument that remains visible outside the patient. The estimated head location is compared in step 154 ​​to the pre-mapping path. In step 155, a check is performed to determine whether the head location is within the selected boundary (e.g., within a threshold distance) of the pre-mapping path. If it is within the boundary, the monitoring continues back to step 152. If it is not within the desired boundary, a warning is provided to the user in step 156. [Explanation of symbols]

[0044] 10 patients 11 Saphenous vein 12 Lower Limbs 13. Incision site 14 Puncture wound 16 Dissection Instrument Unit 17 Dissection instrument tip 18 handle 19 Longitudinal rods 20 receivers 21 Inhalation tube 22 Sampling device cutting unit 23 Handle 24. Long, slender sleeve member 25 Endoscope receiver 26 V-Keeper 27 V-Cutter 28 V-Key Buttons 29 V-Cutter Extension Button 30 Inhalation device tube 31 Bipolar Code 32 Endoscopy Units 33 Insertion part 34 End adapters 35 light guide ports 36 Dissection Instrument Unit 37 Longitudinal rods 38 Handle section 39 Non-sharp tips of dissection instruments 40 Trocar 41 Holding part 43. Living tissue 44 Work Tunnel 45 Target Vessel 46 Peripheral tissues 47 Cables 48 displays 50 side branches 51 patients 52 Incision site 53 Collection equipment 54. Source of inhaled gases 55 Power supply 56 Light source 57 Image Processors 58 Conventional displays 59 Selector 60 Augmented Reality Display 61 Viewfield 62 Limbs 63 Endoscopic Instruments 64 Overlay Display 65 Overlay Display 70 Virtual Grids 71 Virtual Grid 72 Feature section 73 Grid Square 74 Grid Square 75 Series of Images 76 Feature Sub-list 77 Tracking List 78 3D spatial models 79 Maps 80 Viewfield 81. Map Overlay Display 82 Marker 83 Actual size / distance marker 84 Viewfield 85 Matrix 86 Pattern Recognition Engine 87. Structural Identification Indicator 88. Structural Identification Indicator 89 Non-sharp tip 90 Discrete section 91 Bar part 92 TOF sensor 93 Image Sensor 94 Image Sensors 95 Active Sensors 96 light source 100 Viewfield 101 Map Overlay Display 102 Video Images 103 Foreground overlay display 104 Overlaying background display 105 Viewfield 106 Endoscopic video images 107 Depth Overlay Display 108 Depth Overlay Display 110 Video Images 111 divisions 112 divisions 115 Instantaneous Endoscopic Images 116 Actual Size Indicator 120 Lower Limbs 121 Ultrasound probe 122 Ultrasonic control device 123 Marker 124 traces 125 Incision site 126 Anatomy instruments 127 Augmented Reality Displays 128 video cameras 131 Target Vessel 132 Side branches 133 Marking 135 Viewfield 136 Overlay Display 137 Endoscopic Camera View 140 Overlay Display 141 Edit icon

Claims

1. 1. A vessel harvesting system comprising: an endoscopic camera of the endoscopic instrument that captures images from the distal tip of the endoscopic instrument within a dissection tunnel around the vessel to be harvested; an image processor that assembles a three-dimensional model of the tunnel from a series of images captured by the endoscopic camera; a display coupled to the image processor that displays: 1) an integrated map representing the three-dimensional model; and 2) a marker associated with the map indicating the current location of the distal tip. A vessel harvesting system comprising:

2. The system of claim 1 , wherein the display comprises an augmented reality display.

3. 2. The system of claim 1, wherein the image processor is configured to extract feature points using a comparison of change positions of the detected features in the series of images to determine an estimated distance between the detected features in the three-dimensional model.

4. The system of claim 1 , wherein the display further represents at least one size marker associated with the map on the display to indicate a corresponding length of at least a portion of the three-dimensional model.

5. The system of claim 1 , wherein the display further uses the captured image from the image processor to present an instantaneous endoscopic image.

6. The system of claim 5 , wherein the display further represents a depth indicator of at least one visible structure within the instantaneous endoscopic image.

7. The system of claim 6 , wherein the spatial distance to the visible structure is determined based on stereo images of the visible structure captured by the camera from two different imaging locations.

8. The system of claim 6 , further comprising a distance sensor for determining a distance to the visible structure, the depth indicator corresponding to the determined distance.

9. The system of claim 8 , wherein the ranging sensor comprises a 3D camera that simultaneously captures stereoscopic images.

10. The system of claim 6 , wherein the depth indicator identifies a relative depth relationship between visible structures, one visible structure being designated as a foreground structure and another visible structure being designated as a background structure.

11. The system of claim 5 , wherein the display further represents an actual size indicator of at least one visible structure within the instantaneous endoscopic image.

12. The system of claim 5 , wherein the display further represents a structural identification indicator of at least one visible structure within the instantaneous endoscopic image.

13. The system of claim 12 , wherein the image processor utilizes image analysis to identify side branches, and the structural identification indicator comprises a side branch indicator for highlighting the identified side branches.

14. the display comprises an augmented reality display, and the system comprises: a selector coupled to the image processor for receiving screen update commands generated by a wearer of the augmented reality display while the endoscopic instrument is held by the wearer, the image processor selecting data to display on the augmented reality display in response to the screen update commands. The system of claim 1 further comprising:

15. 2. The system of claim 1, wherein the image processor is configured to receive a pre-mapped representation of the vessel to be sampled for use in constructing the three-dimensional model, the pre-mapped representation being obtained by percutaneous sensing.

16. The system of claim 15 , wherein the transcutaneous sensing comprises ultrasound imaging.

17. 16. The system of claim 15, wherein the pre-mapped representation defines an anatomical path, the image processor compares a current location of the distal tip to the anatomical path, and the display presents a warning to a user of the vessel harvesting system if a deviation between the current location of the distal tip and the anatomical path exceeds a predetermined threshold.

18. A method of operating a vascular harvesting system having an image processor, comprising: receiving, by the image processor, a series of images from an endoscopic camera of an endoscopic instrument; said image processor assembling a three-dimensional model from said series of images; the image processor presenting on a display an integrated map representing the three-dimensional model; the image processor representing on the display a marker positioned relative to the map, the marker configured to indicate a location of a distal tip of the endoscopic instrument; A method of operation comprising:

19. The method of claim 18 , wherein the display comprises an augmented reality display.

20. said step of assembling said three-dimensional model further comprising: said image processor analyzing said series of images to extract at least a first feature; the image processor tracking a change location of the first feature in subsequent images of the series of images; the image processor extracting a second feature in one of the subsequent images and tracking a change location of the second feature in additional subsequent images; the image processor connecting the extracted features according to the change positions to define the three-dimensional model; 20. The method of claim 18, comprising:

21. The method of claim 20, wherein the image processor uses transcutaneous sensing of a vessel to pre-map the anatomical path of the vessel; the image processor comparing the location of the distal tip to the anatomical path; the image processor presenting a warning on the display if a deviation between the location of the distal tip and the anatomical path exceeds a predetermined threshold; 20. The method of claim 18, further comprising:

22. The image processor representing at least one size marker associated with the map on the display to indicate a corresponding length of at least a portion of the three-dimensional model.

20. The method of claim 18, further comprising:

23. The method of claim 22, wherein the image processor displays an instantaneous endoscopic image on the display. the image processor representing on the display an actual size indicator of at least one visible structure within the instantaneous endoscopic image; 20. The method of claim 18, further comprising:

24. The method of claim 23, wherein the image processor displays an instantaneous endoscopic image on the display. the image processor representing on the display a depth indicator of at least one visible structure within the instantaneous endoscopic image; 20. The method of claim 18, further comprising:

25. 25. The method of claim 24, wherein the depth indicator identifies a relative depth relationship between at least two visible structures, one visible structure being indicated as a foreground structure and another visible structure being indicated as a background structure.

26. The method of claim 25, wherein the image processor displays an instantaneous endoscopic image on the display. the image processor utilizing image analysis to identify side branches to the vessel; the image processor presenting a side branch indicator to highlight the identified side branch on the display of the instantaneous endoscopic image; 20. The method of claim 18, further comprising:

27. the display comprises an augmented reality display, and the operating method comprises: the image processor selecting data to be displayed on the augmented reality display in response to a screen update command.

20. The method of claim 18, further comprising:

Images