Computer-assisted annotation of subsurface structures onto a surface

Abstract

The present disclosure describes a system and method for annotating subsurface structures onto a surface of a surface structure. The system receives first imaging data captured using a first imaging device of a first imaging modality, the first imaging data showing the surface of the surface structure, receives second imaging data captured using a second imaging device of a second imaging modality, the second imaging data showing a first subsurface structure, determines a first point on the surface based at least on where on the surface is closest to the first subsurface structure, generates a first projection of the first subsurface structure onto the surface at the first point, and displays, on a display device, the first projection at the first point on the surface of the surface structure.

Description

Attorney Docket No. P06633-WO (153360) COMPUTER-ASSISTED ANNOTATION OF SUBSURFACE STRUCTURES ONTO A SURFACE CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit of co-pending United States provisional patent application Serial No. 63 / 572,406 filed April 1, 2024. The aforementioned related patent application is herein incorporated by reference in its entirety. TECHNICAL FIELD

[0002] The present disclosure relates generally to medical systems. Specifically, the present disclosure relates to using a medical system to annotate a subsurface structure onto a surface. BACKGROUND

[0003] During surgery, it is important that surgeons adequately visualize the anatomy of a subject. As surgical targets often lie in the subsurface, a range of visualization techniques have been investigated. This includes visualization of nerves, blood vessels, and tumors. Ultrasound has been used to provide registered visualization of tumors in robotic surgery, but ultrasound suffers from noise, poor sensitivity and specificity, and is primarily useful for locating large tumors buried deep below the surface, or guiding instruments to them. As such, it is difficult to clearly visualize subsurface structures located at different depths, even by employing ultrasound with a stereoscopic endoscope. A stereoscopic endoscope uses stereo cameras. Stereo cameras include two cameras (e.g., a left camera and a right camera). A triangulation technique may be performed using the images from the two cameras to determine the depths of different points in the field-of-view to assist a medical system in projecting a rendering or annotation of the subsurface structure onto a surface of an anatomical structure. However, the rendering or annotation of the subsurface structure is usually projected on a non-optimal location on the surface of the anatomical structure. SUMMARY

[0004] The present disclosure describes a system and method for annotating subsurface structures onto a surface of a surface structure. According to anAttorney Docket No. P06633-WO (153360) embodiment, a system for or annotating subsurface structures onto a surface of a surface structure includes a memory and a controller communicatively coupled to the memory. The system receives first imaging data captured using a first imaging device of a first imaging modality, the first imaging data showing the surface of the surface structure, receives second imaging data captured using a second imaging device of a second imaging modality, the second imaging data showing a first subsurface structure, determines a first point on the surface based at least on where on the surface is closest to the first subsurface structure, generates a first projection of the first subsurface structure onto the surface at the first point, and displays, on a display device, the first projection at the first point on the surface of the surface structure.

[0005] According to another embodiment, a method for annotating subsurface structures onto a surface of a surface structure includes receiving first imaging data captured using a first imaging device of a first imaging modality, the first imaging data showing the surface of the surface structure, receiving second imaging data captured using a second imaging device of a second imaging modality, the second imaging data showing a first subsurface structure, determining a first point on the surface based at least on where on the surface is closest to the first subsurface structure, generating a first projection of the first subsurface structure onto the surface at the first point, and displaying, on a display device, the first projection at the first point on the surface of the surface structure.

[0006] The foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1A illustrates an example medical system.

[0008] Figures 1B and 1C illustrate example components in the system of Figure 1A.

[0009] Figure 2A illustrates an example medical system.Attorney Docket No. P06633-WO (153360)

[0010] Figures 2B and 2C illustrate an example medical instrument system in the system of Figure 2A.

[0011] Figure 3 illustrates an example operation performed by the system of Figures 1A or 2A.

[0012] Figure 4 illustrates an example operation performed by the system of Figures 1A or 2A.

[0013] Figure 5 illustrates an example operation performed by the system of Figures 1A or 2A.

[0014] Figure 6 illustrates an example operation performed by the system of Figures 1A or 2A.

[0015] Figure 7 is a flowchart of an example method performed by the system of Figures 1A or 2A.

[0016] Figure 8 illustrates an example operation performed by the system of Figures 1A or 2A.

[0017] Figure 9 illustrates an example operation performed by the system of Figures 1A or 2A.

[0018] Figure 10 illustrates an example operation performed by the system of Figures 1A or 2A.

[0019] Figures 11A through 11C illustrate an example operation performed by the system of Figures 1A or 2A.

[0020] Figure 12 illustrates an example operation performed by the system of Figures 1A or 2A.

[0021] Figure 13 illustrates an example operation performed by the system of Figures 1A or 2A.

[0022] Figure 14 illustrates an example operation performed by the system of Figures 1A or 2A.Attorney Docket No. P06633-WO (153360)

[0023] Figure 15 illustrates an example operation performed by the system of Figures 1A or 2A.

[0024] Figure 16 is a flowchart of an example method performed by the system of Figures 1A or 2A.

[0025] Figure 17 illustrates an example operation performed by the system of Figures 1A or 2A. DETAILED DESCRIPTION

[0026] Endoscopy is a medical procedure to visualize the human body’s internal organs or natural cavities. The endoscope is a rigid or flexible tubular device allowing a direct view into the body. An endoscopic system can be built up as a purely optical device containing lenses, transparent rods or fibers, or in combination with integrated or add-on cameras. The endoscope tip can be inserted through a small access created by incision or through the natural lumina of the body. A light source on the endoscope provides sufficient illumination to the examined cavity. Cameras on the endoscope provide a field of view that can be visualized on a screen and recorded for later diagnosis or documentation. Two cameras (e.g., a left camera and a right camera) can be placed side-by-side on the endoscope to form a stereo camera, which provides a stereo view and allows for depth perception. For example, the two cameras may capture images from slightly different perspectives, similar to how human eyes perceive depth. These dual images are then processed to create a 3D image or video, providing surgeons and medical professionals with improved spatial awareness during procedures. For example, a triangulation technique may be performed using the images from the two cameras to determine the distances between different points in the view and the camera (e.g., the depths of the points) and the distances between these points, which allows surgeons to better understand the relative distances between different structures or instruments in the view. This enhanced spatial awareness may be beneficial for performing delicate and precise maneuvers.

[0027] In existing systems, anatomical structures below a surface are projected on a non-optimal location of the surface that is along the perspective view or line of sight of the imaging device (e.g., line of sight of the two cameras of the imaging device). Such perspective views may not indicate, to the surgeon, the optimal path to accessAttorney Docket No. P06633-WO (153360) the subsurface structure. Thus, such projections on non-optimal locations of the surface do not provide the surgeon with the most desirable surgical access points. However, enhanced spatial awareness can be achieved by projecting the anatomical structure on the surface that is physically closest or nearest to the subsurface structure. This new projection site or location provides a more optimal path for the surgeon to access the subsurface structure of interest.

[0028] The present disclosure describes a medical system that annotates subsurface structures onto a surface of an anatomical structure. The medical system creates a projection (e.g., an outline, colored shape, etc.) of the subsurface structure onto the surface of the anatomical structure. The location on the surface of the anatomical structure for the projection is determined to be the location on the surface that is physically closest or nearest to the subsurface structure. A depth map and a 3D reconstruction of the anatomical structure and the subsurface structure are used to determine the point closest to the subsurface structure. A projection representing the subsurface structure is displayed or projected on the closest point on the surface of the anatomical structure. A depth of the subsurface structure is indicated using a characteristic of the projection (e.g., color, blur, transparency, etc.). In one example, the appearance of the projection (e.g., color, transparency, etc.) may change to indicate the distance of a medical instrument to the projection.

[0029] In certain embodiments, the system provides several technical advantages. For example, the system may provide better access points to subsurface structures for the surgeon. The more accurate access points allow a surgeon to precisely locate anatomical structures, lesions, or target areas within the patient's body. This precision is important for performing delicate and targeted surgical procedures, which minimizes errors. Surgeons can also rely on the information to navigate through complex anatomical structures and to avoid unintentional damage to surrounding tissues. Surgeons can use this information to plan and execute procedures with a high level of confidence, ensuring that critical structures are identified and treated appropriately. More accurate spatial information also helps guide the placement of instruments and allows for effective navigation through narrow and confined spaces, which contributes to the efficient use of surgical resources, including time and equipment. Thus, theAttorney Docket No. P06633-WO (153360) system may improve the health and safety of a patient and reduce recovery times and postoperative complications.

[0030] In some examples, one or more components of a medical system may be implemented as a computer-assisted surgical system. It is understood, however, that the medical system may be implemented in any type of medical system (e.g., digital fiducial systems, anatomy detection systems, and clinical guidance systems). Figures 1A and 2A show example computer-assisted surgical systems 100 and 200 that may implement some of the features described herein.

[0031] The surgical system 100 includes a manipulator assembly 102, a user control apparatus 104, and an auxiliary apparatus 106, all of which are communicatively coupled to each other. The surgical system 100 is utilized by a medical team to perform a computer-assisted medical procedure or other similar operation on a body of a patient 108 or on any other body as may serve a particular implementation. The medical team includes a first user 110-1 (such as a surgeon for a surgical procedure), a second user 110-2 (such as a patient-side assistant), a third user 110-3 (such as another assistant, a nurse, a trainee, etc.), and a fourth user 110- 4 (such as an anesthesiologist for a surgical procedure), all of whom are collectively referred to as users 110, and each of whom may control, interact with, or otherwise be a user of the surgical system 100. More, fewer, or alternative users may be present during a medical procedure as may serve a particular implementation. For example, team composition for different medical procedures, or for non-medical procedures, may differ and include users with different roles.

[0032] Although Figure 1A illustrates an ongoing minimally invasive medical procedure such as a minimally invasive surgical procedure, it will be understood that the surgical system 100 may similarly be used to perform open medical procedures or other types of operations. For example, operations such as exploratory imaging operations, mock medical procedures used for training purposes, and / or other operations may also be performed.

[0033] The manipulator assembly 102 includes one or more manipulator arms 112 (e.g., manipulator arms 112-1 through 112-4) to which one or more instruments may be coupled. The instruments are used for a computer-assisted surgical procedure onAttorney Docket No. P06633-WO (153360) the patient 108 (e.g., by being at least partially inserted into the patient 108 and manipulated within the patient 108). While the manipulator assembly 102 is depicted and described herein as including four manipulator arms 112, the manipulator assembly 102 may include a single manipulator arm 112 or any other number of manipulator arms as may serve a particular implementation. Although the example of Figure 1 illustrates the manipulator arms 112 as robotic manipulator arms, one or more instruments may be partially or entirely manually controlled, such as by being handheld and controlled manually by a person. These partially or entirely manually controlled instruments are used in conjunction with, or as an alternative to, computer- assisted instrumentation that is coupled to the manipulator arms 112.

[0034] During the medical operation, the user control apparatus 104 facilitates tele- operational control by the user 110-1 of the manipulator arms 112 and instruments attached to the manipulator arms 112. To this end, the user control apparatus 104 provides the user 110-1 with imagery of an operational area associated with the patient 108 as captured by an imaging device. The manipulator arms 112 or any instruments coupled to the manipulator arms 112 mimic the dexterity of the hand, wrist, and fingers of the user 110-1 across multiple degrees of freedom of motion. In this manner, the user 110-1 intuitively performs a procedure (e.g., an incision procedure, a suturing procedure, etc.) using one or more of the manipulator arms 112 or any instruments coupled to the manipulator arms 112.

[0035] The auxiliary apparatus 106 includes one or more computing devices that perform auxiliary functions in support of the procedure, such as providing insufflation, electrocautery energy, illumination or other energy for imaging devices, image processing, or coordinating components of the surgical system 100. The auxiliary apparatus 106 includes a display monitor 114 that displays one or more user interfaces, or graphical or textual information in support of the procedure. In some instances, the display monitor 114 is a touchscreen display that provides user input functionality. Augmented content provided by a region-based augmentation system may be similar to, or differ from, content associated with the display monitor 114 or one or more display devices in the operation area (not shown).

[0036] The manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 are communicatively coupled one to another in any suitable manner.Attorney Docket No. P06633-WO (153360) The manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 may be communicatively coupled by way of control lines 116, which represent any wired or wireless communication link as may serve a particular implementation. To this end, the manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, and so forth.

[0037] Figure 1B illustrates an example manipulator assembly 102. As seen in Figure 2A, the manipulator assembly 102 includes a base 118, a manipulator arm 112- 1, a manipulator arm 112-2, a manipulator arm 112-3, and a manipulator arm 112-4. Each manipulator arm 112-1, 112-2, 112-3, and 112-4 is pivotably coupled to the base 118. Although the base 118 may include casters to allow ease of mobility, in some embodiments, the manipulator assembly 102 is fixedly mounted to a floor, ceiling, operating table, structural framework, or the like.

[0038] In a typical procedure, two of the manipulator arms 112-1, 112-2, 112-3, or 112-4 hold surgical instruments and a third holds a stereo endoscope. The remaining manipulator arms are available so that other instruments may be introduced at the work site. Alternatively, the remaining manipulator arms may be used for introducing another endoscope or another image capturing device, such as an ultrasound transducer, to the work site.

[0039] Each of the manipulator arms 112-1, 112-2, 112-3, and 112-4 are formed of links that are coupled together and manipulated through actuatable joints. Each of the manipulator arms 112-1, 112-2, 112-3, and 112-4 may include a setup arm and a device manipulator. The setup arm positions its held device so that a pivot point occurs at its entry aperture into the patient. The device manipulator may then manipulate its held device so that the held device may be pivoted about the pivot point, inserted into and retracted out of the entry aperture, and rotated about its shaft axis. Each of the manipulator arms 112-1, 112-2, 112-3, and 112-4 may include sensors (e.g., joint sensors, position sensors, accelerometers, etc.) that detect or track movement of the manipulator arms 112-1, 112-2, 112-3, and 112-4. For example, these sensors may detect how far or how quickly a manipulator arm 112-1, 112-2, 112-3, or 112-4 moves in a certain direction.Attorney Docket No. P06633-WO (153360)

[0040] Figure 1C illustrates an example user control apparatus 104. The user control apparatus 104 includes a stereo vision display 120 so that the user may view the surgical work site in stereo vision from images captured by the stereoscopic camera of the manipulator assembly 102. Left and right eyepieces 122 and 124 are provided in the stereo vision display 120 so that the user may view left and right display screens inside the display 120 respectively with the user's left and right eyes. While viewing typically an image of the surgical site on a suitable viewer or display, the surgeon performs the surgical procedures on the patient by manipulating master control input devices, which in turn control the motion of robotic instruments.

[0041] The user control apparatus 104 also includes left and right input devices 126 and 128 that the user grasps respectively with his / her left and right hands to manipulate devices (e.g., surgical instruments) being held by the manipulator arms 112-1, 112-2, 112-3, and 112-3 of the manipulator assembly 102 in preferably six or more degrees of freedom (“DOF”). Foot pedals 130 with toe and heel controls are provided on the user control apparatus 104 so the user may control movement and / or actuation of devices associated with the foot pedals.

[0042] A processing device 132 is provided in the user control apparatus 104 for control and other purposes. The processing device 132 performs various functions in the surgical system 100. One function performed by processing device 132 is to translate and transfer the mechanical motion of input devices 126 and 128 to actuate their corresponding joints in their associated manipulator arms 112-1, 112-2, 112-3, and 112-4 so that the surgeon can effectively manipulate devices, such as the surgical instruments. Another function of the processing device 132 is to implement the methods, crosscoupling control logic, and controllers or processors described herein. The auxiliary apparatus 106 may include a processing device 132 that performs the functions or actions described herein. The processing device 132 includes a controller and a memory that perform the functions described herein. The controller may include one or more processors.

[0043] The controller may include any electronic circuitry, including, but not limited to one or a combination of microprocessors, microcontrollers, application specific integrated circuits (ASIC), application specific instruction set processor (ASIP), and / or state machines, that communicatively couples to a memory and controls the operationAttorney Docket No. P06633-WO (153360) of the user control apparatus 104 and / or the auxiliary apparatus 106. The controller may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. The controller may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The controller may include other hardware that operates software to control and process information. The controller executes software stored on a memory to perform any of the functions described herein. The controller controls the operation and administration of the user control apparatus 104 or the auxiliary apparatus 106 by processing information (e.g., information received from the user control apparatus 104, the manipulator assembly 102, the auxiliary apparatus 106, and / or a memory). The controller is not limited to a single processing device and may encompass multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. The controller is considered to perform a set of functions or actions if the multiple processing devices collectively perform the set of functions or actions, even if different processing devices perform different functions or actions in the set.

[0044] Figure 2A illustrates an example computer-assisted surgical system 200 that implements some of the features described herein. The surgical system 200 can be used, for example, in surgical, diagnostic, therapeutic, biopsy, or non-medical procedures. As shown in Figure 2A, the surgical system 200 (which may be a robotically-assisted surgical system) includes one or more manipulator assemblies 202 for operating one or more medical instrument systems 204 in performing various procedures on a patient P positioned on a table T in a medical environment. For example, the manipulator assembly 202 can drive catheter or end effector motion, can apply treatment to target tissue, and / or can manipulate control members. The manipulator assembly 202 can be teleoperated, non-teleoperated, or a hybrid teleoperated and non-teleoperated assembly with select degrees of freedom of motion that can be motorized and / or teleoperated and select degrees of freedom of motion that can be non-motorized and / or non-teleoperated. An operator input system 206, which can be inside or outside of the medical environment, generally includes one or more control devices for controlling the manipulator assembly 202. The manipulatorAttorney Docket No. P06633-WO (153360) assembly 202 supports a medical instrument system 204 and can optionally include a plurality of actuators or motors that drive inputs on the medical instrument system 204 in response to commands from a control system 212. The actuators can optionally include drive systems that when coupled to the medical instrument system 204 can advance the medical instrument system 204 into a natural or surgically created anatomic orifice. Other drive systems can move the distal end of the medical instrument in multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along the x, y, and z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the x, y, and z Cartesian axes). The manipulator assembly 202 can support various other systems for irrigation, treatment, or other purposes. Such systems can include fluid systems (e.g., reservoirs, heating / cooling elements, pumps, and valves), generators, lasers, interrogators, and ablation components.

[0045] The surgical system 200 also includes a display system 210 for displaying an image or representation of the surgical site and a medical instrument system 204. The image or representation is generated by an imaging system 209, which may include an endoscopic imaging system. The display system 210 and operator input system 206 may be oriented so that an operator O can control the medical instrument system 204 and the operator input system 206 with the perception of telepresence. A graphical user interface can be displayable on the display system 210 and / or a display system of an independent planning workstation.

[0046] In some examples, the imaging system 209 includes an endoscopic imaging system with components that are integrally or removably coupled to the medical instrument system 204. However, in some examples, a separate imaging device, such as an endoscope, attached to a separate manipulator assembly can be used with the medical instrument system 204 to image the surgical site. The imaging system 209 can be implemented as hardware, firmware, software, or a combination thereof, which interact with or are otherwise executed by one or more computer processors, which can include the controller 214 of the control system 212.

[0047] The surgical system 200 also includes a sensor system 208. The sensor system 208 may include a position / location sensor system (e.g., an actuator encoder or an electromagnetic (EM) sensor system) and / or a shape sensor system (e.g., anAttorney Docket No. P06633-WO (153360) optical fiber shape sensor) for determining the position, orientation, speed, velocity, pose, and / or shape of the medical instrument system 204. These sensors may also detect a position, orientation, or pose of the patient P on the table T. For example, the sensors may detect whether the patient P is face-down or face-up. As another example, the sensors may detect a direction in which the head of the patient P is directed. The sensor system 208 can also include temperature, pressure, force, or contact sensors, or the like.

[0048] The surgical system 200 can also include a control system 212, which includes at least one memory 216 and at least one controller 214 (which may include a processor) for effecting control between the medical instrument system 204, the operator input system 206, the sensor system 208, and the display system 210. The control system 212 includes programmed instructions (e.g., a non-transitory machine- readable medium storing the instructions) to implement a procedure using the surgical system 200, including for navigation, steering, imaging, engagement feature deployment or retraction, applying treatment to target tissue (e.g., via the application of energy), or the like.

[0049] The control system 212 may further include a virtual visualization system to provide navigation assistance to the operator O when controlling medical instrument system 204 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system can be based upon reference to an acquired pre-operative or intra-operative dataset of anatomic passageways. The virtual visualization system processes images of the surgical site imaged using imaging technology, such as computerized tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like. The control system 212 uses a pre-operative image to locate the target tissue (using vision imaging techniques and / or by receiving user input) and create a pre-operative plan, including an optimal first location for performing treatment. The pre-operative plan can include, for example, a planned size to expand an expandable device, a treatment duration, a treatment temperature, and / or multiple deployment locations.

[0050] The controller 214 is any electronic circuitry, including, but not limited to one or a combination of microprocessors, microcontrollers, application specific integratedAttorney Docket No. P06633-WO (153360) circuits (ASIC), application specific instruction set processor (ASIP), and / or state machines, that communicatively couples to the memory 216 and controls the operation of the control system 212. The controller 214 may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. The controller 214 may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The controller 214 may include other hardware that operates software to control and process information. The controller 214 executes software stored on the memory 216 to perform any of the functions described herein. The controller 214 controls the operation and administration of the control system 212 by processing information (e.g., information received from the manipulator assembly 202, the operator input system 206, and the memory 216). The controller 214 is not limited to a single processing device and may encompass multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. The controller 214 is considered to perform a set of functions or actions if the multiple processing devices collectively perform the set of functions or actions, even if different processing devices perform different functions or actions in the set.

[0051] The memory 216 may store, either permanently or temporarily, data, operational software, or other information for the controller 214. The memory 216 may include any one or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, the memory 216 may include random access memory (RAM), read only memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage device or a combination of these devices. The software represents any suitable set of instructions, logic, or code embodied in a computer-readable storage medium. For example, the software may be embodied in the memory 216, a disk, a CD, or a flash drive. In particular embodiments, the software may include an application executable by the controller 214 to perform one or more of the functions described herein. The memory 216 is not limited to a single memory and may encompass multiple memories contained in the same device or computer or distributed across multiple devices or computers. The memory 216 is considered to store a set of data, operational software, or informationAttorney Docket No. P06633-WO (153360) if the multiple memories collectively store the set of data, operational software, or information, even if different memories store different portions of the data, operational software, or information in the set.

[0052] Figure 2B illustrates an example medical instrument system 204 in the surgical system 200. In some embodiments, the medical instrument system 204 is used in an image-guided medical procedure. For example, the medical instrument system 204 may be used for non-teleoperational exploratory procedures or in procedures involving traditional manually operated medical instruments, such as endoscopy.

[0053] The medical instrument system 204 includes an elongate flexible device 220, such as a flexible catheter or endoscope (e.g., gastroscope, bronchoscope), coupled to a drive unit 222. The elongate flexible device 220 includes a flexible body 224 having a proximal end 226 and a distal end, or tip portion, 228. In some embodiments, the flexible body 224 has an approximately 14–20 millimeter outer diameter. Other flexible body outer diameters may be larger or smaller. The flexible body 224 has an appropriate length to reach certain portions of the anatomy, such as the lungs, sinuses, throat, or the upper or lower gastrointestional region, when the flexible body 224 is inserted into a patient’s oral or nasal cavity.

[0054] The medical instrument system 204 includes a tracking system 230 for determining the position, orientation, speed, velocity, pose, and / or shape of the distal end 228 and / or of one or more segments 232 along the flexible body 224 using one or more sensors and / or imaging devices. The entire length of the flexible body 224, between the distal end 228 and the proximal end 226, is effectively divided into the segments 232. The tracking system 230 is implemented as hardware, firmware, software, or a combination thereof, which interact with or are otherwise executed by one or more computer controllers, which may include the controllers 214 of control system 212.

[0055] The tracking system 230 tracks distal the end 228 and / or one or more of the segments 232 using a shape sensor 234. In some embodiments, the tracking system 230 tracks the distal end 228 using a position sensor system 236, such as an electromagnetic (EM) sensor system. In some examples, the position sensor systemAttorney Docket No. P06633-WO (153360) 236 measures six degrees of freedom (e.g., three position coordinates x, y, and z and three orientation angles indicating pitch, yaw, and roll of a base point) or five degrees of freedom (e.g., three position coordinates x, y, and z and two orientation angles indicating pitch and yaw of a base point).

[0056] The flexible body 224 includes one or more channels 238 sized and shaped to receive one or more medical instruments 240. In some embodiments, the flexible body 224 includes two channels 238 for separate instruments 240, however, a different number of channels 238 can be provided. Figure 2C illustrates an example portion of the medical instrument system 204 of Figure 2B. As seen in Figure 2C, the medical instrument 240 extends through the flexible body 224. In some embodiments, the medical instrument 240 can be used for procedures and aspects of procedures, such as surgery, biopsy, ablation, mapping, imaging, illumination, irrigation, or suction. The medical instrument 240 is deployed through the channel 238 of the flexible body 224 and is used at a target location within the anatomy. The medical instrument 240 includes, for example, image capture devices, biopsy instruments, ablation instruments, catheters, laser ablation fibers, and / or other surgical, diagnostic, or therapeutic tools. The medical tools include end effectors having a single working member such as a scalpel, a blunt blade, a lens, an optical fiber, an electrode, and / or the like. Other end effectors include, for example, forceps, graspers, balloons, needles, scissors, clip appliers, and / or the like. Other end effectors further include electrically activated end effectors such as electrosurgical electrodes, transducers, sensors, imaging devices, and / or the like. The medical instrument 240 is advanced from the opening of the channel 238 to perform the procedure and then retracted back into the channel when the procedure is complete. The medical instrument 240 is removed from the proximal end 226 of the flexible body 224 or from another optional instrument port (not shown) along the flexible body 224. The medical instrument 240 may be used with an image capture device (e.g., an endoscopic camera) also within the elongate flexible device 220. Alternatively, the medical instrument 240 may itself be the image capture device.

[0057] The medical instrument 240 additionally houses cables, linkages, or other actuation controls (not shown) that extend between the proximal and distal ends to controllably bend the distal end of the medical instrument 240. The flexible body 224Attorney Docket No. P06633-WO (153360) also houses cables, linkages, or other steering controls (not shown) that extend between the drive unit 222 and the distal end 228 to controllably bend the distal end 228 as shown, for example, by the broken dashed line depictions 242 of the distal end 228. In some examples, at least four cables are used to provide independent “up- down” steering to control a pitch motion of the distal end 228 and “left-right” steering to control a yaw motion of the distal end 228. In embodiments in which the medical instrument system 204 is actuated by a robotically-assisted assembly, the drive unit 222 can include drive inputs that removably couple to and receive power from drive elements, such as actuators, of the teleoperational assembly. In some embodiments, the medical instrument system 204 includes gripping features, manual actuators, or other components for manually controlling the motion of the medical instrument system 204. The information from the tracking system 230 can be sent to a navigation system 244, where the information is combined with information from the visualization system 246 and / or the preoperatively obtained models to provide the physician or other operator with real-time position information.

[0058] Figures 3 through 18 illustrate example operations performed by a computer system in a medical system (e.g., the surgical system 100 of Figure 1A or the surgical system 200 of Figure 2A). Generally, the computer system (which may be implemented in the user control apparatus 104 and / or the auxiliary apparatus 106 of the surgical system 100 using the processing device 132 and / or in the control system 212 of the surgical system 200 using the controller 214 and the memory 216) determines a more optimal path for the surgeon to access one or more subsurface structures of interest.

[0059] Figure 3 illustrates an example operation 300 performed by the computer system. The computer system receives a first video captured by a camera of an imaging device 310 (e.g., an endoscope). The camera is positioned at the front of the imaging device 310. The video may be captured from the perspective of the front of the imaging device 310. The first imaging data 316 from the video may show the surface 318 of an anatomical structure. The anatomical structure can also be referred to as a surface structure. The computer system also receives second imaging data 326 from a second imaging device 320 (e.g., an ultrasound probe). The secondAttorney Docket No. P06633-WO (153360) imaging data 326 may show a subsurface structure 328 that is beneath the surface 318 of the anatomical structure.

[0060] The first imaging device 310 provides a first imaging modality. For example, the first imaging device 310 may be an endoscope with a camera. The camera provides visual light imaging data, such as a video. The second imaging device 320 provides a second imaging modality that is different from the first imaging modality. For example, the second imaging device may be an ultrasound probe that captures an ultrasound image.

[0061] The first imaging data 316 and the second imaging data 326 are used to determine a point 330 on the surface 318 of the anatomical structure that is closest or nearest to the subsurface structure 328. Once the point 330 is determined, a projection 340 of the subsurface structure 328 is generated at the point 330 of the surface 318. For example, the computer system may determine the point 330 on the surface 318. The computer system may use a three-dimensional (3D) model of the subsurface structure 328 to determine the location or position of the subsurface structure 328 beneath the surface 318. The computer system may then determine the point 330 on the surface 318 that is physically closest to the subsurface structure 328. The computer system then generates the projection 340 of the subsurface structure 328 at the point 330. The boundary or size of the projection 340 may indicate the boundaries of the subsurface structure 328 when viewed along a line that intersects the subsurface structure 328 and the point 330.

[0062] Figure 4 illustrates an example operation 400 performed by the computer system. Generally, Figure 4 shows the computer system generating and / or using a depth map to determine a location of a surface of an anatomical structure. The computer system receives the first video captured by the camera of the imaging device 310. The first imaging data 316 from the video may show the surface 318 of an anatomical structure. The first imaging data 316 may include visual light imaging data 402, such as a video.

[0063] The computer system determines depth measurements of points on the anatomical structure using a depth map 410. The computer system may generate the depth map 410 by capturing and processing information to represent one or moreAttorney Docket No. P06633-WO (153360) distances of points within the imaging or surgical field. For example, the first imaging device 310 may include multiple cameras that capture videos of the anatomical structure from offset perspectives. Using these videos, the computer system can determine the coordinates for points on the anatomical structure and triangulate the depths of the depths of these points. The computer system then generates the depth map 410, which includes the coordinates of the points and their corresponding depths.

[0064] For a point on the surface 318, the computer system determines the coordinates 412 for the point. The coordinates 412 indicate the position 414 of the point on the surface 318 of the anatomical structure. The computer system may also use the depth map 410 to determine the depth corresponding to the coordinates 412. The computer system may then determine this depth as the depth 416 of the point. The computer system may use this process to determine the positions 414 and depths 416 of any number of points on the surface 318.

[0065] Figure 5 illustrates an example operation 500 performed by the computer system. Generally, Figure 5 shows the computer system generating and using a 3D model 510 to determine a location of the subsurface structure 328.

[0066] The computer system receives the second imaging data 326 that may include ultrasound imaging data 404 (e.g., ultrasound images) and may show the subsurface structure 328. The computer system may generate a 3D model 510 of the subsurface structure 328 from the ultrasound imaging data 404.

[0067] The ultrasound imaging data 404 may show the size, location, and / or orientation of the subsurface structure 328 beneath the surface 318 of the anatomical structure. The computer system may generate the 3D model 510 of the subsurface structure 328 with the same size, location, and / or orientation shown in the ultrasound data 404. As a result, the 3D model 510 is the size of the subsurface structure 328 and positioned and oriented as the subsurface structure 328. Thus, the 3D model 510 shows the location, size, and orientation of the subsurface structure 328 beneath the surface 318 of the anatomical structure.

[0068] The computer system may create the 3D model 510 by merging or stitching together two-dimensional (2D) ultrasound images of the subsurface structure 328. Each of the ultrasound images may be a 2D image that shows a slice of the subsurfaceAttorney Docket No. P06633-WO (153360) structure 328. The computer system may apply volumetric reconstruction to the ultrasound images to arrange the slices into a 3D volume. The computer system may then convert the 3D volume into a mesh representation, where each voxel in the volume becomes a vertex in the mesh. The mesh representation forms the 3D model 510.

[0069] Figure 6 illustrates an example operation 600 performed by the computer system. Generally, the computer system performs the operation 600 to generate a projection of the subsurface structure on a point on the surface of the anatomical structure closest to the subsurface structure.

[0070] Because the depth map 410 indicates the depths of points on the surface 318 of the anatomical structure and because the 3D model 510 indicates the location, size, and orientation of the subsurface structure 328, the computer system can use the depth map 410 and the 3D model 510 to determine the point 330 on the anatomical structure that is closest to the 3D model 510. For example, merging the depth map 410 with the 3D model 510 may show the relative positioning of the surface 318 of the anatomical structure and the subsurface structure 328. Using the depths of points in the depth map 410 and the 3D model 510, the computer system can calculate the distances between points on the surface 318 of the anatomical structure and the surface of the 3D model 510. The computer system may then select the point 330 on the surface 318 of the anatomical structure with the smallest such distance as the point 330 on the anatomical structure closest to the subsurface structure 328.

[0071] The computer system may generate a projection 340 of the anatomical structure at the point 330. The boundary or size of the projection 340 may indicate the boundaries of the subsurface structure 328 when viewed along a line that intersects the subsurface structure 328 and the point 330. A depth of the subsurface structure 328 may be indicated using a characteristic of the projection 340 (e.g., color, blur, transparency, etc.). In one example, the appearance of the projection 340 (e.g., color, transparency, etc.) may change to indicate the distance of a medical instrument to the projection 340. Moreover, a color, a transparency, a blur, or a size of the projection 340 at least partly indicates a depth of the subsurface structure 328 from the surface 318 of the surface structure. The color, the transparency, or the blur mayAttorney Docket No. P06633-WO (153360) vary across the projection 340 to indicate that the subsurface structure 328 occupies multiple depths from the surface 318 of the surface structure.

[0072] Figure 7 is a flowchart of an example method 700 performed by the system 100 of Figure 1A or the system 200 of Figure 2A. In particular embodiments, the computer system performs the method 700. By performing the method 700, the computer system implements certain features that assist the user to annotate subsurface structures onto a surface of a surface structure (e.g., an anatomical structure). As a result, the subsurface structure is projected onto a point on the surface of the surface structure closest to the subsurface structure, which may indicate to a user where is the closest entry path to access the subsurface structure (e.g., to successfully treat a health or medical condition of a patient).

[0073] In block 710, the computer system receives first imaging data captured using a first imaging device of a first imaging modality. For example, the first imaging device may be an endoscope with a camera positioned near the surface structure. The first imaging data may include visual light imaging data (e.g., a video) captured by the camera and may show the surface of the surface structure. The computer system uses the first imaging data to generate a depth map in block 720. The depth map may include coordinates for points on the surface of the surface structure. Each coordinate may indicate a position and a depth of a point on the surface.

[0074] In block 730, the computer system receives second imaging data captured using a second imaging device of a second imaging modality. For example, the second imaging device may be an ultrasound probe, and the second imaging data may include ultrasound imaging data (e.g., 2D ultrasound images) that shows a first subsurface structure beneath the surface of the anatomical structure. The computer system uses the second imaging data to generate a 3D model of the first subsurface structure in block 740. The ultrasound imaging data and / or the 3D model may indicate the location of the subsurface structure. The 3D model may also indicate the size and orientation of the subsurface structure.

[0075] In block 750, the computer system determines a first point on the surface of the surface structure that is closest to the first subsurface structure using the depth map and the 3D model. Because the depth map indicates the depths of points on theAttorney Docket No. P06633-WO (153360) surface of the anatomical structure and because the 3D model indicates the location, size, and orientation of the subsurface structure, the computer system can use the depth map and the 3D model to determine the point on the anatomical structure that is closest to the 3D model. For example, merging the depth map with the 3D model may show the relative positioning of the surface of the anatomical structure and the subsurface structure. Using the depths of points in the depth map and the 3D model, the computer system can calculate the distances between points on the surface of the anatomical structure and the surface of the 3D model. The computer system may then select the point on the surface of the anatomical structure with the smallest such distance as the point on the anatomical structure closest to the subsurface structure.

[0076] A distance or depth between the first point on the surface and the first subsurface structure can be referred to as a closest path or nearest path or shortest path or entry path. The nearest path may be misaligned with a viewing axis of the first imaging device. The computer system may adjust the entry path between the first subsurface structure and the first point on the surface of the surface structure to avoid one or more protected structures located between the first subsurface structure and the first point on the surface.

[0077] In block 760, the computer system generates and displays a first projection of the first subsurface structure onto the surface at the first point. The first projection of the first subsurface structure forms a zero-dimensional, one-dimensional, or two- dimensional annotation on the surface of the surface structure. A boundary of the first projection aligns with a boundary of the first subsurface structure or a center line of the subsurface structure. A color, a transparency, a blur, or a size of the first projection at least partly indicates a depth of the first subsurface structure from the surface of the surface structure. The color, the transparency, or the blur may vary across the first projection to indicate that the first subsurface structure occupies multiple depths from the surface of the surface structure.

[0078] Figure 8 illustrates an example operation 800 performed by the computer system. The computer system annotates a subsurface structure 328 onto a point 825 on the surface 318 of an anatomical structure that is in the line of sight of imaging device 310 (e.g., the endoscope capturing a video of the surface 318) or that of the user. For example, the computer system may determine the point 825 as the point onAttorney Docket No. P06633-WO (153360) the surface 318 that is on a line (e.g., a viewing axis) between the imaging device 310 and the subsurface structure 328. The computer system may use the 3D model 510 of the subsurface structure 328 to determine the location or position of the subsurface structure 328 beneath the surface 318. The computer system may then determine the point 825 on the surface 318 that is on the line between the imaging device 310 and the subsurface structure 328. The computer system then generates the projection 826 of the subsurface structure 328 at the point 825. The boundary or size of the projection 826 may indicate the boundaries of the subsurface structure 328 when viewed from the imaging device 310. If the imaging device 310 moves, the computer system may move the point 825 and the projection 826 to maintain the point 825 and the projection on the line between the imaging device 310 and the subsurface structure 328. The computer system may also adjust the size of the projection 826 for the change in perspective of the subsurface structure 328.

[0079] The rendering or annotation of the subsurface structure 328 in this manner, however, may result in the projection being on a non-optimal location on the surface 318 of the anatomical structure. For example, this perspective view of the subsurface structure 328 may not indicate a shortest path from the surface 318 to the subsurface structure 328.

[0080] Figure 9 illustrates an example operation 900 performed by the computer system. The computer system annotates a subsurface structure 328 onto a point 330 on the surface 318 of an anatomical structure, which may be the point 330 on the surface 318 that is closest to the subsurface structure 328. As discussed previously, the computer system may have used the depth map 410 and the 3D model 510 of the subsurface structure 328 to determine the point 330 on the surface 318 with the shortest distance to the subsurface structure 328. The computer system then generates the projection 911 of the subsurface structure 328 at the point 330 (which may or may not be aligned with the viewing axis of the imaging device 310). The boundary or size of the projection 911 may indicate the boundaries of the subsurface structure 328 when viewed along a line that intersects the subsurface structure 328 and the point 330.

[0081] Boundary lines 916 and 918 are shown to visualize how the boundary of the projection 911 aligns with the boundaries of the subsurface structure 328. TheAttorney Docket No. P06633-WO (153360) boundary lines 916 and 918 may extend from opposed edges or surfaces of the boundary of the projection 911. Because the point 330 is closest to the subsurface structure 328, the point 330 indicates a shortest surgical path to access the subsurface structure 328. The determination of the point 330, the generation of the projection, and the display of the projection are independent of movement of the imaging device. For example, the point 330 and the positioning of the projection 911 on the point 330 may not change regardless of where the imaging device 310 is positioned or where the imaging device 310 may move.

[0082] The computer system may depict the boundary of the projection 911 as a solid line or a dashed line of varying colors and intensities. The computer system may also depict the projection 911 as a single point (zero-dimensional annotation) or a single line (one-dimensional annotation). In Figure 9, the projection 911 is depicted as a two-dimensional annotation with a solid line. The computer system may adjust or vary the color, transparency, blur, hatching, line style, size, etc. of the projection 911 to indicate any characteristic or information about the subsurface structure 328 (e.g., the depth of the subsurface structure or a distance between a surgical instrument and the subsurface structure).

[0083] Figure 10 illustrates an example operation 1000 performed by the computer system. The computer system creates a heat map for the subsurface structure 328. In particular embodiments, the computer system may display the heat map to provide an indication of where the subsurface structure 328 is located with respect to the surface 318 of the anatomical structure. The heat map may be included as part of the projection 911 on the surface 318 of the anatomical structure, or the heat map may be projected onto the surface 318 along with the projection 911.

[0084] The heat map is a graphical representation that may use a system of color coding to represent different values. The color coding may include other visual attributes, such as, but not limited to, transparency, blur, etc. The heat map of Figure 10 includes a first ring 1010, a second ring 1012, and a third ring 1014. The first ring 1010 is formed directly around the subsurface structure 328 and may be shown with a first color of a first intensity. The second ring 1012 is formed directly around the first ring 1010 and may be shown with the first color having a second intensity. The third ring 1014 is formed directly around the second ring 1012 and may be shown with theAttorney Docket No. P06633-WO (153360) first color having a third intensity. The first intensity may be greater than the second intensity, and the second intensity may be greater than the third intensity. As such, the color intensity may decrease as the distance from the subsurface structure 328 increases. The color intensity provides the user with an indication of a distance of the subsurface structure 328 from the surface 318 of the anatomical structure. Thus, when the user attempts to access the subsurface structure 328 with a surgical instrument, based on the color intensity, the user may infer how far or how close the tip of the surgical instrument is from the subsurface structure 328.

[0085] Figures 11A, 11B, and 11C illustrate example operations 1100A, 1100B, and 1100C performed by the computer system. The computer system annotates a subsurface structure 328 onto the surface 318 of an anatomical structure. In particular embodiments, the computer system may project the subsurface structure 328 onto the surface 318 that is physically closest to the subsurface structure 328 such that the subsurface structure 328 forms a zero-dimensional, one-dimensional, or two- dimensional annotation on the surface of the surface structure.

[0086] In Figure 11A, the computer system projects the subsurface structure 328 as a single point 1110 (a zero-dimensional annotation) on the surface 318. The point 1110 may be positioned on the point 330 on the surface 318 that is closest to the subsurface structure 328.

[0087] In Figure 11B, the computer system projects the subsurface structure 328 as a line 1120 (a one-dimensional annotation) on the surface 318. The line 1120 may be aligned with a line 822 of the subsurface structure 328. The line 822 may extend across a center point of the subsurface structure 328. As a result, the line 1120 may represent or indicate a length or width of the subsurface structure 328. The computer system may depict the line 1120 as a solid line or a dashed line of varying thicknesses to indicate a proximity of the subsurface structure 328 to the surface 318 of the anatomical structure. A dashed line may indicate a first distance between the subsurface structure 328 and the surface 318. A thin solid line may indicate a second distance between the subsurface structure 328 and the surface 318 greater than the first distance. A thick solid line may indicate a third distance between the subsurface structure 328 and the surface 318 greater than both the first and second distances.Attorney Docket No. P06633-WO (153360)

[0088] In Figure 11C, the computer system projects the subsurface structure 328 using a boundary line 1130 on the surface 318 (a two-dimensional annotation). The boundary line 1130 may be aligned with the boundary line 1135 of the subsurface structure 328. These boundary lines may also be referred to as perimeter lines. As such, the boundary line 1130 may represent a perimeter of the subsurface structure 328. The computer system may depict the boundary line 1130 as a solid line or a dashed line of varying thicknesses to indicate a proximity of the subsurface structure 328 to the surface 318 of the anatomical structure. A dashed line may indicate a first distance between the subsurface structure 328 and the surface 318. A thin solid line may indicate a second distance between the subsurface structure 328 and the surface 318 greater than the first distance. A thick solid line may indicate a third distance between the subsurface structure 328 and the surface 318 greater than both the first and second distances.

[0089] In one example, the user may preselect how to project the annotation of the subsurface structure 328. The user may select whether to present the subsurface structure 328 as a zero-dimensional, one-dimensional, or two-dimensional annotation on the surface 318 of the anatomical structure.

[0090] Figure 12 illustrates an example operation 1200 performed by the computer system. The computer system determines the points 1205 in the projection on the surface 318 of the anatomical structure. The computer system also determines the depths 1201 of the points on the subsurface structure corresponding to the points 1205. The computer system may then set different colors 1210, transparencies 1220, or blurs 1230 for the points 1205 based on the depths 1201. As a result, the color 1210, transparency 1220, or blur 1230 of a point of the points 1205 in the projection may indicate a depth 1201 of the point on the subsurface structure corresponding to that point 1205 (or the distance between the point 1205 and the corresponding point on the subsurface structure). The computer system may use any number of colors, transparency levels, or blur levels to indicate any number of depths 1201. The user may select whether to use different colors, different transparency levels, or different blur levels to indicate the depths 1201.

[0091] In some instances, the size 1240 of the projection may also indicate the depth of the subsurface structure 328. For example, because the boundaries of theAttorney Docket No. P06633-WO (153360) projection may be aligned with the boundaries of the subsurface structure 328, if the subsurface structure 328 is far beneath the surface 318 of the anatomical structure, the boundaries of the projection may be smaller. If the same subsurface structure 328 is close to the surface 318 of the anatomical structure, the boundaries of the projection would be larger. Thus, the projection size 1240 may also provide an indicator of the depths 1201 of the points on the subsurface structure 328.

[0092] Figure 13 illustrates an example operation 1300 performed by the computer system. The computer system determines a distance 1310 between a surgical instrument and the subsurface structure 328. For example, the computer system may determine the coordinates of the surgical instrument and then determine the distance between these coordinates and the 3D model of the subsurface structure 328. The computer system then adjusts or sets the color, transparency level, blur level, or size of the projection to indicate the distance 1310. For example, as the surgical instrument is brought closer or further away from the subsurface structure, the computer system may change the colors 1210, transparencies 1220, blurs 1230, or size 1240 of the projection to indicate the changing distance between the surgical instrument and the subsurface structure. The user may select whether to use the color, transparency level, blur level, or projection size as the indicator of the distance between the surgical instrument and the subsurface structure.

[0093] Figure 14 illustrates an example operation 1400 performed by the computer system. The computer system may determine that the path between the point 330 on the surface 318 of the anatomical structure and the subsurface structure 328 is obstructed by an object 1405. In response, the computer system may determine a point 1420 on the surface of the anatomical structure that provides the shortest, unobstructed path 1410 to the subsurface structure. The computer system may then generate a projection 1415 at the point 1420 to indicate how to access the subsurface structure 328 while avoiding the object 1405.

[0094] The object 1405 may be a protected structure (e.g., a tumor, a vein, an artery, or another anatomical object) that should be avoided when surgically accessing the subsurface structure 328. The computer system may determine the position, location, and / or orientation of the object 1405 from the ultrasound imaging data. For example, the object 1405 may appear in ultrasound images as the second imagingAttorney Docket No. P06633-WO (153360) device (e.g., an ultrasound probe) is swept over the surface 318 of the anatomical structure. The 3D model 510 may include a model of the object 1405, which indicates the position, location, and / or orientation of the object 1405.

[0095] The computer system may determine, from the model of the object 1405, that the object 1405 is positioned on the path from the point 330 to the subsurface structure 328. As a result, the object 1405 obstructs that path, and accessing the subsurface structure 328 from the point 330 may damage the object 1405. In response, the computer system uses the depth map 410 and the 3D model 510 to determine other points on the surface 318 for which the paths between those points and the subsurface structure are not obstructed by the object 1405. The computer system determines the distances between these points and the subsurface structure 328 and selects the point with the shortest distance as the point 1420. The computer system then generates the projection 1415 of the subsurface structure 328 on the surface 318 at the point 1420. As seen in Figure 14, the path 1410 from the point 1420 to the subsurface structure is unobstructed by the object 1405.

[0096] In some embodiments, the computer system generates a warning or alert when the computer system determines that the object 1405 obstructs the path between the point 330 and the subsurface structure 328. The user may then instruct the computer system to determine an alternate point or access path. In response to that instruction, the computer system determines the point 1420 and generates the projection 1415.

[0097] In particular embodiments, the computer system determines the point 1420 in response to determining that the point 330 is hidden from view or inaccessible by a surgical instrument. Because the point 330 is hidden from view or inaccessible by the surgical instrument, it may be impractical for a user to attempt to access the subsurface structure 328 from the point 330. In response, the computer system determines the point 1420 on the surface 318 of the anatomical structure that is closest to the subsurface structure and is visible and accessible by a surgical instrument. The computer system then generates the projection 1415 at the point 1420 to indicate to the user how to access the subsurface structure 328.Attorney Docket No. P06633-WO (153360)

[0098] Figure 15 illustrates an example operation 1500 performed by the computer system. The computer system allows the user to place markings on the surface 318. In particular embodiments, the computer system may provide the user with the capability to place virtual markers in any location with respect to or adjacent to the projection.

[0099] After the computer system detects the subsurface structure 328, the computer system provides a projection 1525 on the surface 318. A center point 1505 of the projection is also provided. The user may apply different types of markings, markers, or indicators on or adjacent to the projection 1525. For example, the user may apply markers 1510 along the boundary line of the projection 1525. In another example, the user may apply markers 1520 adjacent the center point 1505. The user may apply any type of markings, with any color, transparency, blur, or size in any location in the vicinity of the projection 1525. The markers may represent potential access points or entry paths to guide surgical instruments to the subsurface structure 328. The user may also save such markings for future reference. The markings provide the user with different entry options based on the surgical procedure. In one example surgical procedure, the user may need to access the entire subsurface structure 328 for removal purposes. As such, the markers 1510 provide the boundary or outline of the subsurface structure 328, which may guide the user in cutting or removing the entire subsurface structure 328. In another example surgical procedure, the user may only need to access one central point of the subsurface structure 328 to, e.g., place staples or sutures. As such, the markers 1520 provide central points relative to the subsurface structure where the computer system may allow the user to apply the staples or sutures.

[0100] Figure 16 is a flowchart of an example method 1600 performed by the computer system. By performing the method 1600, the computer system annotates subsurface structures onto a surface of a surface structure. These annotations may indicate visible and accessible points that provide unobstructed paths to the subsurface structure.

[0101] In block 1610, the computer system receives first imaging data captured using a first imaging device of a first imaging modality. For example, the first imaging data may include a video captured by a camera of an endoscope. The first imagingAttorney Docket No. P06633-WO (153360) data may show the surface of the surface structure. The computer system uses the first imaging data to generate a depth map with coordinates indicating positions and depths for points on the surface of the surface structure.

[0102] In block 1620, the computer system receives second imaging data captured using a second imaging device of a second imaging modality. For example, the second imaging data may include ultrasound images captured by an ultrasound probe sweeping the surface of the surface structure. The second imaging data may show a subsurface structure beneath the surface of the surface structure. The computer system uses the second imaging data to generate a 3D model of the subsurface structure. The 3D model may indicate the location, position, and / or orientation of the first subsurface structure.

[0103] In block 1630, the computer system determines a first point on the surface of the surface structure that is closest to the subsurface structure. The computer system may use the depth map and the 3D model to determine the first point. For example, the depth map may indicate the locations and depths of points on the surface of the surface structure, and the 3D model may indicate the location and position of the subsurface structure. Using this information, the computer system can determine the distances between the points on the surface of the surface structure and the subsurface structure. The computer system may select the point on the surface of the surface structure with the shortest distance to the subsurface structure as the first point.

[0104] In block 1640, the computer system generates a first projection of the subsurface structure onto the surface at the first point. The first projection may be a zero-dimensional, one-dimensional, or two-dimensional annotation on the surface of the surface structure. The computer system may set the color, transparency level, or blur level of the first projection to indicate a depth of the subsurface structure from the surface of the surface structure. The color, the transparency level, or the blur level may vary across the first projection to indicate that the subsurface structure occupies multiple depths from the surface of the surface structure.

[0105] In block 1650, the computer system determines that the path from first point to the subsurface structure is obstructed. For example, the computer system mayAttorney Docket No. P06633-WO (153360) determine that an object (e.g., a tumor, vein, artery, or anatomical feature) obstructs the path. To avoid damaging the object, the user may prompt the computer system to calculate a second shortest path or distance to avoid the object.

[0106] In block 1660, the computer system determines a second point on the surface of the surface structure using the depth map and the 3D model. The second point may have the shortest, unobstructed path to the subsurface structure.

[0107] In block 1670, the computer system generates and displays a second projection of the subsurface structure onto the surface of the surface structure at the second point. The second projection indicates the unobstructed path to the subsurface structure.

[0108] Figure 17 illustrates an example operation 1700 performed by the computer system. The computer system determines multiple subsurface structures of interest. In particular embodiments, the computer system may provide various indicators for the subsurface structures.

[0109] The second imaging device 320 captures second imaging data 326 (e.g., ultrasound images). The second imaging data 326 may show a first subsurface structure 328. The computer system determines the first point 330 on the surface of the anatomical structure that is nearest the first subsurface structure 328. The computer system generates and displays the first projection 340 of the first subsurface structure 328 onto the surface at the first point 330. The first projection 340 may be marked or identified by a first indicator 1726. The first indicator 1726 may be a color, a transparency level, a blur level, or a projection size. The user may select whether to use the color, transparency level, blur level, or projection size as the indicator of the distance between the second imaging device 320 and the first subsurface structure 328.

[0110] The second imaging data 326 may also show a second subsurface structure 1730 in the vicinity of the first subsurface structure 328. The computer system determines a second point 1732 on the surface of the anatomical structure that is nearest the second subsurface structure 1730. The computer system generates a second projection 1734 of the second subsurface structure 1730 onto the surface at the second point 1732. The second projection 1734 may be marked or identified by aAttorney Docket No. P06633-WO (153360) second indicator 1736. The second indicator 1736 may be a color, a transparency level, a blur level, or a projection size. The user may select whether to use the color, transparency level, blur level, or projection size as the indicator of the distance between the second imaging device 320 and the second subsurface structure 1730.

[0111] The computer system may simultaneously or concurrently display the first projection 340 and the second projection 1734 on a display 1740. As such, the display 1740 may provide an indication of the first subsurface structure 328 in relation to the second subsurface structure 1730 on the surface of the surface structure. Additionally, the first subsurface structure 328 may be displayed simultaneously with the first projection 340. Similarly, the second subsurface structure 1730 may be displayed simultaneously with the second projection 1734. Moreover, in one example, the first projection 340 may be represented as a first color and the second projection 1734 may be represented as a second color on the surface of the surface structure.

[0112] In another example, the user may switch between the first projection 340 and the second projection 1736. The user may view the first projection 340 independently of the second projection 1736, after generation of the first projection 340 and the second projection 1736.

[0113] In summary, a medical system annotates subsurface structures onto a surface of an anatomical structure. The medical system creates a projection (e.g., an outline, colored shape, etc.) of the subsurface structure onto the surface of the anatomical structure. The location on the surface of the anatomical structure for the projection is determined to be the location on the surface that is physically closest or nearest to the subsurface structure. A depth map and a 3D model of the anatomical structure and the subsurface structure are used to determine the point closest to the subsurface structure. A projection is displayed representing the subsurface structure on the closest point on the surface of the anatomical structure. A depth of the subsurface structure is indicated using a characteristic of the projection (e.g., color, blur, transparency, etc.). In one example, the appearance of the projection (e.g., color, transparency, etc.) may change to indicate the distance of a medical instrument to the projection. The system may provide better access points to subsurface structures for the surgeon. The more accurate access points allow a surgeon to precisely locate anatomical structures, lesions, or target areas within the patient's body. This precisionAttorney Docket No. P06633-WO (153360) is important for performing delicate and targeted surgical procedures, which minimizes errors. Surgeons can also rely on the information to navigate through complex anatomical structures and to avoid unintentional damage to surrounding tissues. Surgeons can use this information to plan and execute procedures with a high level of confidence, ensuring that critical structures are identified and treated appropriately. More accurate spatial information also helps guide the placement of instruments and allows for effective navigation through narrow and confined spaces, which contributes to the efficient use of surgical resources, including time and equipment. Thus, the system may improve the health and safety of a patient and reduce recovery times and postoperative complications.

[0114] This description and the accompanying drawings that illustrate aspects, embodiments, or modules should not be taken as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure other features. Like numbers in two or more figures represent the same or similar elements.

[0115] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0116] Further, the terminology in this description is not intended to be limiting. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like may be used to describe one element’s or feature’s relationship to another element or feature as illustrated in the figures. These spatiallyAttorney Docket No. P06633-WO (153360) relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. Thus, the exemplary term “below” can encompass both positions and orientations of above and below. A device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. And, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0117] Elements described in detail with reference to one embodiment, or module may, whenever practical, be included in other embodiments, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, or application may be incorporated into other embodiments, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or embodiments non-functional, or unless two or more of the elements provide conflicting functions.

[0118] In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.Attorney Docket No. P06633-WO (153360)

[0119] This disclosure describes various devices, elements, and portions of computer-assisted devices and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (three degrees of rotational freedom – e.g., roll, pitch, and yaw). As used herein, the term “shape” refers to a set positions or orientations measured along an element. As used herein, and for a device with repositionable arms, the term “proximal” refers to a direction toward the base of the computer-assisted device along its kinematic chain and “distal” refers to a direction away from the base along the kinematic chain.

[0120] Aspects of this disclosure are described in reference to computer-assisted systems and devices, which may include systems and devices that are teleoperated, remote-controlled, autonomous, semiautonomous, robotic, and / or the like. Further, aspects of this disclosure are described in terms of an embodiment using a medical system, such as the DA VINCI SURGICAL SYSTEM or ION SYSTEM commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that aspects disclosed herein may be embodied and implemented in various ways, including robotic and, if applicable, non-robotic embodiments. Techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperational systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like. Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.Attorney Docket No. P06633-WO (153360)

[0121] Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the disclosure should be limited only by the following claims, and it is appropriate that the claims be construed broadly and, in a manner, consistent with the scope of the embodiments disclosed herein.

Claims

Attorney Docket No. P06633-WO (153360) WHAT IS CLAIMED IS:

1. A system for annotating subsurface structures onto a surface of a surface structure, the system comprising: a memory; and a controller communicatively coupled to the memory, the controller configured to: receive first imaging data captured using a first imaging device of a first imaging modality, the first imaging data showing the surface of the surface structure; receive second imaging data captured using a second imaging device of a second imaging modality, the second imaging data showing a first subsurface structure; determine a first point on the surface based at least on where on the surface is closest to the first subsurface structure; generate a first projection of the first subsurface structure onto the surface at the first point; and display, on a display device, the first projection at the first point on the surface of the surface structure.

2. The system of Claim 1, wherein the first imaging data comprises visual light imaging data or the second imaging data comprises ultrasound imaging data.

3. The system of Claim 1, wherein the controller is further configured to generate a three-dimensional (3D) model of the first subsurface structure based on the second imaging data.

4. The system of Claim 3, wherein the controller is further configured to generate a depth map comprising coordinates for the surface of the surface structure, wherein each coordinate of the depth map indicates a position and a depth for a portion of the surface of the surface structure.Attorney Docket No. P06633-WO (153360) 5. The system of Claim 4, wherein determination of the first point on the surface closest to the first subsurface structure comprises merging the depth map with the 3D model.

6. The system of Claim 1, wherein the first projection of the first subsurface structure forms a zero-dimensional, one-dimensional, or two-dimensional annotation on the surface of the surface structure.

7. The system of Claim 1, wherein a boundary of the first projection aligns with a boundary of the first subsurface structure or a center line of the subsurface structure.

8. The system of Claim 1, wherein a color, a transparency, a blur, a heat map, or a size of the first projection at least partly indicate a depth of the first subsurface structure from the surface of the surface structure.

9. The system of Claim 8, wherein the depth represents a nearest distance between the first subsurface structure and the surface of the surface structure.

10. The system of Claim 8, wherein the color, the transparency, the heat map, or the blur varies across the first projection to indicate that the first subsurface structure occupies multiple depths from the surface of the surface structure.

11. The system of Claim 1, wherein the controller is further configured to determine a distance between an instrument and the first subsurface structure; and wherein a color, a transparency, a blur, a heat map, or a size of the first projection at least partly indicate the determined distance.

12. The system of Claim 1, wherein an entry path between the first subsurface structure and the first point on the surface of the surface structure is misaligned with a viewing axis of the first imaging device.

13. The system of Claim 12, wherein the entry path between the first subsurface structure and the first point on the surface of the surface structure isAttorney Docket No. P06633-WO (153360) determined further to avoid one or more protected structures located between the first subsurface structure and the first point on the surface.

14. The system of Claim 13, wherein the entry path between the first subsurface structure and the first point on the surface of the surface structure is determined based on markings made by a user.

15. The system of Claim 1, wherein the controller is further configured to: determine a second point on the surface based at least on where on the surface is closest to the first subsurface structure, a distance from the first point to the first subsurface structure and the distance from the second point to the first subsurface structure being within a predetermined range; generate a second projection of the first subsurface structure onto the surface at the second point; and display, on the display device, the second projection at the second point on the surface of the surface structure.

16. The system of Claim 15, wherein, when the display of the second projection occurs when the first point is determined to be hidden from view or inaccessible by an instrument.

17. The system of Claim 1, wherein the determination of the first point, the generation of the first projection, and the display of the first projection are independent of movement of the first imaging device.

18. The system of Claim 1, wherein the first subsurface structure is displayed simultaneously with the first projection.

19. The system of Claim 1, wherein the second imaging data further shows a second subsurface structure and the controller is further configured to: determine a second point on the surface based at least on where on the surface is closest to the second subsurface structure; generate a second projection of the second subsurface structure onto the surface at the second point; andAttorney Docket No. P06633-WO (153360) display, on the display device, the second projection at the second point on the surface of the surface structure.

20. The system of Claim 19, wherein the first projection and the second projection are simultaneously displayed on the surface of the surface structure.

21. The system of Claim 20, wherein the first projection is represented as a first color and the second projection is represented as a second color on the surface of the surface structure.

22. A method for annotating subsurface structures onto a surface of a surface structure, the method comprising: receiving first imaging data captured using a first imaging device of a first imaging modality, the first imaging data showing the surface of the surface structure; receiving second imaging data captured using a second imaging device of a second imaging modality, the second imaging data showing a first subsurface structure; determining a first point on the surface based at least on where on the surface is closest to the first subsurface structure; generating a first projection of the first subsurface structure onto the surface at the first point; and displaying, on a display device, the first projection at the first point on the surface of the surface structure.

23. The method of Claim 22, wherein the first imaging data comprises visual light imaging data or the second imaging data comprises ultrasound imaging data.

24. The method of Claim 22, wherein a controller is further configured to generate a three-dimensional (3D) model of the first subsurface structure based on the second imaging data.

25. The method of Claim 24, wherein a controller is further configured to generate a depth map comprising coordinates for the surface of the surface structure,Attorney Docket No. P06633-WO (153360) wherein each coordinate of the depth map indicates a position and a depth for a portion of the surface of the surface structure.

26. The method of Claim 25, wherein determination of the first point on the surface closest to the first subsurface structure comprises merging the depth map with the 3D model.

27. The method of Claim 22, wherein the first projection of the first subsurface structure forms a zero-dimensional, one-dimensional, or two-dimensional annotation on the surface of the surface structure.

28. The method of Claim 22, wherein a boundary of the first projection aligns with a boundary of the first subsurface structure or a center line of the subsurface structure.

29. The method of Claim 22, wherein a color, a transparency, a blur, a heat map, or a size of the first projection at least partly indicate a depth of the first subsurface structure from the surface of the surface structure.

30. The method of Claim 29, wherein the depth represents a nearest distance between the first subsurface structure and the surface of the surface structure.

31. The method of Claim 29, wherein the color, the transparency, the heat map, or the blur varies across the first projection to indicate that the first subsurface structure occupies multiple depths from the surface of the surface structure.

32. The method of Claim 22, wherein a controller is further configured to determine a distance between an instrument and the first subsurface structure; and wherein a color, a transparency, a blur, a heat map, or a size of the first projection at least partly indicate the determined distance.Attorney Docket No. P06633-WO (153360) 33. The method of Claim 22, wherein an entry path between the first subsurface structure and the first point on the surface of the surface structure is misaligned with a viewing axis of the first imaging device.

34. The method of Claim 33, wherein the entry path between the first subsurface structure and the first point on the surface of the surface structure is determined further to avoid one or more protected structures located between the first subsurface structure and the first point on the surface.

35. The method of Claim 34, wherein the entry path between the first subsurface structure and the first point on the surface of the surface structure is determined based on markings made by a user.

36. The method of Claim 22, wherein a controller is further configured to: determine a second point on the surface based at least on where on the surface is closest to the first subsurface structure, a distance from the first point to the first subsurface structure and the distance from the second point to the first subsurface structure being within a predetermined range; generate a second projection of the first subsurface structure onto the surface at the second point; and display, on the display device, the second projection at the second point on the surface of the surface structure.

37. The method of Claim 36, wherein, when the display of the second projection occurs when the first point is determined to be hidden from view or inaccessible by an instrument.

38. The method of Claim 22, wherein the determination of the first point, the generation of the first projection, and the display of the first projection are independent of movement of the first imaging device.

39. The method of Claim 22, wherein the first subsurface structure is displayed simultaneously with the first projection.Attorney Docket No. P06633-WO (153360) 40. The method of Claim 22, wherein the second imaging data further shows a second subsurface structure and the method further comprises: determining a second point on the surface based at least on where on the surface is closest to the second subsurface structure; generating a second projection of the second subsurface structure onto the surface at the second point; and displaying, on the display device, the second projection at the second point on the surface of the surface structure.

41. The method of Claim 40, wherein the first projection and the second projection are simultaneously displayed on the surface of the surface structure.

42. The method of Claim 41, wherein the first projection is represented as a first color and the second projection is represented as a second color on the surface of the surface structure.

43. A non-transitory machine-readable medium storing instructions for annotating a subsurface structure onto a surface, when executed by a controller, cause the controller to: perform the method of any of Claims 22 through 42.

Images