A system for providing synthetic indicators in a user interface for a robot assistance system.

Abstract

Solves the problems of conventional technology. [Solution] The medical system may include a display system and a control system. The control system may include a processing unit that includes one or more processors. The processing unit may be configured to display a field of view image of the surgical environment generated by an imaging component on the display system. The processing unit may also be configured to generate a three-dimensional composite indicator for the positions of instruments outside the field of view of the surgical environment and to display the three-dimensional composite indicator together with the field of view image of the surgical environment.

Description

Technical Field

[0001] (Reference to Related Applications) This application claims the benefit of U.S. Provisional Application No. 63 / 119,549, filed Nov. 30, 2020, the entire disclosure of which is incorporated herein by reference.

[0002] (Technical Field) The present disclosure is directed to medical procedures and methods for manipulating tissue during a medical procedure. More specifically, the present disclosure relates to systems and methods for providing depth-aware synthetic indicators, indicators for components outside the field of view, and indicators for components occluded within the field of view.

Background Art

[0003] Minimally invasive medical techniques are intended to reduce the amount of external tissue damaged during a diagnostic or surgical procedure, thereby reducing the patient's recovery time, discomfort, and adverse side effects. Such minimally invasive techniques may be performed through natural orifices in the patient's anatomical structure or through one or more surgical incisions. Through these natural orifices or incisions, a clinician may insert a medical tool to reach a target tissue location. Minimally invasive medical tools include instruments such as treatment instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools may also include imaging instruments such as endoscope instruments that provide a view within the patient's anatomical structure to the user.

[0004] Some minimally invasive medical tools may be robot-assisted, including being teleoperated, remotely operated, or otherwise computer-assisted. During medical procedures, clinicians may be provided with a graphical user interface that includes a three-dimensional field of view image of the patient's anatomical structure. To improve the clinician's experience and efficiency, various indicators may be needed to provide additional information about the medical tool within the field of view, the medical tool obstructing the field of view, and components outside the field of view. [Overview of the project]

[0005] Embodiments of the present invention are best summarized by the claims that follow the specification.

[0006] In one embodiment, the medical system may include a display system and a control system. The control system may include a processing unit comprising one or more processors. The processing unit may be configured to display images of the field of view generated by the imaging component, showing a view of the surgical environment, on the display system. The processing unit may be configured to generate a three-dimensional composite indicator for the positions of instruments outside the field of view of the surgical environment and to display the three-dimensional composite indicator together with images of the field of view of the surgical environment.

[0007] In another embodiment, the medical system may include a display system and an input system including a first pedal and a second pedal. The first pedal may have a spatial relationship with respect to the second pedal. The medical system may also include a control system. The control system may include a processing unit including one or more processors. The processing unit may be configured to display images of the surgical field on the display system. The images may be generated by an imaging component. The processing unit may be configured to generate a first composite indicator indicating the engagement state of the first pedal, a second composite indicator indicating the engagement state of the second pedal, and to display the first composite indicator on the display system relative to the second composite indicator based on its spatial relationship with the images of the surgical field.

[0008] In another embodiment, the medical system may include a display system and an input system including a first pedal and a second pedal. The first pedal may have a spatial relationship with respect to the second pedal. The medical system may also include a control system. The control system may include a processing unit including one or more processors. The processing unit may be configured to display images of the field of view of the surgical environment on the display system. The images may be generated by an imaging component. The processing unit may be configured to generate a first composite indicator related to instruments in the surgical environment, generate a depth mapping including structures in the field of view and the first composite indicator, and determine from the depth mapping the obstructed portion of the first composite indicator that is obstructed by structures. The processing unit may also be configured to display the first composite indicator on the display system. The obstructed portion of the first composite indicator may have a graphic appearance distinct from the unobstructed portion of the first composite indicator.

[0009] The general statements above and the detailed statements below are essentially illustrative and descriptive, and should be understood as being intended to provide an understanding of the disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of the disclosure will be apparent to those skilled in the art from the detailed statements below. [Brief explanation of the drawing]

[0010] [Figure 1A] This is a schematic diagram of a medical system according to one embodiment. [Figure 1B] This is a perspective view of an assembly according to one embodiment. [Figure 1C] This is a perspective view of a surgeon's control console for a medical system according to one embodiment. [Figure 2A] The illustration shows a graphical user interface with a composite indicator pointing to the direction of an off-screen tool, according to several embodiments. [Figure 2B] The illustration shows a graphical user interface with a composite indicator pointing to the direction of an off-screen tool, according to several embodiments. [Figure 2C] The illustration shows a graphical user interface with a composite indicator pointing to the direction of an off-screen tool, according to several embodiments. [Figure 2D] The illustration shows a graphical user interface with a composite indicator pointing to the direction of an off-screen tool, according to several embodiments. [Figure 3A] The illustration shows a composite indicator in various three-dimensional orientations pointing to different locations on a medical tool, according to several embodiments. [Figure 3B] The illustration shows a composite indicator in various three-dimensional orientations pointing to different locations on a medical tool, according to several embodiments. [Figure 3C] The illustration shows a composite indicator in various three-dimensional orientations pointing to different locations on a medical tool, according to several embodiments. [Figure 3D]The illustration shows a composite indicator in various three-dimensional orientations pointing to different locations on a medical tool, according to several embodiments. [Figure 3E] The illustration shows a composite indicator in various three-dimensional orientations pointing to different locations on a medical tool, according to several embodiments. [Figure 3F] The images show top views of stereoscopic viewing frustums for endoscopes according to several embodiments. [Figure 3G] Several embodiments provide a progression of images showing modulation of the length of a composite indicator. [Figure 3H] Several embodiments provide a progression of images showing modulation of the length of a composite indicator. [Figure 3I] Several embodiments provide a progression of images showing modulation of the length of a composite indicator. [Figure 3J] Several embodiments provide a progression of images showing modulation of the length of a composite indicator. [Figure 4] This is a top view of an input control device, including a foot pedal panel and a sensor system, according to several embodiments. [Figure 5A] Several embodiments of a graphical user interface are shown, which includes a composite indicator that provides status information regarding a foot pedal associated with a screen tool. [Figure 5B] Several embodiments of a graphical user interface are shown, which includes a composite indicator that provides status information regarding a foot pedal associated with a screen tool. [Figure 5C] Several embodiments of a graphical user interface are shown, which includes a composite indicator that provides status information regarding a foot pedal associated with a screen tool. [Figure 5D] Several embodiments are shown, including a graphical user interface with a synthetic indicator that provides status information regarding a foot pedal related to an on-screen tool. [Figure 6A] A graphical user interface is shown that includes a composite indicator that provides status information regarding a foot pedal associated with an on-screen tool, according to some embodiments. [Figure 6B] A graphical user interface is shown that includes a composite indicator that provides status information regarding a foot pedal associated with an on-screen tool, according to some embodiments. [Figure 6C] A graphical user interface is shown that includes a composite indicator that provides status information regarding a foot pedal associated with an on-screen tool, according to some embodiments. [Figure 6D] A graphical user interface is shown that includes a composite indicator that provides status information regarding a foot pedal associated with an on-screen tool, according to some embodiments. [Figure 7A] A graphical user interface is illustrated that includes a composite indicator that may conditionally move to remain visible when an endoscope that generates a component or field of view is moved, according to some embodiments. [Figure 7B] A graphical user interface is illustrated that includes a composite indicator that may conditionally move to remain visible when an endoscope that generates a component or field of view is moved, according to some embodiments. [Figure 7C] A graphical user interface is illustrated that includes a composite indicator that may conditionally move to remain visible when an endoscope that generates a component or field of view is moved, according to some embodiments. [Figure 7D] A graphical user interface is illustrated that includes a composite indicator that may conditionally move to remain visible when an endoscope that generates a component or field of view is moved, according to some embodiments. [Figure 8]The illustration shows an endoscope 550 extending into the patient's anatomical structure to visualize a synthetic indicator on a medical tool, according to several embodiments. [Figure 9A] The diagram illustrates a graphical user interface, comprising a synthetic indicator that remains visible when blocked, according to several embodiments. [Figure 9B] The diagram illustrates a graphical user interface, comprising a synthetic indicator that remains visible when blocked, according to several embodiments. [Figure 10A] The diagram illustrates a graphical user interface comprising a composite indicator with a blocked portion, according to several embodiments. [Figure 10B] The diagram illustrates a graphical user interface comprising a composite indicator with a blocked portion, according to several embodiments. [Figure 10C] The diagram illustrates a graphical user interface comprising a composite indicator with a blocked portion, according to several embodiments. [Figure 11A] The diagram illustrates a graphical user interface with a composite indicator for guiding tool changes, according to several embodiments. [Figure 11B] The diagram illustrates a graphical user interface with a composite indicator for guiding tool changes, according to several embodiments. [Figure 11C] The diagram illustrates a graphical user interface with a composite indicator for guiding tool changes, according to several embodiments. [Figure 11D] The diagram illustrates a graphical user interface with a composite indicator for guiding tool changes, according to several embodiments. [Figure 12] This flowchart describes methods for displaying a composite indicator and off-screen tools that point in a direction, according to several embodiments. [Figure 13]This flowchart describes a method for displaying a composite indicator showing the state of foot pedal engagement, according to several embodiments. [Figure 14] This flowchart describes a method for displaying a synthetic indicator that is at least partially obscured by a structure within the field of view, according to several embodiments. [Modes for carrying out the invention]

[0011] The embodiments of this disclosure and their advantages are best understood by referring to the detailed description below. Similar reference numerals are used to identify similar elements shown in one or more of the figures, and it should be understood that those shown therein are for illustrative purposes only and not to limit the embodiments of this disclosure.

[0012] In robot-assisted medical procedures, endoscopic images of the surgical environment can provide clinicians with a view of the patient's anatomical structures and any medical tools located within those structures. Augmenting endoscopic images with various indicators allows clinicians to access information while maintaining the field of view. Such indicators may include depth-aware graphical indicators, indicators for components outside the field of view, and indicators for components obstructed within the field of view.

[0013] Figures 1A, 1B, and 1C together provide an overview of a medical system 10 that may be used in medical procedures, including, for example, diagnostic, therapeutic, or surgical procedures. The medical system 10 is located within a medical environment 11, which is depicted as an operating room in Figure 1A. In other embodiments, the medical environment 11 may be an emergency room, a medical training environment, a medical laboratory, or any other type of environment in which any number of medical or medical training procedures may be performed. In yet another embodiment, the medical environment 11 may include an operating room and a control area located outside the operating room.

[0014] In one or more embodiments, the medical system 10 may be a robot-assisted medical system under the remote control of a surgeon. In an alternative embodiment, the medical system 10 may be under the partial control of a computer programmed to perform a medical procedure or subprocedure. In yet another alternative embodiment, the medical system 10 may be a fully automated medical system under the complete control of a computer programmed to perform a medical procedure or subprocedure in the medical system 10. An example of a medical system 10 that may be used to implement the systems and technologies described herein is the da Vinci® Surgical System manufactured by Intuitive Surgical, Inc. in Sunnyvale, California.

[0015] As shown in Figure 1A, the medical system 10 generally includes an assembly 12 which may be attached to or positioned near the operating table O on which the patient P is positioned. The assembly 12 may be referred to as a patient-side cart, surgical cart, or surgical robot. In one or more embodiments, the assembly 12 may be a remotely operated assembly. The remotely operated assembly may be referred to, for example, a remotely operated arm cart. A medical instrument system 14 and an endoscopic imaging system 15 are operably coupled to the assembly 12. An operator input system 16 allows a surgeon S or other type of clinician to view or represent images of the surgical site and to control the operation of the medical instrument system 14 and / or the endoscopic imaging system 15.

[0016] The medical device system 14 may include one or more medical devices. In embodiments where the medical device system 14 includes multiple medical devices, the multiple medical devices may include multiple identical medical devices and / or multiple different medical devices. Similarly, the endoscopic imaging system 15 may include one or more endoscopes. In the case of multiple endoscopes, the multiple endoscopes may include multiple identical endoscopes and / or multiple different endoscopes.

[0017] The operator input system 16 may be located on the surgeon's control console, which may be located in the same room as the operating table O. In some embodiments, the surgeon S and the operator input system 16 may be located in different rooms or in entirely different buildings from the patient P. The operator input system 16 generally includes one or more control devices for controlling the medical instrument system 14. The control devices may include one or more of any number of input devices, such as hand grips, joysticks, trackballs, data gloves, trigger guns, foot pedals, manual controllers, voice recognition devices, touchscreens, body movement or presence sensors, and other types of input devices.

[0018] In some embodiments, the control device has as many degrees of freedom as the medical instrument of the medical instrument system 14 to provide the surgeon with telepresence, where telepresence is the perception that the control device is integrated with the instrument, giving the surgeon a strong sense of direct control over the instrument as if it were present at the surgical site. In other embodiments, the control device may still provide the surgeon with more or fewer degrees of freedom than the associated medical instrument. In some embodiments, the control device is a manual input device that moves with six degrees of freedom and may include an operable handle for operating the instrument (e.g., for closing a gripping jaw end effector, for applying potential to an electrode, for delivering a medical procedure, and for operating other types of instruments).

[0019] Assembly 12 supports and operates a medical instrument system 14 while the surgeon S views the surgical site through an operator input system 16. Images of the surgical site may be obtained by an endoscopic imaging system 15, which may be operated by assembly 12. Assembly 12 may include an endoscopic imaging system 15 and, similarly, may include multiple medical instrument systems 14. The number of medical instrument systems 14 used at one time generally depends, among other things, on the diagnostic or surgical procedure being performed and the spatial constraints within the operating room. Assembly 12 may include a kinematic structure of one or more non-servo-controlled links (e.g., one or more links, commonly referred to as a setup structure, which may be manually positioned and locked in place) and a manipulator. When the manipulator takes the form of a remotely operated manipulator, assembly 12 is a remotely operated assembly. Assembly 12 includes multiple motors that drive inputs on the medical instrument system 14. In one embodiment, these motors move in response to commands from a control system (e.g., control system 20). The motor includes a drive system that, when connected to the medical instrument system 14, may advance the medical instrument into a naturally or surgically constructed anatomical orifice. Other motor drive systems may move the distal end of the medical instrument with multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the X, Y, and Z Cartesian axes) and three degrees of rotational motion (e.g., rotation around the X, Y, and Z Cartesian axes). Additionally, the motor may be used to actuate an articulated end effector of the medical instrument to grasp tissue in the jaws of a biopsy device, for example. The medical instruments of the medical instrument system 14 may include end effectors having a single working member, such as a scalpel, a blunt blade, an optical fiber, or an electrode. Other end effectors may include, for example, forceps, grippers, scissors, or clip applicators.

[0020] The medical system 10 also includes a control system 20. The control system 20 includes at least one memory 24 and at least one processor 22 for bringing control between the medical instrument system 14, the operator input system 16, and other auxiliary systems 26, which may include, for example, an imaging system, an audio system, a fluid delivery system, a display system, a lighting system, a steering control system, a cleaning system, and / or a suction system. The clinician may patrol within the medical environment 11 and, for example, access the assembly 12 during setup procedures or view the displays of the auxiliary systems 26 from the patient's bedside.

[0021] Although depicted as external to assembly 12 in Figure 1A, the control system 20 may, in some embodiments, be entirely housed within assembly 12. The control system 20 also includes programmed instructions (for example, stored on a non-temporary computer-readable medium) to implement some or all of the methods described according to the embodiments disclosed herein. Although the control system 20 is shown as a single block in the simplified diagram of Figure 1A, the control system 20 may include two or more data processing circuits, where one part of the processing is optionally performed on or adjacent to assembly 12, and another part of the processing is performed in the operator input system 16, and so on.

[0022] A wide variety of centralized or distributed data processing architectures may be used. Similarly, programmed instructions may be implemented as numerous separate programs or subroutines, or programmed instructions may be integrated into numerous other embodiments of the systems described herein, including remote control systems. In one embodiment, the control system 20 supports wireless communication protocols such as Bluetooth®, IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.

[0023] The control system 20 communicates with a database 27 which may store one or more clinician profiles, a list of patients and patient profiles, a list of procedures to be performed on patients, a list of clinicians scheduled to perform said procedures, other information, or a combination thereof. The clinician profile may include information about the clinician, including how long the clinician has worked in the medical field, the level of education the clinician has received, the level of experience the clinician has gained in the medical system 10 (or a similar system), or any combination thereof.

[0024] The database 27 may be stored in memory 24 and may be dynamically updated. Additionally or alternatively, the database 27 may be stored on a device such as a server or portable storage device that is accessible by the control system 20 via an internal network (e.g., a secure network of a medical facility or remote control system provider) or an external network (e.g., the internet). The database 27 may be distributed across two or more locations. For example, the database 27 may reside on multiple devices, which may include devices of different entities and / or cloud servers. Additionally or alternatively, the database 27 may be stored on portable user-assigned devices such as computers, mobile devices, smartphones, laptops, electronic badges, tablets, pagers, and other similar user devices.

[0025] In some embodiments, the control system 20 may include one or more servo controllers that receive force and / or torque feedback from the medical instrument system 14. In response to the feedback, the servo controllers transmit signals to the operator input system 16. The servo controllers may transmit signals instructing the assembly 12 to move the medical instrument system 14 and / or endoscopic imaging system 15 that extend into the internal surgical site within the patient's body through an opening in the body. Any suitable conventional or dedicated servo controllers may be used. The servo controllers may be separate from the assembly 12 or integrated with the assembly 12. In some embodiments, the servo controllers and the assembly 12 are provided as part of a remotely operated arm cart positioned adjacent to the patient's body.

[0026] The control system 20 can be coupled with the endoscopic imaging system 15 and may include a processor for processing captured images for continued display to the surgeon, such as on the surgeon's control console or on another suitable display located locally and / or remotely. For example, if a stereoscopic endoscope is used, the control system 20 can process the captured images to present the surgeon with a coordinated stereoscopic image of the surgical site. Such coordination may include alignment between opposing images and adjusting the stereoscopic working distance of the stereoscopic endoscope.

[0027] In an alternative embodiment, the medical system 10 may include more than one assembly 12 and / or more than one operator input systems 16. The exact number of assemblies 12 depends, among other things, on the surgical procedure and the spatial constraints within the operating room. The operator input systems 16 may be juxtaposed (located in the same place) or they may be located in separate places. Multiple operator input systems 16 allow more than one operator to control one or more assemblies 12 in various combinations. The medical system 10 may also be used for training and practicing medical procedures.

[0028] Figure 1B is a perspective view of one embodiment of an assembly 12, which may be referred to as a patient-side cart, surgical cart, remotely operated arm cart, or surgical robot. The illustrated assembly 12 provides operation of three surgical tools 30a, 30b, and 30c (e.g., medical instrument system 14) and an imaging device 28, such as a stereoscopic endoscope (e.g., endoscopic imaging system 15), used for capturing images of the treatment site. The imaging device may transmit signals to a control system 20 via a cable 56. Operation is provided by a remote surgical mechanism having multiple joints. The imaging device 28 and surgical tools 30a-30c can be positioned and manipulated through the patient's incision so that the kinematic remote center is maintained in the incision to minimize the size of the incision. Images of the surgical site may include images of the distal ends of the surgical tools 30a-30c as they are positioned within the field of view of the imaging device 28.

[0029] Assembly 12 includes a drivable base 58. The drivable base 58 is connected to a telescopic column 57, which allows for height adjustment of the arms 54. The arms 54 may include a rotary joint 55 that rotates and moves up and down. Each of the arms 54 may be connected to an orientation platform 53. The arms 54 may be labeled to facilitate troubleshooting. For example, each of the arms 54 may be decorated with different numbers, letters, symbols, other identifiers, or combinations thereof. The orientation platform 53 may be capable of 360-degree rotation. Assembly 12 may also include a telescopic horizontal cantilever 52 for moving the orientation platform 53 in a horizontal direction.

[0030] In this example, each of the arms 54 is connected to a manipulator arm 51. The manipulator arm 51 may be directly connected to a medical instrument, for example, one of the surgical tools 30a to 30c. The manipulator arm 51 may be remotely controllable. In some examples, the arms 54 connected to the orientation platform 53 may not be remotely controllable. Rather, such arms 54 may be positioned as desired before the surgeon S begins surgery with the remote surgical component. Throughout the surgical procedure, medical instruments may be removed and replaced with other instruments, so that the association of instruments with the arms may change during the procedure.

[0031] Endoscopic imaging systems (e.g., endoscopic imaging system 15 and imaging device 28) may be provided in various configurations, including rigid or flexible endoscopes. A rigid endoscope includes a rigid tube housing a relay lens system for transmitting images from the distal end to the proximal end of the endoscope. A flexible endoscope transmits images using one or more flexible optical fibers. Digital image-based endoscopes have a "chip-on-the-tip" design in which a distal digital sensor, such as one or more charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor (CMOS) elements, stores image data. Endoscopic imaging systems may provide two-dimensional or three-dimensional images to the viewer. Two-dimensional images may provide a limited sense of depth. Three-dimensional stereoscopic images may provide a more accurate sense of depth to the viewer. Stereoscopic endoscopes use a stereoscopic camera to capture three-dimensional images of the patient's anatomical structures. Endoscopic instruments may be fully sterile assemblies, all rigidly coupled and hermetically sealed, comprising endoscopic cables, handles, and shafts.

[0032] Figure 1C is a perspective view of one embodiment of an operator input system 16 in a surgeon's control console. The operator input system 16 includes a left-eye display 32 and a right-eye display 34 to present the surgeon S with coordinated stereoscopic vision of the surgical environment enabling depth perception. The left-eye display 32 and the right-eye display 34 may be components of a display system 35. In other embodiments, the display system 35 may include one or more other types of displays. The display system 35 may present, for example, images captured by an imaging system 15 to display the endoscopic view to the surgeon. The endoscopic view may be augmented by virtual or composite menus, indicators, and / or other graphic or text information to provide additional information to the viewer.

[0033] The operator input system 16 further includes one or more input control devices 36, which then cause the assembly 12 to operate one or more instruments of the endoscopic imaging system 15 and / or the medical instrument system 14. The input control devices 36 can provide the same degree of freedom as the associated instruments to give the surgeon S a sense of telepresence, or a strong sense that the surgeon is directly controlling the instruments, by giving the input control devices 36 a sense of being integrated with the instruments. To achieve this objective, position, force, and tactile feedback sensors (not shown) may be used to transmit and return position, force, and tactile sensations from medical instruments, e.g., surgical tools 30a-30c, or imaging devices 28, to the surgeon's hands through the input control devices 36. An input control device 37 is a foot pedal that receives input from the user's feet. Embodiments of the operator input system 16, assembly 12, and auxiliary system 26 may be adjustable and customizable to meet the physical needs, skill level, or preferences of the surgeon S.

[0034] During medical procedures performed using the medical system 10, a surgeon S or another clinician may need to access medical tools within patient anatomical structures outside the field of view of the imaging system 15, engage a foot pedal to activate a medical tool or perform other system functions, and / or identify tools that are occluded within the field of view. Furthermore, in a stereoscopic field of view, it may be desirable that synthetic elements presented in the field of view be displayed at a depth corresponding to the tissue or component indicated by the synthetic element. Thus, the synthetic element may appear to be attached to the component within the field of view rather than floating in front of the field of view. Various embodiments described below provide methods and systems that enable a surgeon S to view depth-aware graphical indicators, indicators for components outside the field of view, and indicators for components occluded within the field of view.

[0035] Figures 2A, 2B, 2C, and 2D illustrate a graphical user interface 200 that may be displayed on, for example, a display system 35. The graphical user interface 200 may include a field of view portion 202 for displaying an image of the field of view 203 of the surgical environment 201 captured by an imaging system (e.g., imaging system 15). The surgical environment is in Cartesian coordinate system X S , Y S , Z S The graphical user interface 200 may include an information block 210 that displays information about the medical tool 204, an information block 212 that displays information about the imaging system that captures the image in the field of view 202, an information block 214 that displays information about the medical tool 206, and an information block 216 that displays information about the medical tool 208. Information blocks 210, 212, 214, and 216 may include the type of tool, the number of manipulator arms to which the tool is attached, status information about the arm or tool, and / or operational information about the arm or tool.

[0036] The graphical user interface 200 may include one or more composite indicators 218, 220, 222 that may appear within the field of view 202 when a corresponding medical tool is within the surgical environment but outside the field of view of the imaging system and therefore not visible within the field of view 202. Composite indicator 218 indicates tool 204. Composite indicator 220 indicates tool 206. Composite indicator 222 indicates tool 208. Each composite indicator 218, 220, 222 may have a three-dimensional shape and may point to the three-dimensional direction of the corresponding tool outside the field of view.

[0037] In Figure 2A, the field of view 202 contains a three-dimensional image of a portion of the surgical environment, and composite indicators 218, 220, and 222 around the image periphery 219 point to the respective tools 204, 206, and 208, which are within the surgical environment but outside the field of view of the imaging system. In Figure 2B, the imaging system (e.g., an endoscope) is moved in the +Y direction to capture different images of the surgical environment within the field of view 202. The distal ends of tools 204 and 206 are now visible. Tool 208 remains outside the field of view, and consequently, the composite indicator 222 indicating the direction of tool 208 is displayed. In Figure 2C, the imaging system is moved further in the +Y direction to capture different images of the surgical environment within the field of view 202. The distal ends of tools 204, 206, and 208 are now visible within the field of view 202. Therefore, the composite indicator is not displayed. In Figure 2D, the imaging system is moved in the -Y and +X directions to capture different images of the surgical environment within the field of view 202. Tool 206 remains visible within the field of view 202, while tools 204 and 208 are now outside the field of view 202. Thus, composite indicators 218 and 222 are displayed, pointing to the three-dimensional locations of tools 204 and 208 within the surgical environment, respectively.

[0038] Figures 3A to 3E illustrate the field of view 203 of a surgical environment 201 having a composite indicator 218 in various three-dimensional orientations to point to different locations on a medical tool 204. In each illustration, the medical tool 204 is within the surgical environment 201 but outside the field of view 203, and therefore the composite indicator 218 is displayed within the field of view portion 202. The composite indicator 218 includes an indicator body 250 including a directional portion 252. The directional portion 252 may include a taper that may point toward the medical tool 204. In this embodiment, the composite indicator 218 may have a teardrop shape, but in other embodiments, an arrow, triangle, or other pointed symbol that can indicate direction may be used for the composite indicator. The indicator body 250 may have a three-dimensional shape with dimensions of height H, depth D, and width W. The indicator body 250 may have at least one flat surface 253 and an icon 254 that may appear as a decal fixed along the flat surface 253. The icon 254 may contain an identifier, such as an identifier for the manipulator arm to which the pointed tool is joined, or an identifier for the pointed tool itself. As the indicator body 250 moves in three-dimensional space, the orientation of the icon 254 may rotate relative to the indicator body and the directional section 252 such that the text or symbol on the icon 254 remains upright to the viewer. The orientation of the icon 254 may also remain aligned with the orientation of the faces of the indicator body 250.

[0039] As the tool 204 moves within the surgical environment 201, or as the field of view 203 changes within the surgical environment, the composite indicator 218 may pivot so that the directional portion 252 remains pointed toward the tool 204 and the flat surface 253 remains visible to the viewer. In Figure 3A, the directional portion 252 points toward the tool 204 located outside the field of view 203 in the +Y direction relative to the composite indicator 218. In Figure 3B, the directional portion 252 points toward the tool 204 located outside the field of view 203 in the -Y, +X, and -Z directions relative to the composite indicator 218. In Figure 3C, the directional portion 252 points toward the tool 204 located outside the field of view 203 in the +Y, -X, and -Z directions relative to the composite indicator 218. In Figure 3D, the directional portion 252 points toward the tool 204 located outside the field of view 203 in the -X and +Z directions relative to the composite indicator 218. In Figure 3E, the directional portion 252 points towards the tool 204, which is located outside the field of view 203 in the -Y, +X, and +Z directions relative to the composite indicator 218. Examples in Figures 3D and 3E may illustrate the use of the composite indicator when the tool tip is behind the endoscope tip. Without a directional indicator, the user may become confused about the direction in which to move the endoscope. For example, when the tool tip is directly behind the endoscope tip, lateral movement of the endoscope itself can be counterproductive, resulting in the projected tool indicator rapidly swinging from one side of the field of view to the other, making it difficult to locate the tool.

[0040] In some embodiments, the composite indicator 218 or a portion thereof may have color coding or other visual treatment to indicate the state and / or control mode of the associated tool 204 (e.g., active or inactive; clutch initiated position). In some embodiments, the orientation of the composite indicator 218 may be determined based on presentation purposes, including visibility to the viewer. For example, a flat surface 253 may be oriented toward the endoscope, and the icon 254 may be oriented at the location of the surface 253 so that it stands upright in the view. The directional portion 252 may be constrained so that its normal to the flat surface 253 is oriented within the viewing cone or viewing frustum of the endoscope to ensure the visibility of the icon 254. The stereoscopic depth of the location of the composite indicator 218 may be constrained for ease of fusion to reduce depth mismatch with the endoscope scene content and to eliminate obstruction and depth to other composite elements in the field of view portion 202. The apparent size of the composite indicator 218 may be constrained based on its depth.

[0041] In some embodiments, the position of the directional portion 252 along the perimeter 219 of the field of view portion 202 is calculated by ray intersection with the stereoscopic image. Figure 3F provides a top view of the stereoscopic observation platform 270 of the endoscope 272 (e.g., imaging system 15) that provides the field of view portion 202. The stereoscopic observation platform 270 is formed from a right eye frustum 274 corresponding to the right eye field of view and a left eye frustum 276 corresponding to the left eye field of view. The stereoscopic convergence location 286 is at a convergence depth 287 from the distal tip of the endoscope. Marker 278 corresponds to the projection location of the tip of the directional portion of a three-dimensional composite indicator (e.g., indicator 218) pointing toward a keypoint on the instrument tip portion 280. The location of marker 278 is resolved to be within the minimum depth range 282 and maximum depth range 284 of the distal tip of the endoscope 272 and within the observation platforms 274, 276 of both the left and right eyes. Determining the minimum and maximum depth ranges can provide stereoscopic comfort and enhance the user's perception of the relative spatial relationships of off-screen tools. In this example, with the instrument tip portion 280 positioned as shown, the tip of the directional portion of the composite indicator may appear at the marker 278, i.e., at the intersection of the minimum depth range 282 and the left eye frustum 276. The directional portion of the composite marker may be nominally aligned along the pointing direction 288 between the focal point 286 and the marker 278.

[0042] For example, a ray extending from a point along the centerline of an imaging component (e.g., an endoscope) to a distal keypoint of the associated instrument (e.g., a predetermined point on the instrument end effector or joint) may be determined. This determination resolves a point along the periphery 219 that may be visible to both eyes within a comfortable depth range for fusion.

[0043] In some embodiments, the composite indicator may deform in shape and size as the endoscope and / or medical tool moves. For example, the composite indicator may transition from a circular badge to a teardrop shape. The length of the teardrop or arrow shape may indicate the distance of the tool from the field of view. The composite indicator may also highlight the direction and / or distance of movement to locate off-screen tools. For example, when a tool is located beyond a threshold distance from the field of view, all or part of the composite indicator may be animated to generate a gestural cue to highlight that the tool is beyond the threshold distance from the field of view. Figures 3G–3J provide a progression of images depicting the modulation of the length of the composite indicator 218 in relation to the importance of the distance of the instrument tip 204 outside the field of view portion 202 and the direction of movement to the instrument tip 204. As shown in Figure 3J, when the instrument tip 204 is at a distance D4 far outside the field of view volume (e.g., greater than 3 cm) or its Z-direction position is outside the minimum and maximum depth ranges, the composite indicator, in particular the directional portion 252, is stretched or otherwise emphasized to convey both the direction and distance of progression. In contrast, as shown in Figure 3G, when the instrument tip 204 is close to the field of view (e.g., <3 cm) and its Z-direction position is within the minimum and maximum depth ranges, the directional portion 252 is de-emphasized so that the composite indicator 218 becomes more circular. In the example in Figure 3G, the position along the periphery of the display is sufficient to indicate the lateral spatial location of the instrument tip 204 relative to the endoscopic view volume. Figure 3H illustrates a more prominent and longer directional portion 252 than in Figure 3G, indicating that the distance D2 to the instrument tip 204 is greater than the distance D1. Figure 3I illustrates a directional portion 252 that is longer than that in Figure 3H but not longer than that in Figure 3J, showing that the distance D3 to the instrument tip 204 is longer than the distance D2 but not as long as the distance D4.

[0044] A method 800 for displaying a three-dimensional composite indicator (e.g., composite indicators 218, 220, or 222) is illustrated in the flowchart of Figure 12. The methods described herein are illustrated as a series of operations or processes and are described with further reference to additional figures. Not all illustrated processes are performed in all embodiments of the methods. In addition, one or more processes not explicitly illustrated may be included before, after, between, or as part of the illustrated processes. In some embodiments, one or more processes may be implemented in the form of executable code stored in a non-temporary, tangible, machine-readable medium, which may cause one or more processors (e.g., processors of a control system) to execute one or more of the processes when executed by one or more processors. In one or more embodiments, the processes may be executed by a control system.

[0045] In process 802, an image of the field of view (e.g., field of view portion 202) within the surgical environment (e.g., surgical environment 201) is displayed, for example, on the display 35. In some embodiments, process 802 may include one or more of processes 804a to 804f. In process 804a, the visibility of the instrument tip keypoint relative to the endoscopic field of view volume may be determined.

[0046] In process 804b, a decision may be made regarding whether a composite indicator should be displayed for an off-screen instrument based on context and predetermined rules. Displaying a composite indicator at all times while the tool tip is out of field of view may introduce undesirable distraction. Therefore, predetermined rules may be imposed on when a composite indicator is displayed so that it is more contextual and aligns with operator workflow steps where its visibility benefits from user awareness of the off-screen tool location. For example, a composite indicator may be displayed when the movement of the endoscope is active from either the bedside or the surgeon's console. A composite indicator may be displayed when a guided tool exchange configuration is active on the tool's manipulator arm. A composite indicator may be displayed when an instrument clutch is active for the manipulator arm controlling an off-screen tool. A composite indicator may be displayed when a surgeon console user is about to initiate control of the manipulator arm controlling an off-screen tool. A composite indicator may be displayed when a surgeon console user initiates control of the manipulator arm controlling an off-screen tool. The composite indicator may appear when a surgeon console user is changing the association of a hand to a manipulator arm that is coupled to an off-screen tool. The composite indicator may also appear when an off-screen tool is coupled or when a notification about the manipulator arm is displayed.

[0047] In process 804c, the projected three-dimensional position of the composite indicator along the lateral extension of the field volume may be determined. In process 804d, the orientation of the three-dimensional composite indicator may be determined so that it faces the endoscope tip within the visible cone or frustum. In process 804e, the upright orientation of an icon (e.g., icon 254) on the surface of the composite indicator may be calculated. In process 804f, both left and right views of the composite indicator may be rendered using a calibrated stereoscopic camera model corresponding to the endoscope optics.

[0048] In process 804, a three-dimensional composite indicator (e.g., indicator 218) may be generated to show the position of an instrument outside the field of view. More specifically, in some embodiments, a composite rendering of the left and right composite indicators may be superimposed on the endoscopic video.

[0049] In process 806, the three-dimensional composite indicator may be displayed along with an image of the surgical environment field of view.

[0050] Figure 4 provides a top view of an input control device 300 of an operator input system (e.g., operator input system 16), which includes an input panel 301 that forms a common platform for input control devices 302, 304, 306, 308, 310, 312 (e.g., input control device 37) configured as foot pedals that receive input from the user's feet. The foot pedals 302, 304, 306, 308, 310, 312 may be engaged to control the function of a medical tool coupled to a remote operation assembly (e.g., assembly 12) and / or the arm of the remote operation assembly. The input control device 300 may also include a sensor system 314 that detects the position of the user (e.g., the user's feet or legs) relative to the input control device. The sensor system 314 may include a camera, optical sensor, motion sensor, or other sensor that senses or tracks the presence of the user in the vicinity of one or more of the input control devices 302-312. The sensor system 314 may also include pressure sensors, displacement sensors, or other types of sensors that detect when one or more of the input control devices are activated or engaged.

[0051] Figures 5A, 5B, 5C, and 5D illustrate the graphical user interface 200. Medical tools 400 and 402 are visible in the field of view 202. The function of a medical tool may be initiated by engaging the corresponding foot pedal on the input panel 301. For example, medical tool 400 may be operated by the manipulator arm 1 as shown in the information block 210 and may be a vascular sealer, which may perform a cutting function when foot pedal 302 is engaged and a sealing function when foot pedal 304 is engaged. As shown in Figure 5A, the tool 400 may be labeled with a composite indicator 404. In this embodiment, the composite indicator 404 may be a generally circular badge including an upper semicircular portion 406 and a lower semicircular portion 408. The upper semicircular portion 406 includes a contour portion 410 and a central portion 412, and the lower semicircular portion 408 includes a contour portion 414 and a central portion 416. The upper semicircular portion 406 may correspond to the function of the secondary foot pedal 302 and may indicate the engaged state of the pedal 302 (e.g., hovering, activated). The lower semicircular portion 408 may correspond to the function of the primary foot pedal 304 and may indicate the engaged state of the pedal 304 (e.g., hovering, activated). The spatial relationship between the upper semicircular portion 406 and the lower semicircular portion 408 may be the same or similar to the spatial relationship between the pedals 302 and 304. When the sensor system 314 detects that the operator's foot is hovering above or below a threshold distance from the foot pedal 302, the contour portion 410 of the upper semicircular portion 406 may change its appearance (e.g., change color, animate) to indicate to the operator that the operator's foot is close to the foot pedal 302. Thus, the operator can determine the foot position while their vision remains directed towards the graphical user interface 200.When the operator engages the foot pedal 302 (e.g., by pressing or pushing down the pedal), the central portion 412 of the upper semicircular portion 406 may change its appearance (e.g., change color, be animated) to indicate to the operator that the operator's foot has engaged the foot pedal 302 and that the function of the foot pedal 302 (e.g., cutting) has been initiated. In some embodiments, the hovering or engagement state of the foot pedal 302 may be indicated in the information block 210 using the same or similar graphical indicators. The left bank of foot pedals (e.g., pedals 302, 304) may be associated with a left-hand input control device, and the right bank of foot pedals (e.g., pedals 306, 308) may be associated with a right-hand input control device. Each hand may be associated to control any instrument arm. A juxtaposed composite indicator reflects this association of the instrument with the corresponding hand and foot. In some configurations, the pose of the instrument relative to the endoscopic field of view may otherwise appear to have an ambiguous association with the left or right side; therefore, a juxtaposed composite indicator clarifies this association.

[0052] As shown in Figure 5C, the lower semicircular portion 408, like the upper semicircular portion 406, may function as an indicator of hovering and engagement of the foot pedal 304. When the operator engages with the primary foot pedal 304 (e.g., by pressing or pushing down the pedal), the central portion of the lower semicircular portion 408 may change its appearance (e.g., change color, be animated) to indicate to the operator that the operator's foot has engaged with the foot pedal 304 and that the function of the foot pedal 304 (e.g., sealing) has been initiated. The pedals on the surgeon's console may be color-coded. For example, the primary pedals 304, 308 may be colored blue, and the secondary pedals 302, 306 may be colored yellow. This color coding is reflected in the associated highlights and fills in the colors of the pedal function composite indicators on the graphical user interface.

[0053] As shown in Figure 5B, the tool 402 may be labeled with a synthetic indicator 420. In this embodiment, the synthetic indicator 420 may be substantially similar in appearance and function to the synthetic indicator 404, but may provide information about the set of foot pedals 306, 308. The tool 402 may be actuated by the manipulator arm 3 as shown in the information block 214, and may be a monopolar cautery instrument, which may serve to deliver energy for cutting when the foot pedal 306 is engaged, and may serve to deliver energy for coagulation when the foot pedal 308 is engaged. When the sensor system 314 detects that the operator's foot is hovering above or below a threshold distance from the secondary foot pedal 306, the contour portion of the upper semicircular section may change its appearance to indicate to the operator that the operator's foot is close to the foot pedal 306. When the sensor system 314 determines that the operator has engaged or activated the foot pedal 306, the central portion of the upper semicircular section may change its appearance to indicate to the operator that the operator's foot has engaged the foot pedal 306 and that the function of the foot pedal 306 (e.g., delivering energy for cutting) has been initiated. In some embodiments, the hovering or engagement state of the secondary foot pedal 302 may be indicated in the information block 214 using the same or similar graphical indicators.

[0054] As shown in Figure 5D, the lower semicircular portion of the indicator 420, like the upper semicircular portion, may function as an indicator of the hovering and engagement of the primary foot pedal 308. When the operator engages with the primary foot pedal 308, the central portion of the lower semicircular portion may change its appearance to indicate to the operator that the operator's foot has engaged with the primary foot pedal 308 and that the function of the foot pedal 308 (e.g., delivering energy for solidification) has been initiated.

[0055] The position and orientation of the composite indicators 404, 420 may be determined to create the appearance that the composite indicator is, for example, a decal bonded to a tool clevis or shaft. As the tool or endoscope providing the field of view is moved, the composite indicators 404, 420 may change their orientation in three-dimensional space to maintain tangency with respect to the tool surface and to maintain spatial understanding of the upper and lower pedals.

[0056] Various types, shapes, and configurations of composite indicators may be displayed to provide information regarding the state of foot pedal engagement. In an alternative embodiment, as shown in Figures 6A, 6B, 6C, and 6D, the graphical user interface 200 with medical tools 400, 402 is visible in the field of view portion 202. In this embodiment, the composite indicators 450, 452, 454, and 456 may take the form of elongated bars extending along the periphery 219. In this example, the composite indicators 450-456 are within the boundary of the periphery 219, but in an alternative embodiment, they may be outside the periphery 219 of the field of view 202.

[0057] In this embodiment, composite indicators 450 and 452 may perform a similar function to composite indicator 404 when providing information about the set of foot pedals 302 and 304. As shown in Figure 6A, when the sensor system 314 detects that the operator's foot is hovering above or elsewhere within a threshold distance from the primary foot pedal 308, the composite indicator 456 is contoured to indicate to the operator that the operator's foot is near the primary foot pedal 308. As shown in Figure 6B, when the operator engages with the foot pedal 308, the composite indicator 456 may become a filled bar to indicate to the operator that the operator's foot has engaged with the foot pedal 308 and that the function of the foot pedal 308 has been initiated. In some embodiments, the hovering or engagement state of the foot pedal 308 may be indicated in the information block 214 using the same or similar graphical indicators.

[0058] As shown in Figure 6C, when the sensor system 314 detects that the operator's foot is hovering above or otherwise within a threshold distance from the secondary foot pedal 302, the composite indicator 450 is contoured to indicate to the operator that the operator's foot is near the foot pedal 302. As shown in Figure 6D, when the operator engages with the foot pedal 302, the composite indicator 456 may become a filled bar to indicate to the operator that the operator's foot has engaged with the foot pedal 302 and that the function of the foot pedal 302 has been initiated. In some embodiments, the hovering or engagement state of the foot pedal 302 may be indicated in the information block 210 using the same or similar graphical indicators.

[0059] In alternative embodiments, audio cues may be provided instead of, or in addition to, the synthetic indicators, to provide instructions or spatial directions (e.g., up / down / left / right) to move the operator's foot to a hovering position for the foot pedal. The system may distinguish between hovering the foot over the pedal and activating the pedal, and there may be clear visual and audio cues for the hovering state versus the engaged or activated state. The system may also depict when the pedal function is enabled or disabled. When the pedal function is disabled (e.g., when the instrument function cable is not plugged in or when the instrument function is not set), the highlight color may appear gray.

[0060] A method 820 for displaying composite indicators corresponding to a set of foot pedals is illustrated in the flowchart of Figure 13. In process 822, an image of the field of view (e.g., field of view portion 202) of the surgical environment (e.g., environment 201) is displayed, for example, on a display 35. In process 824, a first composite indicator (e.g., semicircular portion 406) indicating the engagement state of the first pedal 302 is generated. In process 826, a second composite indicator (e.g., semicircular portion 408) indicating the engagement state of the second pedal 304 is generated. In process 828, the first composite indicator is displayed relative to the second composite indicator based on the spatial relationship between the first and second pedals. The first and second indicators are displayed together with the image of the field of view.

[0061] As shown in Figures 7A to 7D, composite indicators, which appear as badges or labels on components within the field of view 202, may appear near the components and may conditionally move to remain close to and visible to the components as the components or the endoscope generating the field of view are moved. The composite indicators may be used for any of the purposes described above, but may also be used to identify medical tools or other components within the field of view 202, identify the manipulator arm to which the medical tool is coupled, provide status information about the medical tool, provide operational information about the medical tool, or provide any other information about the tool or the manipulator arm to which the tool is coupled.

[0062] As shown in Figure 7A, the composite indicator 500 may be associated with the tool 502. In this embodiment, the composite indicator 500 may be a badge configured to have the appearance of a decal on the tool 502. The badge 500 may appear close to the jaws 504a, 504b of the tool 502, but may be positioned to avoid obstructing the jaws. The positioning may include a bias away from the jaws based on the positional uncertainty of the underlying motion tracking technology. The default position of the badge 500 may be at a predetermined key point 501 on the tool 502. As shown in Figure 7A, the badge 500 may be positioned at a key point 501 located in the crevice of the tool. The badge 500 may pivot and translate as the endoscope or tool 502 moves, so that the badge 500 remains at the key point and is oriented along the surface of the crevice. When the surface of the clevis is no longer visible within the field of view 202, the badge 500 may be moved to another key point 503 (at a predetermined joint location) as shown in Figure 7B or (along the shaft of the tool 502) as shown in Figure 7D.

[0063] Badge 500 may remain at its original keypoint location if the keypoint location remains visible within the field of view 202. Figure 8 illustrates an endoscope 550 (e.g., imaging system 15) extending into a patient anatomical structure 551. An observation cone 552 extends from the distal end of the endoscope 550 to the tissue surface 553. The area within the observation cone 552 may be an area visible within the field of view 202. A tool 554 extends into the patient anatomical structure 551. Badge 556 may have a default location at keypoint 557. A line 558 normal to the surface of badge 556 may be considered to determine whether the default location is visible on the display. Since the normal 558 does not extend into the observation cone 552, a determination may be made that badge 556 in its default location is not visible. Based on the condition that the normal vector 558 does not extend into the observation cone 552, the badge 556 may be moved to a secondary default position at keypoint 559. The normal vector 562 to the badge 556 at the second keypoint 559 is within the observation cone 552, so that the badge 556 may remain at the second keypoint 559 until the movement of the tool 554 or endoscope 550 causes the normal vector to the badge to no longer extend into the observation cone 552. Referring again to Figure 7B, the normal vector to the badge 500 at the original keypoint (in Figure 7A) is no longer within the observation cone, so the badge 500 may be moved to the second default keypoint.

[0064] The orientation of the badge 500 at a keypoint may be constrained so that the perpendicular to the badge surface is within the field of view cone and therefore visible within the field of view portion 202. If the badge cannot be oriented at a keypoint so that the normal is within the observation cone, the badge 500 may be moved to a different keypoint. As shown in Figure 7D, the orientation of the badge 500 may be pivoted to align with the orientation of the tool 502 shaft while the surface of the badge 500 remains visible to the viewer. The size of the badge 500 may also change as the distance of the keypoint to which the badge is attached moves closer to or further from the distal end of the endoscope, or when the zoom function of the endoscope is activated. The size of the badge may be controlled to remain within the range of maximum and minimum thresholds to avoid it becoming too large or too small on the display. As shown in Figure 7C, the badge 500 may be smaller because the keypoint in Figure 7C is further from the endoscope than the keypoint in Figure 7A.

[0065] The position, orientation, and depth of a composite indicator associated with a tool within the surgical environment may be determined based on tool tracking and depth map analysis by the control system. Tool tracking alone may introduce some residual error that causes the composite indicator to appear to float or interpenetrate the tool surface. This can distract the viewer's attention and lead to fusion problems between the composite indicator and the associated tool. A depth map, which provides information about the distance of surfaces within the field of view 202 from the distal end of the endoscope, may be used to refine the placement of the composite indicator on the tool surface. More specifically, a raycast projection may be calculated within a tolerance of the reference composite indicator position. The generated error may be used to estimate a radial offset correction for more accurate placement of the composite indicator on the tool surface. The quality and accuracy of the depth map are better when the tool is static or quasi-static than when the tool is moving. Therefore, raycasting and updating of radial offset corrections may be performed when the instrument keypoint velocity is below a threshold velocity. Alternatively, projection texturing may be used to directly place the composite indicator on the extracted depth map surface. Systems and methods for generating depth maps are further described in Patent Documents 1 and 2, the full texts of which are incorporated herein by reference.

[0066] Figures 9A and 9B illustrate a graphical user interface 200 with medical tools 600 and 602 visible within the field of view 202. A composite indicator 604 is displayed on medical tool 600, and a composite indicator 606 is displayed on medical tool 602. As shown in Figure 9B, as tool 602 moves behind tool 600 from the endoscope's viewpoint, the position and orientation of the composite indicator 606 relative to tool 602 may be maintained at the same three-dimensional depth as the surface of tool 602, which appears to be fixed. The composite indicator 606 remains visually juxtaposed with its keypoint even when positioned behind another object. Rather than being obscured by tool 600, the composite indicator 606 may be indicated by visual processing (e.g., a gostested appearance, a faded appearance, translucency, a dotted border) that shows the viewer that the composite indicator 606 is being viewed through the semi-opaque shaft of tool 600. A depth map may be used to perform depth-aware blending of the composite indicator 606 with the field of view image.

[0067] Three-dimensional composite indicators simply superimposed on a stereoscopic image may be intersected or occluded by content within the field of view, without considering depth. This can lead to misleading spatial relationships between real-world objects and composite objects, as well as difficulties in stereoscopic fusion. Using depth maps can improve the spatial appearance of composite indicators placed within the field of view. In some embodiments, depth mapping may be used for occlusion culling, which prevents rendering and displaying portions of the composite indicator that are deeper than the depth map. Complete culling of a composite indicator, or even partial culling thereof, can result in a loss of physical collocation state information. In some examples, when a collocated composite indicator is displayed under suboptimal tracking or rendering conditions (e.g., depth occlusion, field of view culling, poor tracking performance, poor depth map quality, poor stereoscopic alignment, etc.), the graphical user interface may gradually shift from the collocated indicators shown in Figures 5A–5D to the spatially aligned peripheral indicators shown in Figures 6A–6D.

[0068] In other embodiments, when using a depth map, the complete composite indicator may be preserved, or the closed portion of the composite indicator may be rendered with different visual treatment (e.g., semi-transparency) than the unclosed portion. To achieve special visual treatment for the closed portion of the composite indicator, rendering to the composite indicator may be performed in two stages. In the first stage, the composite indicator may be rendered with reduced opacity without referencing or modifying the depth map. In the second stage, the composite indicator may be rendered with increased opacity while applying selective deselection of the depth map so that only the unclosed pixels appear more opaque and are rendered on the pixels generated in the first stage. Thus, the closed portion of the composite indicator appears with reduced opacity (e.g., more semi-transparency), and the unclosed portion of the composite indicator appears with greater or complete opacity. In some embodiments, a stereoscopic display may be used to render a composite indicator for one eye (e.g., the viewer's non-dominant eye) by selectively removing (cull) the occluded portion of the composite indicator, while the composite indicator rendering for the other eye (e.g., the viewer's dominant eye) may be generated using a depth map and depth-aware blending. In some embodiments, the composite indicator may be generated based on a user-generated graphic. The user-generated graphic may be based on a monocular image when the composite indicator is generated stereoscopically.

[0069] Figures 10A and 10B illustrate a graphical user interface 200 having a medical tool 650 visible within a field of view 202. In Figure 10A, the composite indicator 652 is rendered within the field of view 202, but appears to float above the tool 650 in the stereoscopic image. In Figure 10B, the composite indicator 654 is rendered in the same position as the indicator 652, but the rendering in Figure 10AB produces a visual appearance in which the composite indicator is embedded within the shaft of the tool 650. To create this visual appearance, the inner portion 656 of the composite indicator 654 is rendered with shading to indicate that the inner portion 656 is covered by the tool 650 or is inside the tool 650. The outer portion 658 of the composite indicator 654 is rendered with full opacity to indicate that the outer portion 658 is outside the tool 650.

[0070] Figure 10C illustrates a graphical user interface 200 having medical tools 650 and 651 visible in the field of view 202. In Figure 10C, the composite indicator 660 is rendered as a ring that appears to surround tool 650, and the composite indicator 662 is rendered as a ring that appears to surround tool 651. Using the depth-aware blending technique described above, a portion 664 of the composite indicator 660 appearing behind tool 650 may be rendered with a different shading or color than a portion 666 of the composite indicator 660 appearing on tool 650 and surrounding tissue. Similarly, a portion 668 of the composite indicator 662 appearing behind tool 651 may be rendered with a different shading or color than a portion 670 of the composite indicator 662 appearing on tool 651.

[0071] In some embodiments, the graphical user interface 200 may be used to display a composite indicator for use in guided tool changes. The composite indicator may be rendered as a composite tube that serves as a path to guide the insertion of a new tool into a distal target mark. In some embodiments, all or part of the composite tube may be obstructed by tissue or other tools. Figures 11A–11D illustrate the graphical user interface 200 with different variations of the field of view portion 202. Figure 11A illustrates the visualization of the depth map 701 of the field of view portion 202. Tools 700 and 702 are visible within the field of view portion. A composite indicator 704 in the form of a composite tube may be provided to guide the insertion of tool 706 into a target mark 708. The depth map may indicate whether a portion of the composite tube 704 or the target mark 708 is obstructed by other structures. In Figure 11B, neither the composite tube 704 nor the target mark 708 is obstructed, so the graphics for the composite tube 704 and the target mark 708 are presented without any special visual characteristics or processing. In Figure 11C, the tissue 710 blocks a portion of the composite tube 704 and the target mark 708, so the blocked portions of tube 704 and mark 708 may have more translucent visual treatment than the unblocked portions to inform the viewer that the tool exchange path is partially blocked by the tissue 710. In Figure 11D, the tissue 710 blocks a larger portion of the composite tube 704 and completely blocks the target mark 708. The blocked portions of tube 704 and mark 708 may have more translucent visual treatment than the unblocked portions to inform the viewer that the tool exchange path is partially blocked by the tissue 710. In these examples, an opacity cue may provide a visual representation of the blocked portion of the composite indicator. In other examples, other visual properties may be modified in a two-stage rendering process (as described above) to correct color or texture properties to draw more attention to the blocked portion of the guided path.The visual characteristics of the occluded area may be modified in a static or dynamic, time-varying manner. Depth maps may also be used to answer geometric queries regarding the occluded state of the insertion path. One or more rays may be emitted from the tip of tool 706 along the direction of the insertion path toward the target mark. If an unoccluded insertion path closer to the target mark is found, the system may warn the viewer or adjust the composite indicator tube to ensure there are no obstructions.

[0072] A method 840 for displaying a partially occluded composite indicator is illustrated in the flowchart of Figure 14. In process 842, an image of the field of view (e.g., field of view portion 202) of the surgical environment (e.g., environment 201) is displayed, for example, on a display 35. In process 844, a first composite indicator (e.g., composite mark 708) associated with an instrument (e.g., tool 706) in the surgical environment is generated. In process 846, a depth mapping (e.g., depth map 701) including the first composite indicator and structures in the field of view is generated. In process 848, the occluded portion of the first composite indicator, which is occluded by a structure, is generated. In process 850, the first composite indicator is displayed with the occluded portion having a graphic appearance distinct from the unoccluded portion of the first composite indicator.

[0073] Elements described in detail with reference to one embodiment, implementation, or application may, whenever practical, be included in other embodiments, implementations, or applications that are not specifically illustrated or described. For example, if an element is described in detail with reference to one embodiment but not with reference to a second embodiment, that element may nevertheless be claimed as being included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements illustrated and described in relation to one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects, unless otherwise specifically stated, provided that one or more of the elements do not render the embodiment or implementation non-functional, or that two or more of the elements do not provide conflicting functions.

[0074] Any changes and further modifications to the described devices, systems, fixtures, methods, and any further application of the principles of this disclosure are fully assumed to be as commonly conceivable to those skilled in the art relating to this disclosure. In particular, configurations, components, and / or steps described in relation to one embodiment may be combined with configurations, components, and / or steps described in relation to other embodiments of this disclosure. In addition, the dimensions provided herein are for specific examples, and different sizes, dimensions, and / or proportions may be used to implement the concepts of this disclosure. To avoid unnecessary descriptive repetition, one or more components or operations described according to one exemplary embodiment may be used or omitted in a manner applicable from other exemplary embodiments. For the sake of brevity, numerous repetitions of these combinations are not described separately.

[0075] Various systems and parts of systems are described in terms of their states in three-dimensional space. As used herein, the term “position” refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along the Cartesian X, Y, and Z coordinates). As used herein, the term “orientation” refers to the rotational arrangement of an object or part of an object (three rotational degrees of freedom, e.g., roll, pitch, and yaw). As used herein, the term “pose” refers to the position of an object or part of an object in at least one translational degree of freedom and the orientation of an object or part of an object in at least one rotational degree of freedom (up to six degrees of freedom).

[0076] Some of the examples described herein refer to surgical procedures or instruments, or medical procedures and instruments, but the disclosed techniques may optionally apply to non-medical procedures and instruments. For example, the instruments, systems, and methods described herein may be used for non-medical purposes, including industrial applications, general robotic applications, and sensing or manipulating non-tissue workpieces. Other exemplary uses include cosmetic enhancements, imaging of human or animal anatomical structures, data collection from human or animal anatomical structures, and training of medical or non-medical personnel. Additional exemplary uses include use for procedures on tissues removed from human or animal anatomical structures (not returned to human or animal anatomical structures), and performing procedures on human or animal carcasses. Furthermore, these techniques may also be used for surgical and non-surgical medical procedures or diagnostic procedures.

[0077] A computer is a machine that executes mathematical or logical functions on input information in accordance with programmed instructions to produce processed output information. A computer includes logical units that execute mathematical or logical functions, and memory that stores programmed instructions, input information, and output information. The usage of "computer" and similar terms such as "processor," "controller," or "control system" are analogous.

[0078] While certain exemplary embodiments of the present invention are described and illustrated in the accompanying drawings, it should be understood that such embodiments are illustrative only and do not limit the broader invention, and that embodiments of the present invention are not limited to the specific structures and configurations illustrated and described, for various other modifications may come to mind for those skilled in the art. [Prior art documents] [Patent Documents]

[0079] [Patent Document 1] U.S. Patent No. 7,907,166 [Patent Document 2] U.S. Patent No. 8,830,224

Claims

[Claim 1] Display system and, Includes a control system, The control system includes a control unit that includes one or more processors, The control unit is The image of the surgical environment field of view generated by the imaging component is displayed on the display system. A first synthetic indicator is generated that is associated with the instruments in the surgical environment. A depth mapping is generated that includes the structure within the field of view and the first composite indicator. The blocked portion of the first composite indicator, which is blocked by the aforementioned structure, is determined from the depth mapping. The first composite indicator is displayed on the display system. It is configured in such a way, The closed portion of the first composite indicator has a graphic appearance distinct from the unclosed portion of the first composite indicator. Healthcare system.

Images