Systems and methods for facilitating insertion of surgical instruments into a surgical space - Patents.com

Abstract

The exemplary system (100) determines an insertion trajectory that a surgical instrument is predicted to follow as it is inserted into a surgical space. The system provides an image for display by a display device depicting a portion of the surgical space captured by an imaging device and at least a portion of a representation of the insertion trajectory within the surgical space.

Description

[Technical Field]

[0001] (Reference to Related Application) This application claims priority to U.S. Provisional Patent Application No. 62 / 799,258, entitled "SYSTEMS AND METHODS FOR FACILITATING INSERTION OF A SURGICAL INSTRUMENT INTO A SURGICAL SPACE," filed January 31, 2019, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Computer-assisted surgical systems enable surgeons to control surgical instruments to perform surgical procedures on patients. For example, for minimally invasive surgery, surgical instruments are inserted into the patient through one or more cannulas. The surgical instruments typically include an imaging device (e.g., an endoscope) that captures images of the surgical space and one or more surgical tools that are manipulated by the computer-assisted surgical system to perform the surgical procedure. The surgeon views the captured images of the surgical space and uses a master controller of the computer-assisted surgical system to control the movement of the surgical instruments to perform the surgical procedure.

[0003] During a surgical procedure, a guided tool change configuration may be used to facilitate the insertion of a surgical instrument into a predetermined location within a surgical space. However, such a guided tool change configuration may not be available in certain cases. For example, a user may activate a clutch associated with a computer-assisted surgical system during a surgical procedure, thereby disabling the guided tool change configuration. In such cases, the field of view of the imaging device may be adjusted to visualize the opening through which the surgical instrument will be inserted into the surgical space. However, such visualization of the opening from within the surgical space is technically challenging. Moreover, difficulty in visualizing the opening may cause delays if the surgeon must spend significant time adjusting the field of view to find the opening, may harm the patient if the insertion trajectory of the inserted surgical instrument does not clear the anatomical structures and is not visualized, may obscure and / or disrupt the surgical space, and / or may cause a collision between the inserted surgical instrument and one or more additional surgical instruments already inserted within the surgical space. Summary of the Invention

[0004] An exemplary system includes a processor and a memory communicatively coupled to the processor, the memory storing instructions executable by the processor to determine an insertion trajectory that a surgical instrument is predicted to follow as the surgical instrument is inserted into a surgical space, and to provide, for display by a display device, an image depicting at least a portion of a portion of the surgical space captured by the imaging device and a representation of the insertion trajectory within the surgical space.

[0005] An additional exemplary system includes a processor and a memory communicatively coupled to the processor, the memory storing instructions executable by the processor to determine an insertion trajectory that the surgical instrument is expected to follow as it is inserted into the surgical space, provide an image for display by a display device depicting at least a portion of the surgical space captured by the imaging device and a representation of the insertion trajectory within the surgical space, and prevent the surgical instrument from being inserted into the surgical space along the insertion trajectory until a determination is made that the imaging device has been used to view an opening through which the insertion tool is to be inserted.

[0006] An exemplary method includes determining, by an insertion trajectory visualization system, an insertion trajectory that is predicted to be followed when a surgical instrument is inserted into a surgical space, and providing, by the insertion trajectory visualization system, an image depicting at least a portion of the surgical space captured by the imaging device and a representation of the insertion trajectory within the surgical space for display by a display device.

[0007] The accompanying drawings illustrate various embodiments and are a part of the specification. The illustrated embodiments are merely examples and do not limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. [Brief explanation of the drawings]

[0008] [Figure 1] 1 illustrates an exemplary insertion trajectory visualization system in accordance with principles described herein.

[0009] [Figure 2] 1 illustrates an exemplary computer-assisted surgery system in accordance with principles described herein.

[0010] [Figure 3] 1 illustrates a diagram of a surgical instrument within a surgical space according to principles described herein.

[0011] [Figure 4] 1 illustrates an exemplary image of a portion of a surgical space including one or more virtual representations of surgical instruments according to principles described herein. [Figure 5] 1 illustrates an exemplary image of a portion of a surgical space including one or more virtual representations of surgical instruments according to principles described herein. [Figure 6] 1 illustrates an exemplary image of a portion of a surgical space including one or more virtual representations of surgical instruments according to principles described herein. [Figure 7] 1 illustrates an exemplary image of a portion of a surgical space including one or more virtual representations of surgical instruments according to principles described herein. [Figure 8] 1 illustrates an exemplary image of a portion of a surgical space including one or more virtual representations of surgical instruments according to principles described herein. [Figure 9] 1 illustrates an exemplary image of a portion of a surgical space including one or more virtual representations of surgical instruments according to principles described herein.

[0012] [Figure 10] 1 illustrates an exemplary method for facilitating insertion of a surgical instrument into a surgical space in accordance with principles described herein.

[0013] [Figure 11] 1 illustrates an exemplary computing device in accordance with principles described herein. DETAILED DESCRIPTION OF THE INVENTION

[0014] Systems and methods for facilitating insertion of a surgical instrument into a surgical space are described herein. As described in more detail below, an exemplary system includes a memory storing instructions and a processor communicatively coupled to the memory and configured to execute the instructions. The instructions executed by the processor direct the processor to determine an insertion trajectory that the surgical instrument is expected to follow as it is inserted into the surgical space and to provide, for display by a display device (e.g., a display monitor, a stereoscopic image viewer, a virtual reality or augmented reality headset, etc.), an image depicting at least a portion of the surgical space captured by an imaging device (e.g., an endoscope) and a representation of the insertion trajectory within the surgical space.

[0015] Various advantages and benefits are associated with the systems and methods described herein. For example, by providing a representation of an insertion trajectory for display to a user (e.g., a surgeon and / or other users associated with a surgical procedure) of a computer-assisted surgical system described herein, the insertion trajectory visualization system allows the user to quickly and easily see the predicted trajectory a surgical instrument will follow as it is inserted into a surgical space (e.g., without the user having to spend significant time searching for the opening through which the surgical instrument will enter the surgical space). This, in turn, allows the user to adjust the insertion trajectory of the surgical instrument to avoid touching anatomical structures and / or to avoid collision with one or more other surgical instruments already inserted within the surgical space. Additionally, the systems and methods described herein may include features that encourage and / or require the user to use the representation of the insertion trajectory to view the opening through which the surgical instrument will be inserted, which promotes safe surgical practice. These and other benefits that may be realized by the systems and methods described herein will be apparent from the disclosure that follows.

[0016] FIG. 1 illustrates an exemplary insertion trajectory visualization system 100 (“system 100”). As shown, system 100 may include, but is not limited to, a storage facility 102 and a processing facility 104 selectively and communicatively coupled to one another. Facilities 102 and 104 may each include or be implemented by hardware and / or software components (e.g., a processor, memory, a communication interface, instructions stored in memory for execution by the processor, etc.). In some examples, facilities 102 and 104 may be distributed among multiple devices and / or multiple locations, as may be useful for a particular implementation.

[0017] The storage facility 102 may maintain (e.g., store) executable data used by the processing facility 104 to perform any of the operations described herein. For example, the storage facility 102 may store instructions 106 that may be executed by the processing facility 104 to perform any of the operations described herein. The instructions 106 may be embodied by any suitable application, software, code, and / or other executable data instance.

[0018] The storage facility 102 may maintain any data received, generated, managed, used, and / or transmitted by the processing facility 104. For example, as described in more detail below, the storage facility 102 may maintain kinematic data, surgical instrument model data, surgical instrument insertion data, representation data, and the like.

[0019] The processing facility 104 may be configured to perform various processing operations (e.g., by executing instructions 106 stored in the memory facility 102) related to providing a representation of an insertion trajectory for display by a display device. For example, the processing facility 104 may determine an insertion trajectory that a surgical instrument is predicted to follow as it is inserted into the surgical space. When the insertion trajectory extends within the field of view of the imaging device, the processing facility 104 may provide, for display by the display device, an image depicting a portion of the surgical space captured by the imaging device and a representation of the insertion trajectory that is inserted into the surgical space along the insertion trajectory that the surgical instrument is predicted to follow as it is inserted into the surgical space. These and other operations that may be performed by the processing facility 104 are described herein.

[0020] In some implementations, system 100 (e.g., processing facility 104) may operate as part of or in association with a computer-assisted surgical system. As such, an exemplary computer-assisted surgical system is described below. The exemplary computer-assisted surgical system described is exemplary and not limiting. System 100 may operate as part of or in association with a computer-assisted surgical system described herein and / or with other suitable computer-assisted surgical systems.

[0021] 2 illustrates an exemplary computer-assisted surgical system 200 ("surgical system 200"). As shown, surgical system 200 may include an operation system 202, a user control system 204, and an auxiliary system 206 communicatively coupled to each other. In some examples, system 100 may be implemented by one or more of these components.

[0022] The surgical system 200 may be utilized by a surgical team to perform a computer-assisted surgical procedure on a patient 208. As shown, the surgical team may include a surgeon 210-1, an assistant 210-2, a nurse 210-3, and an anesthesiologist 210-4, all of which may be collectively referred to as "surgical team members 210." Additional or alternative surgical team members may be present during a surgical session as may be useful for a particular implementation.

[0023] While FIG. 2 illustrates a minimally invasive surgical procedure in progress, surgical system 200 may likewise be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the precision and convenience of surgical system 200. In addition, it will be understood that a surgical session in which surgical system 200 may be utilized may not only include the operative phase of a surgical procedure as illustrated in FIG. 2 , but may also include pre-operative, post-operative, and / or other appropriate phases of a surgical procedure. A surgical procedure may include any procedure in which manual and / or instrumentation techniques (e.g., remotely operated instrumentation techniques) are used on a patient to investigate, diagnose, or treat a physical condition of the patient. In addition, a surgical procedure may include any procedure not performed on a live patient, such as calibration procedures, training procedures, and experimental or research procedures.

[0024] 2, manipulation system 202 may include multiple manipulator arms 212 (e.g., manipulator arms 212-1 through 212-4) to which multiple surgical instruments (not shown) may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool having a tissue-interacting configuration), medical tool, monitoring instrument (e.g., an imaging device such as an endoscope), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or the like that may be used for computer-assisted surgery (e.g., by being at least partially inserted into patient 208 and manipulated to perform computer-assisted surgery on patient 208). While manipulator system 202 is illustrated and described herein as including four manipulator arms 212, it will be appreciated that manipulator system 202 may include only a single manipulator arm 212 or any other number of manipulator arms that may be useful in a particular implementation.

[0025] The manipulator arm 212 and / or a surgical instrument attached to the manipulator arm 212 may include one or more displacement transducers, orientation sensors, and / or position sensors (hereinafter "surgical system sensors") that are used to generate raw (i.e., uncorrected) kinematic information. One or more components of the surgical system 200 may be configured to track (e.g., determine the position of) and / or control the surgical instrument using the kinematic information.

[0026] Additionally, each manipulator arm 212 may include or be otherwise associated with multiple motors that control movement of the manipulator arm 212 and / or a surgical instrument attached to the manipulator arm. For example, manipulator arm 212-1 may include or be otherwise associated with a first internal motor (not explicitly shown) configured to yaw (yaw) the manipulator arm 212-1 about a yaw axis. Similarly, manipulator arm 212-1 may be associated with a second internal motor (not explicitly shown) configured to drive and pitch (pitch) the manipulator arm 212-1 about a pitch axis. Similarly, manipulator arm 212-1 may be associated with a third internal motor (not explicitly shown) configured to slide the manipulator arm 212-1 along an insertion axis. Manipulator arms 212 may each include a drive train system driven by one or more of these motors to control the pivoting of manipulator arm 212 in any manner that may be useful for a particular implementation. Thus, for example, if a surgical instrument attached to manipulator arm 212-1 is to be mechanically moved, one or more of the motors coupled to the drive train may be energized to move manipulator arm 212-1.

[0027] In particular examples, the manipulator arm 212 may have one or more clutch modes that facilitate disengaging the manipulator arm 212 from one or more motors of the manipulator arm 212. The manipulator arm 212 may have any suitable number of clutch modes that may be useful for a particular implementation. For example, a first clutch mode may be engaged to allow manual rotation of the manipulator arm 212-1 about a yaw axis, a second clutch mode may be engaged to allow manual rotation of the manipulator arm 212-1 about a pitch axis, and a third clutch mode may be engaged to allow manual translation of the manipulator arm 212-1 along an insertion axis. Any suitable number of clutch modes may be engaged at a particular time to facilitate a user manually repositioning the insertion trajectory of a surgical instrument attached to the manipulator arm 212.

[0028] The surgical instruments attached to the manipulator arms 212 may each be positioned in a surgical space associated with the patient. The "surgical space" may, in certain instances, be located entirely within the patient and may include an area within the patient at or near where a surgical procedure is planned to be performed, is being performed, or has been performed. For example, for a minimally invasive surgical procedure performed on tissue within the patient, the surgical space may include the tissue, the anatomical structures underlying the tissue, as well as the space surrounding the tissue where, for example, surgical instruments used to perform the surgical procedure are placed. In other instances, the surgical space may be located at or near where a surgical procedure is planned to be performed on the patient, but at least partially external to the patient. For example, the surgical system 200 may be used to perform an open surgical procedure such that a portion of the surgical space (e.g., the tissue being operated on) is internal to the patient, while another portion of the surgical space (e.g., the space surrounding the tissue where one or more surgical instruments may be placed) is external to the patient. A surgical instrument may be referred to as being positioned or disposed at or within a surgical space when at least a portion of the surgical instrument (e.g., a distal portion of the surgical instrument) is positioned within the surgical space.

[0029] The user control system 204 may be configured to facilitate control by the surgeon 210-1 of the manipulator arm 212 and the surgical instruments attached to the manipulator arm 212. For example, the surgeon 210-1 may interact with the user control system 204 to remotely move or manipulate the manipulator arm 212 and the surgical instruments. To this end, the user control system 204 may provide the surgeon 210-1 with images (e.g., high-definition three-dimensional (3D) imagery) of the surgical space associated with the patient 208 as captured by an imaging device. In particular examples, the user control system 204 may include a stereoscopic image viewer having two displays on which the surgeon 210-1 may view stereoscopic images (e.g., 3D images) of the surgical space associated with the patient 208 and generated by a stereoscopic imaging system. The surgeon 210-1 may use the images to perform one or more procedures using one or more surgical instruments attached to the manipulator arm 212.

[0030] To facilitate control of surgical instruments, user control system 204 may include a set of master controllers (not shown). These master controllers may be operated by surgeon 210-1 to control the movement of the surgical instruments (e.g., by utilizing robotic and / or teleoperation techniques). The master controllers may be configured to detect a wide variety of hand, wrist, and finger movements by surgeon 210-1. In this manner, surgeon 210-1 may intuitively perform a surgical procedure using one or more surgical instruments.

[0031] The user control system 204 may be further configured to facilitate control by the surgeon 210-1 of other components of the surgical system 200. For example, the surgeon 210-1 may interact with the user control system 204 to change the configuration or operating mode of the surgical system 200, change the display mode of the surgical system 200, generate additional control signals used to control surgical instruments attached to the manipulator arm 212, facilitate control of switching from one surgical instrument to another, initiate the display of a representation of the insertion trajectory, or perform any other suitable operation. To this end, the user control system 204 may include one or more input devices (e.g., foot pedals, buttons, switches, etc.) configured to receive input from the surgeon 210-1.

[0032] The auxiliary system 206 may include one or more computing devices configured to perform primary processing operations for the surgical system 200. The one or more computing devices included in the auxiliary system 206 may control and / or coordinate operations performed by various other components of the surgical system 200 (e.g., the manipulation system 202 and / or the user control system 204). For example, a computing device included in the user control system 204 may send instructions to the manipulation system 202 via one or more computing devices included in the auxiliary system 206. As another example, the auxiliary system 206 may receive and process image data from the manipulation system 202 representing an image captured by an imaging device attached to one of the manipulator arms 212.

[0033] In some examples, the auxiliary system 206 may be configured to present visual content to surgical team members 210 who may not have access to the images provided to the surgeon 210-1 by the user control system 204. To this end, the auxiliary system 206 may include a display monitor 214 configured to display one or more user interfaces, such as an image of the surgical space (e.g., a 2D image), information related to the patient 208 and / or the surgical procedure, and / or any other visual content that may be useful in a particular implementation. For example, the display monitor 214 may display an image of the surgical space along with additional content (e.g., a representation of the insertion trajectory, graphical content, contextual information, etc.) displayed simultaneously with the image. In some embodiments, the display monitor 214 is implemented by a touchscreen display with which the surgical team members 210 may interact (e.g., via touch gestures) to provide user input to the surgical system 200.

[0034] The operation system 202, the user control system 204, and the auxiliary system 206 may be communicatively coupled to one another in any suitable manner. For example, as shown in Figure 2, the operation system 202, the user control system 204, and the auxiliary system 206 may be communicatively coupled by control lines 216, which may represent any wired or wireless communication links as may be useful in a particular implementation. To this end, the operation system 202, the user control system 204, and the auxiliary system 206 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0035] FIG. 3 illustrates a view 300 including various surgical instruments that may be coupled to the manipulator arm 212 of the surgical system 200 and disposed within the surgical space. As shown, the surgical instruments may include an imaging device 302 and one or more other surgical instruments 304 in the form of one or more surgical tools (e.g., surgical instruments 304-1 through 304-3). While FIG. 3 illustrates one imaging device 302 and three surgical tools disposed within the surgical space, any number, type, and / or combination of imaging devices and surgical tools may be present in the surgical space during a surgical session. In certain implementations, for example, other types of surgical tools (e.g., diagnostic tools, therapeutic tools, etc.) different from those illustrated in FIG. 3 may additionally or alternatively be provided within the surgical space during a surgical procedure. Tissue 306 represents anatomical tissue in the surgical space.

[0036] The imaging device 302 may capture images in the surgical space. Any surgical instruments 304 and / or tissue 306 within the field of view of the imaging device 302 may be depicted in the images captured by the imaging device 302.

[0037] The imaging device 302 may provide data representing visible light data of the surgical space. For example, the imaging device 302 may capture a visible light image of the surgical space representing the visible light sensed by the imaging device 302. The visible light image may include an image using any suitable color and / or grayscale palette to represent a visible light-based view of the surgical space.

[0038] The imaging device 302 may provide data representing depth data of the surgical space or data that may be processed to derive depth data of the surgical space. For example, the imaging device 302 may capture an image of the surgical space representing the depth sensed by the imaging device 302. Alternatively, the imaging device 302 may capture an image of the surgical space that may be processed to derive depth data of the surgical space. The depth information may be represented as a depth image (e.g., a depth map image obtained using a Z-buffer indicating the distance from the imaging device 302 to each pixel point on the image of the surgical space), which may be configured to visually indicate the depth of objects in the surgical space in any suitable manner, for example, by using different grayscale values ​​to represent different depth values. Images captured by the imaging device (e.g., by the imaging device 302) and / or derived from images captured by the imaging device (e.g., visible light images and depth images) may be used to facilitate insertion of surgical instruments within the surgical space as described herein.

[0039] During a surgical session, a surgical instrument (e.g., surgical instrument 304 and / or one of several other surgical instruments) may need to be inserted or reinserted into the surgical space. To facilitate such insertion of the surgical instrument into the surgical space, system 100 is configured to provide a representation of the insertion trajectory so that a user (e.g., surgeon 210-1, assistant 210-2, etc.) can view the insertion trajectory that the surgical instrument is expected to follow before it is inserted into the surgical space.

[0040] To this end, system 100 is configured to determine an insertion trajectory that the surgical instrument is expected to follow. System 100 may determine the insertion trajectory in any suitable manner using any suitable information related to the surgical instrument. For example, system 100 may determine the insertion trajectory based on the kinematics of a manipulator arm (e.g., manipulator arm 212-1) to which the surgical instrument is coupled. The kinematics of manipulator arm 212 may be defined by kinematic data, which may represent the position, pose, and / or orientation of components of surgical system 200 and / or components coupled to surgical system 200. For example, the kinematic data may represent the position, pose, and / or orientation of manipulator arm 212-1 and / or a surgical instrument coupled to manipulator arm 212-1.

[0041] Additionally or alternatively, system 100 may determine the insertion trajectory based on captured image data (e.g., one or more images captured by one or more imaging devices located within and / or outside the surgical space) of a manipulator arm coupled to the surgical instrument. The captured image data may indicate the position, posture, and / or orientation of the manipulator arm relative to the surgical space. System 100 may process the captured image data in any suitable manner to determine an insertion trajectory that the surgical instrument is expected to follow as it enters the surgical space.

[0042] Additionally or alternatively, system 100 may determine the insertion trajectory based on the last known position of the surgical instrument as it is inserted into the surgical space. System 100 may determine the last known position in any suitable manner. For example, system 100 may utilize surgical instrument insertion data that includes information indicative of a previous position, a previous pose, and / or a previous orientation of the surgical instrument within the surgical space to determine the insertion trajectory that the surgical instrument is expected to follow.

[0043] Determining the predicted insertion trajectory by system 100 includes system 100 determining the location of the predicted insertion trajectory of the surgical instrument relative to the surgical space. For example, system 100 may generate a 3D coordinate space representing the surgical space (e.g., of a portion of the surgical space or the entire surgical space). System 100 may generate the 3D coordinate space representing the surgical space in any suitable manner. For example, kinematic data and / or imaging data captured by one or more imaging devices may be used by system 100 to generate the 3D coordinate space representing the surgical space, e.g., by registering the kinematic data and imaging data to a common 3D coordinate system. Registration may include registering the predicted insertion trajectory of the surgical instrument to the 3D coordinate space representing the surgical space.

[0044] To illustrate one example, the imaging device 302 may capture images including visible light data and depth data related to the surgical space. Based on the captured images, the system 100 may generate a 3D coordinate space representing the surgical space. The system 100 may map a predicted insertion trajectory of the surgical instrument into the 3D coordinate space, and the mapping may be used in any suitable manner, as described herein, to facilitate insertion of the surgical instrument into the surgical space.

[0045] In an alternative example, the system 100 may access kinematic data indicating the pose, position, and / or orientation of a manipulator arm to which a surgical instrument is coupled. Based on the kinematic data, the system 100 may generate a 3D coordinate space representing the surgical space. The system 100 may map images captured by the imaging device 302 in any suitable manner to the 3D coordinate space generated based on the kinematic data to facilitate insertion of the surgical instrument into the surgical space.

[0046] In certain alternative examples, the system 100 may generate a 3D coordinate space based on a combination of images captured by the imaging device 302 and kinematic data associated with a manipulator arm to which a surgical instrument is coupled. For example, the system 100 may first generate a 3D coordinate space representing the surgical space based on the kinematic data in any suitable manner. Following generation of the 3D coordinate space based on the kinematic data, the system 100 may use images captured by the imaging device 302 to correct, update, and / or fine-tune the 3D coordinate space. For example, the 3D coordinate space generated based on the kinematic data may accurately represent the relative positions of one or more surgical instruments within the surgical space. However, the images captured by the imaging device 302 may provide additional information regarding the current positions of anatomical structures within the surgical space. In such examples, the 3D coordinate space may be updated based on the captured images to accurately reflect the current positions of the anatomical structures relative to the relative positions of the one or more surgical instruments.

[0047] When the imaging device 302 captures images of a portion of the surgical space during a surgical session and a predicted insertion trajectory of the surgical instrument extends within the field of view of the imaging device 302, the system 100 may provide at least a portion of a representation of the insertion trajectory along the predicted insertion trajectory for display by a display device (e.g., a stereoscopic image viewer provided as part of the user control system 204 and / or display monitor 214). The system 100 may provide the representation of the insertion trajectory for display along the predicted insertion trajectory in any suitable manner, such that the representation of the insertion trajectory is displayed with images captured by the imaging device 302. In doing so, a user viewing the display device can see the insertion trajectory that the surgical instrument is predicted to follow.

[0048] As used herein, a "representation of an insertion trajectory" may correspond to any suitable image, graphic, or animation that may be used to illustrate the insertion trajectory that a surgical instrument is expected to follow as it is inserted into a surgical space. The representation of an insertion trajectory may have any suitable shape, size, and / or visual appearance as may be useful for a particular implementation. For example, the representation of an insertion trajectory may be transparent, semi-transparent, opaque, colored, and / or patterned. In certain examples, the representation of an insertion trajectory may have a 3D appearance when displayed by a display device.

[0049] In some examples, the representation of the insertion trajectory may be a virtual representation of the surgical instrument inserted along an insertion trajectory that the surgical instrument is predicted to follow as it is inserted into the surgical space. In particular examples, such a virtual representation of the surgical instrument may be a realistic representation of the surgical instrument (e.g., may have the same visual appearance as the surgical instrument represented by the virtual representation). For example, if the surgical instrument to be inserted into the surgical space is a pair of surgical scissors, the virtual representation may have at least the same contours as a pair of surgical scissors. In particular examples, the system 100 may use a virtual model (e.g., a 3D model) of the surgical instrument to create the realistic virtual representation of the surgical instrument. Such a virtual model may be generated based on surgical instrument model data stored by the storage facility 102.

[0050] Alternatively, the representation of the insertion trajectory may be more schematic in nature, such that it has a different visual appearance than the surgical instrument. For example, the representation of the insertion trajectory may, in certain implementations, be represented by a dotted or dashed line provided along the predicted insertion trajectory. Alternatively, the more schematic version of the representation of the insertion trajectory may, in certain implementations, have a generic appearance (e.g., a stick-like appearance).

[0051] The visual appearance that a representation of an insertion trajectory may have in a particular implementation may be defined by representation data stored by storage facility 102. Such representation data may include any data related to a representation of an insertion trajectory. For example, the representation data stored in storage facility 102 may indicate that a representation provided for display by system 100 should have a particular color, size, shape, translucency, brightness, etc. In particular examples, system 100 may provide one or more user interfaces that allow a user to adjust any suitable settings related to a representation of an insertion trajectory. For example, the one or more user interfaces may allow a user to adjust the visual appearance (e.g., color, shape, size, translucency, brightness, etc.) that a representation of an insertion trajectory has when the representation of an insertion trajectory is provided for display by a display device.

[0052] In certain examples, the representation of the insertion trajectory may be color-coded to indicate the type of surgical instrument to be inserted along the predicted insertion trajectory. For example, the representation of the insertion trajectory may have a first color when representing a first type of surgical instrument and a second color, different from the first color, when representing a second type of surgical instrument.

[0053] The representation of the insertion trajectory may be provided for display by the display device to graphically illustrate the predicted insertion trajectory of the surgical instrument in any suitable manner. For example, the representation of the insertion trajectory may be overlaid in any suitable manner on top of the image captured by the imaging device 302 and displayed by the display device. Alternatively, the image captured by the imaging device 302 may be modified in any suitable manner to include the representation of the insertion trajectory prior to and / or along with the image provided for display by the display device. As an example, when a view of the surgical field is rendered and a 3D model representation of the surgical instrument at the predicted insertion trajectory is within the rendered view, the system 100 may insert a 3D model representing the surgical instrument at the predicted insertion trajectory into a virtual 3D coordinate space representing the surgical field such that the rendering may include the 3D model as a natural part of the surgical space as represented in the 3D coordinate space. The system 100 may use any suitable rendering model and / or data representing the virtual representation of the surgical instrument to provide the virtual representation for display. For example, the system may use any suitable 3D rendering module (eg, Qt3D) and / or any suitable 3D file format (eg, glTF).

[0054] In certain examples, system 100 may selectively provide a representation of the insertion trajectory for display by a subset of display devices associated with surgical system 200. For example, the representation of the insertion trajectory may unduly obscure the surgical view of surgeon 210-1 when the representation of the insertion trajectory is provided for display on a stereoscopic image viewer comprising user control system 204. In such examples, the representation of the insertion trajectory may only be provided for display on display monitor 214, as opposed to being provided for display on a stereoscopic image viewer comprising both user control system 204 and display monitor 214.

[0055] In certain examples, system 100 may additionally or alternatively provide a representation of the insertion trajectory for display by one or more additional display devices (e.g., tablet computers, smartphones, etc.) that are not part of surgical system 200 but are communicatively coupled to surgical system 200.

[0056] In certain examples, system 100 may provide a representation of the insertion trajectory for display in a graphical user interface that simultaneously displays multiple different views of the surgical space captured by multiple different imaging devices. For example, such a graphical user interface may simultaneously display a first view including an image of the surgical space captured by imaging device 302 and a second view including an image of the surgical space captured by an additional imaging device. In certain examples, the image captured by the additional imaging device may include an image of an opening (e.g., a cannula) through which the surgical instrument is to be inserted into the surgical space. Such a graphical user interface may be provided for display on a stereoscopic image viewer comprising user control system 204, on display monitor 214, and / or by any other suitable display device that may be part of or communicatively coupled to surgical system 200. With such a graphical user interface, a user may not need to move an imaging device (e.g., imaging device 302) to view the opening through which the surgical instrument is to be inserted into the surgical space.

[0057] System 100 may provide a representation of the insertion trajectory for display at any appropriate time and / or in response to any appropriate input provided by a user. For example, surgeon 210-1 may provide any appropriate input through user control system 204 to initiate a representation of the insertion trajectory being displayed by the display device. Alternatively, assistant 210-2 may provide any appropriate input (e.g., touch input) through display monitor 214 to initiate a representation of the insertion trajectory being displayed by the display device. If the predicted insertion trajectory is not currently within the field of view of the imaging device, the user may adjust the field of view of the imaging device until a representation of the insertion trajectory is displayed by the display device. Alternatively, if the predicted insertion trajectory is currently within the field of view of the imaging device, a representation of the insertion trajectory may be provided for display by the display device.

[0058] System 100 may stop providing a representation of the insertion trajectory for display by the display device at any appropriate time and / or in response to any appropriate input provided by a user. For example, system 100 may detect a command provided by surgeon 210-1 through user control system 204 instructing system 100 to stop displaying the representation of the insertion trajectory. Additionally or alternatively, system 100 may detect a command provided by assistant 210-1 or some other user through assistance system 206 instructing system 100 to stop displaying the representation of the insertion trajectory. In certain examples, system 100 may automatically stop providing a representation of the insertion trajectory for display when the surgical instrument begins to be inserted into the surgical space (e.g., when a distal portion of the surgical instrument exits the cannula and enters the surgical space) without requiring further input from the user. In certain examples, system 100 may automatically stop providing a representation of the insertion trajectory for display when the actual surgical instrument enters the field of view of the imaging device or when the actual surgical instrument is at any other suitable point along the insertion trajectory. In certain examples, system 100 may automatically resume providing a representation of the insertion trajectory for display at any suitable time. For example, system 100 may resume providing a representation of the insertion trajectory for display if the field of view of the imaging device is changed such that the surgical instrument to be inserted is no longer within the current field of view.

[0059] In certain examples, system 100 may provide an updated representation of the insertion trajectory for display by a display device. For example, a representation of the insertion trajectory may be provided for display by a display device before the surgical instrument is inserted into the surgical space, but a user (e.g., assistant 210-2 or some other user) may change the position of, for example, the manipulator arm 212-1 to which the surgical instrument is attached. The change in position of the manipulator arm 212-1 changes the kinematics of the manipulator arm 212-1. Based on the change in the kinematics of the manipulator arm 212-1, system 100 may determine an updated insertion trajectory that the surgical instrument is predicted to follow as it is inserted into the surgical space. System 100 may then provide, for real-time display by a display device, an updated representation of the insertion trajectory that is along the updated insertion trajectory that the surgical instrument is predicted to follow as it is inserted into the surgical space. That is, in certain examples, the representation of the insertion trajectory provided for display by the display device may move in real time in response to changes in the user's position of the manipulator arm 212-1. This allows the user to see in real time how changes in the kinematics of the manipulator arm 212-1 affect the predicted insertion trajectory. This may help the user to beneficially adjust the position of the manipulator arm 212-1 for insertion of the surgical instrument (e.g., to prevent the surgical instrument from colliding with another surgical instrument and / or to prevent the surgical instrument from contacting an anatomical structure when ultimately inserted into the surgical space).

[0060] In certain examples, the system 100 may determine an insertion depth that the surgical instrument is predicted to achieve when inserted into the surgical space. The system 100 may determine the insertion depth in any suitable manner. For example, the system 100 may determine the insertion depth based on depth data captured by the imaging device 302, based on a previous insertion depth of the surgical instrument, based on the kinematics of a manipulator arm to which the surgical instrument is attached, and / or based on any other suitable information. In certain alternative examples, the system 100 may determine the insertion depth based on input specifically provided by the user. For example, the system 100 may be configured to facilitate the user dropping virtual pins in any suitable manner to specify the location and insertion depth at which the surgical instrument is inserted into the surgical space. Alternatively, the system 100 may automatically determine the insertion depth as at least a predetermined distance away from the tissue based on the depth data captured by the imaging device 302. When the insertion trajectory extends within the field of view of the imaging device, the system 100 may provide at least a portion of a representation of the insertion trajectory to virtually represent the predicted insertion trajectory extending to the determined insertion depth (e.g., by visually representing a surgical instrument inserted into the surgical space at the determined insertion depth along the predicted insertion trajectory).

[0061] In certain alternative examples, the representation of the insertion trajectory may extend beyond the determined insertion depth when provided for display by the display device. In doing so, system 100 may provide a visualization of the representation of the insertion trajectory when the insertion trajectory extends into the field of view of the imaging device but the determined insertion depth is not deep enough to result in the representation being provided for display within the current field of view of imaging device 302. Exemplary representations inserted along the predicted insertion trajectory will now be described with reference to FIGS.

[0062] 4A-9 depict various example virtual representations of predicted insertion trajectories and surgical instruments. It is understood that the example virtual representations of surgical instruments and / or example predicted insertion trajectories may be considered as representations of insertion trajectories in particular examples. Moreover, although FIGS. 4A-9 and the accompanying description are described with reference to virtual representations of surgical instruments, it is understood that any suitable representation of insertion trajectories as described herein may be provided in the examples shown in FIGS. 4A-9 in particular implementations.

[0063] 4A and 4B show an example sequence of images 400 (e.g., images 400-1 and 400-2) of a portion of a surgical space captured by imaging device 302. The sequence of images 400 shown in FIGS. 4A and 4B illustrates an animation of a virtual representation 402 of a surgical instrument (e.g., surgical instrument 304-2) entering the surgical space within the field of view of imaging device 302. As shown in FIGS. 4A and 4B, virtual representation 402 is shown entering the surgical space along a predicted insertion trajectory 404 toward a predicted insertion depth 406 in the direction of arrow 408. Virtual representation 402 is an example of a generic stick-like representation that has a different visual appearance than a surgical instrument that is predicted to follow predicted insertion trajectory 404 as it is inserted into the surgical space. In the example shown in FIGS. 4A and 4B, virtual representation 402 is shown moving along predicted insertion trajectory 404 as the image transitions from image 400-1 shown in FIG. 4A to image 400-2 shown in FIG. 4B. In particular examples, the virtual representation 402 may continue to move along the predicted insertion trajectory 404 until the virtual representation 402 reaches the predicted insertion depth 406. Alternatively, as described above, the virtual representation 402 may extend beyond the predicted insertion depth 406 in particular implementations.

[0064] 4A and 4B , the arrow 408, the dashed line indicating the predicted insertion trajectory 404, and the line indicating the predicted insertion depth 406 are shown along with the virtual representation 402 to facilitate illustrating how the virtual representation 402 may be depicted in an image captured by the imaging device 302. However, it is understood that in particular implementations, the arrow 408, the dashed line indicating the predicted insertion trajectory 404, and the line indicating the predicted insertion depth 406 may not be provided for display in the image 400 along with the virtual representation 402. That is, in the example shown in FIGS. 4A and 4B , only the virtual representation 402 may be provided for display in the image 400 in particular examples. Alternatively, only the dashed line indicating the predicted insertion trajectory 404 may be provided for display in the image 400 in particular examples.

[0065] In certain alternative examples, the predicted insertion trajectory 404 may not be within the current field of view of the imaging device 302. In such examples, a user may provide any appropriate instructions to cause the virtual representation 402 to be provided for display. For example, the user may reposition the field of view of the imaging device 302 to find where the virtual representation 402 is located within the surgical space. In such examples, the virtual representation 402 may appear to be already inserted within the surgical space as the field of view pans to where the virtual representation 402 is located, as opposed to being provided for display as an animation depicting the virtual representation 402 being inserted along the predicted insertion trajectory 404.

[0066] In certain instances, it may be useful to visualize the opening through which the surgical instrument will be inserted before the surgical instrument is inserted into the surgical space along the predicted insertion trajectory. This may be beneficial when the predicted insertion trajectory is unclear (e.g., obstructed by anatomical structures and / or one or more other surgical instruments) and / or to promote safe surgical performance. System 100 may facilitate visualization of the opening in any suitable manner. For example, a virtual representation of the surgical instrument provided for display by system 100 may be used to guide the field of view of imaging device 302 back toward the opening in a direction proximal to the virtual representation.

[0067] 5 depicts an image 500 of a portion of a surgical space captured by imaging device 302 after the field of view of imaging device 302 has been moved along the length of virtual representation 402 from the field of view shown in FIGS. 4A and 4B. As shown in FIG. 5, the movement of the field of view along the length of virtual representation 402 results in the visualization of an opening 502 of cannula 504 through which surgical instrument 304-2 is provided. By using virtual representation 402 as a guide, the field of view of imaging device 302 can be remotely manipulated to easily and quickly locate opening 502 of cannula 504, ensuring that instrument 304-2 may be safely inserted into the surgical space (e.g., without contacting anatomical structures and / or other surgical instruments within the surgical space) along the predicted insertion trajectory 404.

[0068] In particular examples, system 100 may prevent a surgical instrument from being inserted into the surgical space along the predicted insertion trajectory until a determination is made that imaging device 302 has been used to visualize the opening through which the surgical tool is to be inserted. This may be achieved in any suitable manner. For example, system 100 may lock one or more motors included in or associated with a manipulator arm (e.g., manipulator arm 212-2) to which the surgical instrument is coupled in place to prevent movement of the manipulator arm. System 100 may verify that the opening has been visualized by imaging device 302 in any suitable manner. For example, system 100 may utilize any suitable image recognition technology to determine that a cannula is included in an image captured by imaging device 302. Alternatively, system 100 may determine that imaging device 302 has visualized the opening by detecting a dedicated computer vision marker on the cannula (e.g., a barcode, color, pattern, etc. integrated into or otherwise attached to the cannula). After the system 100 determines that the imaging device 302 has been used to visualize the opening, the system 100 may unlock one or more motors included in or associated with the manipulator arm to allow the surgical instrument to be inserted along the predicted insertion trajectory.

[0069] In particular examples, system 100 may determine an additional insertion trajectory that the additional surgical instrument is predicted to follow as it is inserted into the surgical space in any suitable manner, as described herein. System 100 may provide, for display by a display device, an additional virtual representation of the additional surgical instrument inserted into the surgical space along the additional insertion trajectory that the additional surgical instrument is predicted to follow as it is inserted into the surgical space. In such examples, the virtual representation of the surgical instrument may be provided for display by the display device during a first time period, and the additional virtual representation of the additional surgical instrument may be provided for display by the display device during a second time period that does not overlap with the first time period. Alternatively, the virtual representation of the surgical instrument and the additional virtual representation of the additional surgical instrument may be provided for simultaneous display by the display device.

[0070] 6A and 6B show an exemplary sequence of images 600 (e.g., images 600-1 and 600-2) of a portion of a surgical space captured by imaging device 302. The sequence of images 600 shown in Figures 6A and 6B illustrates an animation of virtual representations 602 (e.g., virtual representations 602-1 and 602-2) of surgical instruments (e.g., surgical instruments 304-2 and 304-3) simultaneously entering the surgical space. As shown in Figures 6A and 6B, virtual representation 602-1 is shown entering the surgical space along predicted insertion trajectory 604-1 toward predicted insertion depth 606-1, and virtual representation 602-2 is shown entering the surgical space along predicted insertion trajectory 604-2 toward predicted insertion depth 606-2. Similar to the examples shown in Figures 4A and 4B, the images captured by imaging device 302 transition from image 600-1 shown in Figure 6A to image 600-2 shown in Figure 6B, depicting animation of virtual representations 602-1 and 602-2 being inserted into the surgical space in the direction of arrows 608-1 and 608-2, respectively.

[0071] When virtual representations of surgical instruments are provided for simultaneous display by a display device, the virtual representations may be visually distinct from one another in any suitable manner. In the example shown in Figures 6A and 6B, the shading provided on virtual representations 602-1 and 602-2 differs to facilitate a user's distinction between the two.

[0072] 7A and 7B show an exemplary sequence of images 700 (e.g., images 700-1 and 700-2) of a portion of a surgical space captured by imaging device 302, which may be provided for display when the virtual representation of a surgical instrument has a relatively more realistic visual appearance. As shown in FIGS. 7A and 7B, a virtual representation 702 of a surgical instrument is shown being inserted along a predicted insertion trajectory 704 toward a predicted insertion depth 706 in the direction of arrow 708. In the example shown in FIGS. 7A and 7B, virtual representation 702 has a similar outline to surgical instrument 304-2, but is shown with dashed lines to indicate that virtual representation 702 is at least partially transparent, which facilitates a user in distinguishing virtual representation 702 from actual surgical instrument 304-2.

[0073] In certain examples, system 100 may determine that the insertion trajectory predicted for the surgical instrument to follow is unclear (e.g., an object, such as another surgical instrument and / or an anatomical structure, may be obstructing the predicted insertion trajectory). This may be accomplished in any suitable manner. For example, system 100 may utilize one or more depth sensors included in imaging device 302 and / or one or more surgical instruments 304, for example, to determine that the additional surgical instrument and / or anatomical structure is within the predicted insertion trajectory. Additionally or alternatively, system 100 may utilize kinematic data associated with the additional surgical instrument to determine that the additional surgical instrument is within the predicted insertion trajectory. In certain examples, system 100 may provide one or more notifications to facilitate preventing the surgical instrument from contacting an anatomical structure and / or another surgical instrument that is in the way of the predicted insertion trajectory. For example, system 100 may provide, for display by a display device, a virtual representation of the surgical instrument along with a notification indicating that the surgical instrument is predicted to contact additional surgical instruments if the surgical instrument is inserted along the insertion trajectory that the surgical instrument is predicted to follow. Such a notification may have any suitable shape or form and may be provided for display at any suitable location within the image captured by imaging device 302.

[0074] To illustrate, FIG. 8 shows an image 800 of a portion of a surgical space captured by imaging device 302. As shown in FIG. 8, a virtual representation 802 of a surgical instrument is depicted entering the surgical space along a predicted insertion trajectory 804 toward a predicted insertion depth 806. However, because the predicted insertion trajectory 804 results in the surgical instrument represented by virtual representation 802 coming into contact with surgical instrument 304-1, FIG. 8 includes a notification 808 indicating that a collision is predicted to occur if the surgical instrument is inserted along the predicted insertion trajectory 804. In the example shown in FIG. 8, notification 808 is provided for display along insertion trajectory 804. However, notification 808 and / or additional notifications may be provided elsewhere in particular implementations. In response to notification 808, a user may then reposition surgical instrument 304-1 within the cannula and / or change the insertion depth to which the surgical instrument is inserted along insertion trajectory 804 to avoid contacting surgical instrument 304-1.

[0075] In particular examples, system 100 may be configured to provide, for display by a display device, an indicator instructing or encouraging a user to follow a predicted insertion trajectory using imaging device 302 to view the opening through which the surgical instrument will be inserted into the surgical space, along with a virtual representation of the surgical instrument. Such an indicator may be provided for display in any suitable manner. For example, the indicator may be provided on or to the side of the virtual representation of the surgical instrument. To illustrate, FIG. 9 shows an exemplary image 900 of a portion of a surgical space captured by imaging device 302, which includes a virtual representation 902 provided along a predicted insertion trajectory 904 toward a predicted insertion depth 906. As shown in FIG. 9 , indicator 908 includes an eye icon provided over virtual representation 902 to remind the user to adjust the field of view of imaging device 302 to view the opening through which the surgical instrument represented by virtual representation 902 will be inserted.

[0076] System 100 may be configured to provide an indicator, such as indicator 908, for display at any suitable time and / or in response to any suitable event. In particular examples, system 100 may display indicator 908 by default along with the virtual representation 902 of the surgical instrument along the predicted insertion trajectory 904. In other examples, system 100 may display indicator 908 along with the virtual representation 902 in response to an event, such as system 100 detecting that an anatomical structure or another surgical instrument may obstruct the predicted insertion trajectory 904.

[0077] In certain examples, the system 100 may provide an indication to the user when the predicted insertion trajectory is adjusted to intersect with a predetermined location within the surgical space. For example, the system 100 may determine that the user has dropped a virtual pin within the surgical space to define a specific location and insertion depth at which the surgical instrument should be inserted. The user may engage one or more clutch modes to clutch a manipulator arm to which the surgical instrument is attached and manually move the manipulator arm to adjust the predicted insertion trajectory of the surgical instrument. While the manipulator arm is moved, the system 100 may provide a virtual representation of the surgical instrument, shown moving in real time based on the movement of the clutched manipulator arm, for display by any suitable display device. The system 100 may provide an indication to the user when the virtual representation of the surgical instrument intersects with the virtual pin dropped by the user. For example, the system 100 may change the color, texture, size, and / or any other visual aspect of the virtual representation. Additionally or alternatively, system 100 may provide instructions through haptic feedback provided to the user. System 100 may provide such haptic feedback in any suitable manner. For example, while the user manually moves the manipulator arm, system 100 may cause the manipulator arm to vibrate to resist further movement and / or lock into place when the virtual representation of the surgical instrument intersects a virtual pin dropped by the user. Additionally or alternatively, system 100 may provide such haptic feedback through a master control provided in user control system 204. For example, the master control provided in user control system 204 may vibrate when the virtual representation of the surgical instrument intersects a virtual pin dropped by the user and / or may provide any other suitable type of haptic feedback. Such feedback helps the user easily determine when the manipulator arm is positioned so that the insertion trajectory intersects a predetermined location in the surgical space.After the manipulator arm is positioned, the user may then disengage one or more clutch modes to lock the manipulator arm in place so that the surgical instrument may be inserted into the surgical space along the insertion trajectory to the desired position.

[0078] In certain examples, when a surgical instrument is inserted along a predicted insertion trajectory, one or more portions of the surgical instrument may become occluded when positioned behind or inside an object within the surgical space. Accordingly, to realistically display the representation of the insertion trajectory as extending virtually behind or inside such objects as the representation of the insertion trajectory of the surgical instrument is inserted along the predicted insertion trajectory of the surgical instrument, in certain examples, system 100 may provide for display by a display device only a portion or portions of the virtual representation of the insertion trajectory that are not occluded by one or more other objects within the surgical space. For example, the predicted insertion trajectory of the surgical instrument may extend behind another surgical instrument positioned within the surgical space. Additionally or alternatively, the predicted insertion trajectory of the surgical instrument (e.g., a distal portion thereof) may extend into tissue within the surgical space. In such examples, system 100 may provide for display by a display device only a portion(s) of the virtual representation of the insertion trajectory (e.g., a virtual representation of the surgical instrument) that are not occluded by other objects within the surgical space. In doing so, system 100 may be configured to realistically represent the virtual representation of the insertion trajectory as extending behind and / or inside one or more objects within the surgical space while inserted along the predicted insertion trajectory.

[0079] While the exemplary virtual representations of surgical instruments described herein are provided in the context of surgical instrument insertion, it is understood that the principles as described herein may be applied in other contexts to facilitate visualization of surgical instruments within a surgical space. For example, the virtual representation of a surgical instrument may, in certain implementations, show the location of some or all of a surgical instrument already inserted into the surgical space. In such an example, the virtual representation of a surgical instrument may depict a portion of the surgical instrument provided behind tissue and / or another surgical instrument within the surgical space.

[0080] Figure 10 illustrates an exemplary method for facilitating insertion of a surgical instrument into a surgical space. While Figure 10 illustrates exemplary operations according to one embodiment, other embodiments may omit, add to, reorder, and / or modify any of the operations shown in Figure 10. One or more of the operations shown in Figure 10 may be performed by a system such as system 100, any components included therein, and / or any implementation thereof.

[0081] In operation 1002, a system (e.g., insertion trajectory visualization system 100) may determine an insertion trajectory that a surgical instrument is predicted to follow as it is inserted into a surgical space. Operation 1002 may be performed in any of the ways described herein.

[0082] In operation 1004, the system may provide, for display by a display device, an image depicting a portion of the surgical space captured by the imaging device and at least a portion of a representation of a predicted insertion trajectory within the surgical space (e.g., a virtual representation of the surgical instrument inserted into the surgical space along an insertion trajectory that the surgical instrument is predicted to follow as it is inserted into the surgical space). As described herein, in certain examples, the representation of the insertion trajectory is provided for display when the insertion trajectory extends within the field of view of the imaging device. If the insertion trajectory does not extend within the current field of view of the imaging device, the field of view of the imaging device may be adjusted in any suitable manner until a representation of the insertion trajectory is visible within the image captured by the imaging device. Operation 1004 may be performed in any manner described herein.

[0083] In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided in accordance with the principles described herein. The instructions, when executed by a processor of a computing device, may instruct the processor and / or computing device to perform one or more operations, including one or more of the operations described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0084] Non-transitory computer-readable media as referred to herein may include any non-transitory storage medium that participates in providing data (e.g., instructions) that may be read and / or executed by a computing device (e.g., a processor of a computing device). For example, non-transitory computer-readable media may include, but are not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tape, etc.), ferroelectric random access memory ("RAM"), and optical disks (e.g., compact disks, digital video disks, Blu-ray disks, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0085] FIG. 11 illustrates an exemplary computing device 1100 that may be particularly configured to perform one or more processes described herein. As shown in FIG. 11, computing device 1100 may include a communication interface 1102, a processor 1104, a storage device 1106, and an input / output ("I / O") module 1108, communicatively coupled to each other via a communication infrastructure 1110. While an exemplary computing device 1100 is illustrated in FIG. 11, the components illustrated in FIG. 11 are not intended to be limiting. In other embodiments, additional or alternative components may be used. The components of computing device 1100 shown in FIG. 11 will now be described in additional detail.

[0086] The communication interface 1102 may be configured to communicate with one or more computing devices. Examples of the communication interface 1102 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0087] The processor 1104 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, processes, and / or operations described herein. The processor 1104 may perform operations by executing computer-executable instructions 1112 (e.g., applications, software, code, and / or other executable data instances) stored on the storage device 1106.

[0088] The storage device 1106 may include one or more data storage media, devices, or configurations and may utilize any type, form, and combination of data storage media and / or devices. For example, the storage device 1106 may include, but is not limited to, any combination of non-volatile and / or volatile media described herein. Electronic data, including the data described herein, may be stored temporarily and / or permanently in the storage device 1106. For example, data representing computer-executable instructions 1112 configured to instruct the processor 1104 to perform any of the operations described herein may be stored in the storage device 1106. In some examples, the data may be located in one or more databases residing within the storage device 1106.

[0089] I / O module 1108 may include one or more I / O modules configured to receive user input and provide user output. One or more I / O modules may be used to receive input for a single virtual experience. I / O module 1108 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, I / O module 1108 may include hardware and / or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0090] Input / output module 1108 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In particular embodiments, I / O module 1108 is configured to provide graphical data to a display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content as may be useful in a particular implementation.

[0091] In some examples, any of the systems, computing devices, and / or other components described herein may be implemented by a computing device 1100. For example, the storage facility 102 may be implemented by a storage device 1106, and the processing facility 104 may be implemented by a processor 1104.

[0092] To the extent the foregoing embodiments collect, store, and / or use personal information provided by individuals, it should be understood that such information may be used in accordance with all applicable laws and regulations regarding the protection of personal information. Additionally, the collection, storage, and use of such information may be conditioned on the individual's consent to such activities, for example, through an "opt-in" or "opt-out" process that may be appropriate for the context and type of information. The storage and use of personal information may be done in an appropriately secure manner reflecting the type of information, for example, through various encryption and anonymization techniques for particularly sensitive information.

[0093] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the following claims. For example, specific features of one embodiment described herein may be combined with or substituted for features of other embodiments described herein. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense.

Claims

1. a memory for storing instructions; a processor communicatively coupled to the memory; The processor executes the instructions to determining, based on the surgical instrument, an insertion trajectory that the surgical instrument is positioned to follow when inserted into the surgical space; providing an image for display by a display device depicting at least a portion of the portion of the surgical space captured by an imaging device and a representation of the insertion trajectory within the surgical space; providing guidance configured to instruct a user to use, together with the representation of the insertion trajectory, a virtual representation of the surgical instrument moving along the insertion trajectory as a guide for locating a cannula opening through which the surgical instrument will be inserted into the surgical space. It is configured as follows: system.

2. The system of claim 1 , wherein the representation of the insertion trajectory includes an animation depicting the representation of the insertion trajectory entering the surgical space within a field of view of the imaging device along the insertion trajectory.

3. The processor executes the instructions to determining a predicted insertion depth that the surgical instrument will achieve when inserted into the surgical space; providing, for display by the display device, the representation of the insertion trajectory inserted into the surgical space at the determined insertion depth along the insertion trajectory. further configured as follows:

3. The system according to claim 1 or 2.

4. The processor executes the instructions to determining a predicted insertion depth that the surgical instrument will achieve when inserted into the surgical space; providing, for display by the display device, the representation of the insertion trajectory inserted into the surgical space along the insertion trajectory beyond the determined insertion depth. further configured as follows:

3. The system according to claim 1 or 2.

5. The system of any one of claims 1 to 4, wherein the guidance includes an indicator that instructs the user to follow the insertion trajectory using the imaging device to view the opening through which the surgical instrument is inserted into the surgical space.

6. The processor executes the instructions to determining, based on the additional surgical instrument, an additional insertion trajectory that the additional surgical instrument is positioned to follow when inserted into the surgical space; providing, for display by the display device, at least a portion of an additional representation of the additional insertion trajectory within the surgical space along with the image depicting the portion of the surgical space captured by the imaging device. further configured as follows: A system according to any one of claims 1 to 5.

7. The system of claim 6 , wherein the representation of the insertion trajectory and the additional representation of the additional insertion trajectory are provided for simultaneous display by the display device along with the image depicting the portion of the surgical space.

8. The system of claim 6 , wherein the representation of the insertion trajectory is visually distinct from the additional representation of the additional insertion trajectory.

9. The processor executes the instructions to determining that the insertion trajectory that the surgical instrument is positioned to follow will result in the surgical instrument contacting an additional surgical instrument already inserted into the surgical space; providing, for display by the display device, together with the representation of the insertion trajectory, a notification indicating that the surgical instrument will be positioned to contact the additional surgical instrument if the surgical instrument is inserted along the insertion trajectory that the surgical instrument is positioned to follow. further configured as follows: A system according to any one of claims 1 to 8.

10. The system of any one of claims 1 to 9, wherein the insertion trajectory is determined based on kinematic data associated with a manipulator arm to which the surgical instrument is coupled.

11. the display device includes a stereoscopic image viewer configured to provide a three-dimensional (3D) image for display to the user; the representation of the insertion trajectory is provided for display within the 3D image when the insertion trajectory extends within a field of view of the imaging device. A system according to any one of claims 1 to 10.

12. determining, with an insertion trajectory visualization system, based on the surgical instrument, an insertion trajectory that the surgical instrument is positioned to follow when inserted into the surgical space; providing, by the insertion trajectory visualization system, for display by a display device, an image depicting at least a portion of the portion of the surgical space captured by an imaging device and a representation of the insertion trajectory within the surgical space; providing guidance configured to instruct a user to use, together with the representation of the insertion trajectory, a virtual representation of the surgical instrument moving along the insertion trajectory as a guide for locating a cannula opening through which the surgical instrument will be inserted into the surgical space. method.

13. determining, with the insertion trajectory visualization system, a predicted insertion depth that the surgical instrument will achieve when inserted into the surgical space; providing, by the insertion trajectory visualization system, the representation of the insertion trajectory inserted into the surgical space at the determined insertion depth along the insertion trajectory for display by the display device. The method of claim 12.

14. 14. The method of claim 12 or 13, wherein the guidance includes an indicator that instructs the user to follow the insertion trajectory using the imaging device to view the opening through which the surgical instrument is inserted into the surgical space.

15. determining, with the insertion trajectory visualization system, that the insertion trajectory that the surgical instrument is positioned to follow will result in the surgical instrument contacting an additional surgical instrument inserted into the surgical space; providing, by the insertion trajectory visualization system, for display by the display device, along with the representation of the insertion trajectory, a notification indicating that the surgical instrument will come into contact with the additional surgical instrument if the surgical instrument is inserted along the insertion trajectory that the surgical instrument is positioned to follow. The method according to any one of claims 12 to 14.

Images