System and method of tracking position of robotically-operated surgical instrument

Abstract

To solve problems in conventional technologies.SOLUTION: An exemplary surgical instrument tracking system comprises at least one physical computing device that determines on the basis of an endoscopic image of a surgical region, an observation about an object of interest depicted in the endoscopic image using a trained neural network, associates on the basis of a probabilistic framework and kinematics of a robotically-operated surgical instrument located in the surgical region, the observation about the object of interest with the robotically-operated surgical instrument, and determines a physical location of the robotically-operated surgical instrument in the surgical region on the basis of the kinematics of the robotically-operated surgical instrument and the observation associated with the robotically-operated surgical instrument.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] (Reference to Related Application) This application claims priority to U.S. Provisional Patent Application No. 62 / 712,368, entitled "Systems and Methods for Tacking a Position of a Robotically-Manipulated Surgical Instrument," filed July 31, 2018, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Robotic surgical systems allow surgeons to control robotically operated surgical instruments (robotically operated surgical instruments) to perform surgical procedures on patients. For example, for minimally invasive surgery, the robotically operated surgical instruments are inserted into the patient through one or more cannulas. The surgical instruments typically include an endoscope that captures images of the surgical field and one or more surgical tools that are robotically operated by the robotic surgical system to perform the surgical procedure. The surgeon views the endoscopic image of the surgical field and uses a master controller of the robotic surgical system to control the movement of the robotically operated surgical instruments to perform the surgical procedure.

[0003] Positional tracking of robotically operated surgical instruments within a surgical field is a critical component of a robotic surgical system. However, such tracking of surgical instrument position by a robotic surgical system is technically challenging, especially when a certain level of precision, accuracy, efficiency, and / or reliability is desired. Summary of the Invention

[0004] The accompanying drawings illustrate various embodiments and are a part of this specification. The illustrated embodiments are merely examples and do not limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. [Brief explanation of the drawings]

[0005] [Figure 1] 1 illustrates an exemplary robotic surgical system in accordance with principles described herein.

[0006] [Figure 2] 2 illustrates an exemplary patient-side system included within the robotic surgical system of FIG. 1 according to principles described herein.

[0007] [Figure 3] 3 illustrates an exemplary robotic arm included within the patient-side system of FIG. 2 according to principles described herein.

[0008] [Figure 4] 2 illustrates an exemplary surgeon console included within the robotic surgical system of FIG. 1 according to principles described herein.

[0009] [Figure 5] 1 illustrates an exemplary stereoscopic endoscope positioned in an exemplary surgical field associated with a patient according to principles described herein.

[0010] [Figure 6] 1 illustrates an exemplary surgical instrument tracking system in accordance with principles described herein.

[0011] [Figure 7] 1 illustrates a view of a surgical instrument within a surgical field according to principles described herein.

[0012] [Figure 8] 1 illustrates an exemplary endoscopic image of a surgical field in accordance with the principles described herein.

[0013] [Figure 9] 1 illustrates an exemplary probabilistic framework consistent with principles described herein. [Figure 10]1 illustrates an exemplary probabilistic framework consistent with principles described herein.

[0014] [Figure 11] 1 illustrates an exemplary implementation of a surgical instrument tracking system according to principles described herein. [Figure 12] 1 illustrates an exemplary implementation of a surgical instrument tracking system according to principles described herein. [Figure 13] 1 illustrates an exemplary implementation of a surgical instrument tracking system according to principles described herein. [Figure 14] 1 illustrates an exemplary implementation of a surgical instrument tracking system according to principles described herein.

[0015] [Figure 15] 1 illustrates an endoscopic image of a surgical field with a position indicator overlaid thereon in accordance with principles described herein.

[0016] [Figure 16] 1 illustrates an exemplary method for tracking a surgical instrument according to principles described herein. [Figure 17] 1 illustrates an exemplary method for tracking a surgical instrument according to principles described herein.

[0017] [Figure 18] 1 illustrates an exemplary computing system in accordance with principles described herein. DETAILED DESCRIPTION OF THE INVENTION

[0018] Systems and methods for tracking the position of a robotically operated surgical instrument are described herein. As described in more detail below, in certain embodiments, an exemplary surgical instrument tracking system may be configured to: determine observations about an object of interest depicted in an endoscopic imagery of a surgical field using a trained neural network; associate the observations about the object of interest with the robotically operated surgical instrument based on a probabilistic framework and kinematics of a robotically operated surgical instrument positioned in the surgical field; and determine a physical location of the robotically operated surgical instrument in the surgical field based on the kinematics of the robotically operated surgical instrument and the observations associated with the robotically operated surgical instrument.

[0019] Also, as described in further detail herein, in certain implementations, an exemplary surgical instrument tracking system may be configured to determine, based on endoscopic images of a surgical field, observations of an object of interest depicted in the endoscopic images using a trained neural network, and to associate the observations of the object of interest with a robotic surgical instrument or a false positive designation based on the kinematics of a robotic surgical instrument positioned in the surgical field. When the observations of the object of interest are associated with a robotic surgical instrument, the surgical instrument tracking system may determine a physical location of the robotic surgical instrument in the surgical field based on the kinematics of the robotic surgical instrument and the observations associated with the robotic surgical instrument. When the observations of the object of interest are associated with a false positive designation, the surgical instrument tracking system may refrain from using the observations to determine a physical location of the robotic surgical instrument in the surgical field.

[0020] As used herein, the position of an instrument may refer to the three-dimensional ("3D") location and / or orientation of a desired portion of the instrument, such as the end effector (e.g., jaws, shears, etc.), distal tool tip, wrist, shaft, and / or another structural component of the instrument. For example, the position of an instrument may refer to the 3D location and / or orientation of a component of the instrument within a 3D space, such as the 3D space defined by a 3D coordinate system.

[0021] By determining the physical location of a robotically operated surgical instrument as described herein, the exemplary surgical instrument tracking system and method may track surgical instrument position with precision, accuracy, efficiency, and / or reliability at a level that promotes one or more features, benefits, and / or advantages of a robotic surgical system. In certain implementations, such precision, accuracy, efficiency, and / or reliability may be provided by the exemplary surgical instrument tracking system without the use of dedicated computer-observed markers on the surgical instrument (e.g., barcodes, colors, patterns, etc. integrated onto or otherwise attached to the surgical instrument), which may be referred to as “marker-less” tracking of the surgical instrument. In certain implementations, such precision, accuracy, efficiency, and / or reliability may be provided by the exemplary surgical instrument tracking system in real time or near real time using finite computational resources.

[0022] Various embodiments will now be described in more detail with reference to the drawings. The systems and methods described herein may provide one or more of the benefits set forth above and / or various additional and / or alternative benefits set forth herein.

[0023] The surgical instrument tracking systems and methods described herein may operate as part of or in conjunction with a robotic surgical system. Accordingly, to facilitate understanding of the surgical instrument tracking systems and methods described herein, an exemplary robotic surgical system will now be described. The exemplary robotic surgical system described is illustrative and not limiting. The surgical instrument tracking systems and methods described herein may operate as part of or in conjunction with other suitable robotic surgical systems.

[0024] FIG. 1 illustrates an exemplary robotic surgical system 100. As shown, the robotic surgical system 100 may include a patient-side system 102 (sometimes referred to as a patient-side cart), a surgeon console 104, and a vision cart 106 communicatively coupled to each other. The robotic surgical system 100 may be utilized by a surgical team to perform a robotically-enabled surgical procedure on a patient 108. As shown, the surgical team may include a surgeon 110-1, an assistant 110-2, a nurse 110-3, and an anesthesiologist 110-4, all of whom may be collectively referred to as “surgical team members 10.” Additional or alternative surgical team members may be present during a surgical session, as may be useful in a particular implementation. While FIG. 1 illustrates a minimally invasive surgical procedure in progress, it will be understood that the robotic surgical system 100 may similarly be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the robotic surgical system 100. In addition, it will be understood that a surgical session in which the robotic surgical system 100 may be utilized may include not only the operative phase of a surgical procedure as illustrated in FIG. 1, but may also include pre-operative, post-operative, and / or other suitable phases of a surgical procedure.

[0025] As shown, patient-side system 102 may include multiple robotic arms 112 (e.g., robotic arms 112-1 through 112-4), and multiple robotically operated surgical instruments 114 (e.g., surgical instruments 114-1 through 114-4) may be coupled to the multiple robotic arms. Each surgical instrument 114 may include any suitable surgical tool, medical tool, monitoring instrument (e.g., endoscope), diagnostic instrument, or the like, that may be used for a robotically-enabled surgical procedure on patient 108 (e.g., by being at least partially inserted into patient 108 and manipulated to perform a robotically-enabled surgical procedure on patient 108). Note that while patient-side system 102 is depicted and described herein as a cart with multiple robotic arms 112 for illustrative purposes, in various other embodiments, patient-side system 102 can include one or more carts, each cart including one or more robotic arms 112, and the one or more robotic arms are attached to a separate structure within the operating room, such as an operating table or ceiling, and / or any other support structure(s). The patient-side system 102 is described in more detail below.

[0026] The surgical instruments 114 may each be positioned in a surgical area associated with a patient. As used herein, a "surgical area" associated with a patient may, in certain instances, be located entirely within the patient and may include an area within the patient near which a surgical procedure is planned to be performed, is being performed, or has been performed. For example, for a minimally invasive surgical procedure performed on tissue internal to the patient, the surgical area may include the tissue as well as the space surrounding the tissue, for example, where surgical instruments used to perform the surgical procedure are positioned. In other instances, the surgical area may be located at least partially external to the patient. For example, the robotic surgical system 100 may be used to perform an open surgical procedure, such that a portion of the surgical area (e.g., the tissue to be operated on) is internal to the patient, while another portion of the surgical area (e.g., the space surrounding the tissue where one or more surgical instruments may be positioned) is external to the patient. A surgical instrument (e.g., any of the surgical instruments 114) may be referred to as being positioned in (or "positioned within") the surgical area when at least a portion of the surgical instrument (e.g., the distal end of the surgical instrument) is positioned within the surgical area.

[0027] The surgeon console 104 may be configured to facilitate surgeon 110-1's control of the robotic arm 112 and surgical instruments 114. For example, the surgeon console 104 may provide the surgeon 110-1 with an image (e.g., a high-resolution 3D image) of a surgical field associated with the patient 108, as captured by an endoscope. The surgeon 110-1 may use the image to perform one or more procedures using the surgical instruments 114.

[0028] To facilitate control of the surgical instruments 114, the surgeon console 104 may include a set of master controllers 116 (shown in close-up 118). The master controllers 116 may be operated by the surgeon 110-1 to control the movement of the surgical instruments 114. The master controllers 116 may be configured to detect a wide variety of hand, wrist, and finger movements by the surgeon 110-1. In this manner, the surgeon 110-1 may intuitively perform procedures with one or more of the surgical instruments 114. For example, as shown in close-up 120, the functional tips of the surgical instruments 114-1 and 114-4 coupled to the robotic arms 112-1 and 112-4, respectively, may mimic the dexterity of the surgeon's 110-1 hands, wrists, and fingers across multiple degrees of freedom of movement to perform one or more surgical procedures (e.g., dissection, suturing, etc.). Although the surgeon console 104 is depicted and described herein as a single unit for illustrative purposes, in various other embodiments, the surgeon console 104 may include various separate components, such as a wired or wireless master controller 116, separate display element(s) (e.g., a projector or head-mounted display), separate data / communications processing hardware / software, and / or any other structural or functional elements of the surgeon console 104. The surgeon console 104 is described in more detail below.

[0029] The vision cart 106 may be configured to present visual content to surgical team members 110 who may not have access to the images provided to the surgeon 110-1 at the surgeon console 104. To this end, the vision cart 106 may include a display monitor 122 configured to display one or more user interfaces, such as images of the surgical field (e.g., 2D images), information associated with the patient 108 and / or the surgical procedure, and / or any other visual content as may be useful in a particular implementation. For example, the display monitor 122 may display images of the surgical field along with additional content (e.g., graphical content, contextual information, etc.) overlaid on top of or otherwise displayed concurrently with the images. In some embodiments, the display monitor 122 is implemented by a touchscreen display with which the surgical team members 110 may interact (e.g., via touch gestures) to provide user input to the robotic surgical system 100.

[0030] The patient-side system 102, the surgeon console 104, and the vision cart 106 may be communicatively coupled to one another in any suitable manner. For example, as shown in FIG. 1 , the patient-side system 102, the surgeon console 104, and the vision cart 106 may be communicatively coupled via control lines 124, which may represent any wired or wireless communication links that may be useful in a particular implementation. To this end, the patient-side system 102, the surgeon console 104, and the vision cart 106 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0031] The patient-side system 102, the surgeon console 104, and the vision cart 106 may each include at least one computing device configured to control, direct, and / or facilitate the operation of the robotic surgical system 100. For example, the surgeon console 104 may include a computing device configured to send instructions to the patient-side system 102 over one or more of the control lines 124 to control the movement of the robotic arm 112 and / or the surgical instrument 114 pursuant to manipulation by the surgeon 110-1 of the master controller 116. In some examples, the vision cart 106 may include one or more computing devices configured to perform primary processing operations of the robotic surgical system 100. In such a configuration, one or more computing devices included in the vision cart 106 may control and / or coordinate operations performed by various other components of the robotic surgical system 100 (e.g., the patient-side system 102 and / or the surgeon console 104). For example, a computing device included in the surgeon console 104 may send instructions to the patient-side system 102 via one or more computing devices included in the vision cart 106.

[0032] 2 illustrates a perspective view of the patient-side system 102. As shown, the patient-side system 102 includes a cart column 202 supported by a base 204. In some examples, the cart column 202 may include a protective cover 206 that protects the components of the braking and counterbalance subsystems located within the cart column 202 from contaminants.

[0033] The cart column 202 may support multiple setup arms 208 (e.g., setup arms 208-1 through 208-4) mounted thereon. Each setup arm 208 may include multiple links and joints that allow for manual positioning of the setup arm 208, and each may be connected to one of the robotic arms 112. In the example of FIG. 2, the patient-side system 102 includes four setup arms 208 and four robotic arms 112. However, it will be appreciated that the patient-side system 102 may include other numbers of setup arms 208 and robotic arms 112 as may be convenient for a particular implementation.

[0034] The setup arm 208 may be non-robotically controllable and configured to hold each robotic arm 112 stationary in a respective position desired by a person setting up or reconfiguring the patient-side system 102. The setup arm 208 may be coupled to the carriage housing 210 and may be manually moved and positioned during pre-operative, intra-operative, or post-operative phases of a surgical session. For example, the setup arm 208 may be moved and positioned during a pre-operative phase when the robotic surgical system 100 is being prepared and / or targeted for the surgical procedure to be performed. In contrast, the robotic arms 112 may be robotically controlled (e.g., in response to manipulation of the master controller 116, as described above).

[0035] As shown, each robotic arm 112 may have a surgical instrument 114 coupled thereto. In a particular example, three of the four robotic arms 112 may be configured to move and / or position surgical instruments 114 in the form of surgical tools used to manipulate patient tissue and / or other objects (e.g., suture material, patching material, etc.) within a surgical field. Specifically, as shown, robotic arms 112-1, 112-3, and 112-4 may be used to move and / or position surgical instruments 114-1, 114-3, and 114-4, respectively. A fourth robotic arm 112 (e.g., robotic arm 112-2 in the example of FIG. 2) may be used to move and / or position a monitoring instrument (e.g., a stereoscopic endoscope), as described in more detail below.

[0036] The robotic arms 112 may each include one or more displacement transducers, orientation sensors, and / or position sensors (e.g., sensors 212) used to generate raw (i.e., uncorrected) motion information to assist in the control and tracking of the surgical instruments 114. For example, kinematic information generated by the transducers and sensors in the patient side system 102, such as kinematics-based position or other kinematic information for the robotic arms 112, may be transmitted to a surgical instrument tracking system (e.g., a computing device included in the vision cart 106) of the robotic surgical system 100. Each surgical instrument 114 may similarly include displacement transducers, position sensors, and / or orientation sensors (e.g., sensors 214) in certain embodiments, each of which may provide additional kinematic information to the tracking system, such as kinematics-based position or other kinematic information for the surgical instruments 114. The tracking system may process the kinematic information received from sensors included in the robotic arms 112 and / or surgical instruments 114 to perform various operations, such as determining the physical position of the robotic arms 112 and / or surgical instruments 114, as described herein.

[0037] FIG. 3 illustrates a perspective view of an exemplary robotic arm 112 (e.g., any one of robotic arms 112-1 through 112-4). As shown, a surgical instrument 114 may be removably coupled to the robotic arm 112. In the example of FIG. 3, the surgical instrument 114 is an endoscopic device (e.g., a stereoscopic laparoscope, an arthroscope, a hysteroscope, or other type of stereoscopic or planar endoscope). Alternatively, the surgical instrument 114 may be a different type of imaging device (e.g., an ultrasound device, a fluoroscopy device, an MRI device, etc.), a grasping instrument (e.g., forceps), a needle driver (e.g., a device used for suturing), an energy instrument (e.g., a cauterizing instrument, a laser instrument, etc.), a retractor, a clip applier, a probe grasper, a heart stabilizer, or any other suitable instrument or tool.

[0038] In some instances, it may be desirable for the robotic arm 112 and the surgical instrument 114 coupled to the robotic arm 112 to move about a single, fixed center point 302 so as to limit movement of the center point 302. For example, the center point 302 may be located at or near the insertion point of the surgical instrument 114 into the patient 108. In certain surgical sessions (e.g., those associated with laparoscopic surgical procedures), for example, the center point 302 may be aligned with the incision point to an internal surgical site by a trocar or cannula in the abdominal wall. As shown, the center point 302 may be located on an insertion axis 304 associated with the surgical instrument 114.

[0039] The robot arm 112 may include multiple links 306 (e.g., links 306-1 through 306-5) pivotally connected in series at multiple joints 308 (e.g., joints 308-1 through 308-4) near each end of the links 306. For example, as shown, link 306-1 is pivotally connected to drive mount 310 at joint 308-1 near a first end of link 306-1 while being pivotally connected to link 306-2 at joint 308-2 near a second end of link 306-1. Link 306-3 is pivotally connected to link 306-2 near a first end of link 306-3 while being pivotally connected to link 306-4 at joint 308-4 near a second end of link 306-3. Generally, link 306-4 may be substantially parallel to the insertion axis 304 of the surgical instrument 114, as shown. Link 306-5 is slidably coupled to link 306-4 to allow the surgical instrument 114 to be attached to and slide along link 306-5, as shown.

[0040] The robotic arm 112 may be configured to be attached to the setup arm 208 (or a joint connected to the setup arm 208) via a drive mount 310 so as to be commanded and held in place by the setup arm 208, as described above. The drive mount 310 may be pivotally coupled to link 306-1 and may include a first internal motor (not explicitly shown) configured to yaw the robotic arm 112 about a yaw axis about the center point 302. Similarly, link 306-2 may house a second internal motor (not explicitly shown) configured to drive and pitch the linkage of the robotic arm 112 about a pitch axis about the center point 302. Similarly, link 306-4 may include a third internal motor (not explicitly shown) configured to slide link 306-5 and the surgical instrument 114 along the insertion axis 304. The robotic arm 112 may include a drivetrain system driven by one or more of these motors to control the pivoting of the links 306 about the joints 308 in any manner that may be useful for a particular implementation. Thus, if the surgical instrument 114 is to be mechanically moved, one or more of the motors coupled to the drivetrain may be energized to move the links 306 of the robotic arm 112.

[0041] Figure 4 illustrates a perspective view of the surgeon's console 104. As shown, the surgeon's console 104 may include a stereo viewer 402, an arm support 404, a controller workspace 406 in which the master control unit 116 (not shown in Figure 4) is located, foot pedals 408, and a head sensor 410.

[0042] In some examples, the stereo viewer 402 has two displays on which an operator (e.g., the surgeon 110-1) may view stereoscopic images of a surgical area associated with the patient 108 during a surgical session and generated by a stereoscopic endoscope. When using the surgeon console 104, the operator may move their head into alignment with the stereo viewer 402 to view the stereoscopic images of the surgical area. To ensure that the operator is viewing the surgical area when controlling the surgical instruments 114 of the patient-side system 102, the surgeon console 104 may use a head sensor 410 positioned adjacent to the stereo viewer 402. Specifically, when the operator aligns their eyes with the binocular eyepieces of the stereo viewer 402 to view the stereoscopic images of the surgical area, the operator's head may activate the head sensor 410, which enables control of the surgical instruments 114 via the master controller 116. When the operator's head is removed from the field of the stereoscopic viewer 402, the head sensor 410 may be automatically deactivated, which may prevent control of the surgical instrument 114 via the master controller 116. In this manner, the position of the surgical instrument 114 may remain stationary when the robotic surgical system 100 detects that the operator is not actively engaged in attempting to control the surgical instrument 114.

[0043] The arm support 404 may be used to support the operator's elbow and / or forearm while the operator manipulates the master controller 116 to control the robotic arm 112 and / or surgical instrument 114. Additionally, the operator may use their foot to control a foot pedal 408. The foot pedal 408 may be configured to generate additional control signals used to control the surgical instrument 114, to facilitate control of switching from one surgical instrument 114 to another, or to change the configuration or operational mode of the robotic surgical system 100 to perform any other suitable operation.

[0044] 5 illustrates an exemplary stereoscopic endoscope 500 included in the robotic surgical system 100 and positioned in an exemplary surgical field associated with a patient. The stereoscopic endoscope 500 may be any one of the surgical instruments 114 described above.

[0045] As shown, the stereoscopic endoscope 500 may include a tube 502 having a distal tip configured to be inserted into a patient and a camera head 504 configured to be positioned outside the patient. The tube 502 may be coupled to the camera head 504 at a proximal end and may be rigid (as shown in FIG. 5), coupled, and / or flexible as may be useful for a particular implementation.

[0046] Tube 502 may include multiple channels 506 (e.g., right imaging channel 506-R, left imaging channel 506-L, and illumination channel 506-I) configured to conduct light between a surgical area inside the patient and camera head 504. Each channel 506 may include one or more optical fibers configured to carry light along tube 502 such that light generated within camera head 504 is carried by illumination channel 506-I for output at the distal end of tube 502, reflected from the patient's anatomy and / or other objects within the surgical area, and then carried by imaging channels 506-R and 506-L from the distal end of tube 502 back to camera head 504. The arrows shown within channels 506 in FIG. 5 are depicted to indicate the direction that light may travel within each channel. Additionally, it will be understood that tube 502 may be associated with (e.g., may include) one or more lenses or other suitable optics (not explicitly shown) for focusing, diffusing, or otherwise processing the light carried by channel 506 as may be useful for a particular implementation. In various other embodiments, additional imaging and / or illumination channels may be present. In still other embodiments, one or more image sensors and / or illumination devices may be located closer to the distal end of tube 502, thereby minimizing or even eliminating the need for imaging and / or illumination channels through tube 502.

[0047] In some examples, the stereoscopic endoscope 500 may be coupled to a robotic arm of a robotic surgical system (e.g., one of the robotic arms 112 of the robotic surgical system 100) and positioned such that the distal tip of the tube 502 is located in a surgical field on a patient. In this configuration, the stereoscopic endoscope 500 may be referred to as being located in or within the surgical field, even though portions of the stereoscopic endoscope 500 (e.g., the camera head 504 and the proximal portion of the tube 502) may be located outside the surgical field. While the stereoscopic endoscope 500 is positioned in the surgical field, light reflected from the surgical field within the field of view of the stereoscopic endoscope 500 may be captured by the distal tip of the tube 502 and conveyed to the camera head 504 via the imaging channels 506-R and 506-L.

[0048] The camera head 504 may include various components configured to facilitate operation of the stereoscopic endoscope 500. For example, as shown, the camera head 504 may include an image sensor 508 (e.g., an image sensor 508-R associated with the right imaging channel 506-R and an image sensor 508-L associated with the left imaging channel 506-L). The image sensor 508 may be implemented as any suitable image sensor, such as a charge-coupled device ("CCD") image sensor, a complementary metal-oxide semiconductor ("CMOS") image sensor, or the like. Additionally, one or more lenses or other optical systems may be associated with the image sensor 508 (not explicitly shown). The camera head 504 may further include an illumination device 510 (illuminator) configured to generate light that travels from the camera head 504 to the surgical area via the imaging channel 506-I to illuminate the surgical area.

[0049] The camera head 504 may further include a camera control unit 512 disposed therein. Specifically, the camera control unit 512-R may be communicatively coupled to the image sensor 508-R, and the camera control unit 512-L may be communicatively coupled to the image sensor 508-L. The camera control units 512 may be synchronously coupled to each other via a communication link 514 and may be implemented by software and / or hardware configured to control the image sensor 508 to generate respective images 516 (i.e., an image 516-R associated with the right side and an image 516-L associated with the left side) based on light sensed by the image sensor 508. Thus, each respective combination of the imaging channel 506, the imaging sensor 508, the camera control unit 512, and the associated optics may be collectively referred to as a camera included within the stereoscopic endoscope 500. For example, the stereoscopic endoscope 500 may include two such cameras, one for the left side and one for the right side. Such cameras may be said to capture images 516 from a vantage point at the distal end of their respective imaging channels 506. After being generated by the stereoscopic endoscope 500, the images 516 may be accessed by a surgical instrument tracking system and / or used in any of the methods described herein. For example, the images 516 may be used by a surgical instrument tracking system to determine the position of a robotically operated surgical instrument positioned in a surgical field in any of the methods described herein.

[0050] FIG. 6 illustrates an exemplary surgical instrument tracking system 600 (“tracking system 600” or “system 600”) configured to track surgical instruments located in a surgical field. As illustrated, system 600 includes, but is not limited to, an image processing facility 602, an association facility 604, a position determination facility 606, and a storage facility 608 selectively and communicatively coupled to one another. While facilities 602-608 are shown in FIG. 6 as separate facilities, it will be appreciated that facilities 602-608 may be combined into fewer facilities, e.g., into a single facility, or divided into more facilities, as may be useful in a particular implementation. Each of facilities 602-608 may be implemented by any suitable combination of computing hardware and / or software.

[0051] System 600 may be associated with a robotic surgical system, such as robotic surgical system 100, in any suitable manner. For example, system 600 may be implemented by or included in the robotic surgical system. To illustrate, system 600 may be implemented by one or more computing devices included in the patient side system 102, surgeon console 104, and / or vision cart 106 of robotic surgical system 100. In some examples, system 600 may be implemented, at least in part, by one or more computing devices communicatively coupled to, but not included in, the robotic surgical system (e.g., one or more servers communicatively coupled to robotic surgical system 100 via a network).

[0052] The image processing facility 602 may be configured to access endoscopic images (sometimes referred to as "endoscopic images"), which may include a set of one or more images captured by an endoscope positioned in a surgical field associated with a patient or derived from images captured by such an endoscope. The set of one or more endoscopic images may include a 2D image, a 3D image, a 3D image derived from 2D images, one or more pairs of temporally aligned stereoscopic images, frames of video, a sequence of images, an aggregated or combined image (e.g., an image formed by joining multiple images together), or any other suitable form of image. The endoscopic images may be represented as image data in any suitable data format (e.g., data representing an RGB image having any suitable number of image channels), which may be accessed by the image processing facility 602.

[0053] In a particular implementation, for example, the image processing facility 602 may access a first image (e.g., image 516-L) captured from a first vantage point by a first camera included in a stereoscopic endoscope (e.g., stereoscopic endoscope 500) positioned in the surgical field, and a second image (e.g., image 516-R) captured from a second vantage point that is stereoscopically visible relative to the first vantage point by a second camera included in the stereoscopic endoscope. The image processing facility 602 may access the images in any suitable manner. For example, the image processing facility 602 may access the images by receiving the images from the endoscope or by accessing the images from the storage facility 108 or other computer memory (e.g., a memory buffer).

[0054] Although particular examples are described herein with reference to endoscopic images captured by an endoscope, in other examples, the image processing facility 602 may process any suitable images of the surgical field. For example, the image processing facility 602 may process images of the surgical field captured by one or more non-endoscopic cameras, multiple endoscopes, or a combination of at least one endoscope and at least one non-endoscopic camera.

[0055] The image processing facility 602 may process the endoscopic images to determine observations about objects of interest depicted in the endoscopic images. Processing of the endoscopic images may include the image processing facility 602 extracting features representing the objects of interest from the endoscopic images. As used herein, an "object of interest" refers to any object or other feature depicted in the endoscopic image and defined as being of interest to the image processing facility 602. In certain examples, the object of interest may be represented by any feature depicted in the endoscopic image that may be associated with a surgical instrument. For example, the feature may be a visual representation of a surgical instrument depicted in the endoscopic image, a component of a surgical instrument depicted in the endoscopic image, or any object depicted in the endoscopic image that may be accurately or incorrectly determined by the image processing facility 602 to represent a surgical instrument or a component of a surgical instrument.

[0056] In certain embodiments, the object of interest may be a marker-less component of a surgical instrument. The marker-less component of a surgical instrument may be a physical component of the surgical instrument that does not include a dedicated computer vision marker (e.g., a barcode, color, etc.). By processing the endoscopic image to extract the marker-less object of interest instead of the dedicated computer vision marker, the image processing facility 602 may overcome one or more problems associated with the use of dedicated computer vision markers, such as the cost of manufacturing the markers, the limited lifespan of the markers, the inability to use markers on all surgical instruments, occlusion of the marker from the camera view by anatomical tissue, blood, debris, or other instruments, and calculating inefficiencies associated with traditional detection of markers.

[0057] In certain implementations, the object of interest may be a distal clevis of a surgical instrument. The distal clevis of a surgical instrument may be the center of the distal-most joint of the surgical instrument, and the location of the distal clevis may be used to determine the location of the distal tip of the surgical instrument. In some implementations, extracting the distal clevis of a surgical instrument from an endoscopic image may be more accurate, easier, and / or reliable than extracting a small distal tip of a surgical instrument from an endoscopic image. In other implementations, additional or alternative components of a surgical instrument (e.g., the shaft of the surgical instrument) may be defined to be the object of interest.

[0058] Once the image processing facility 602 extracts the object of interest from the endoscopic image, the image processing facility 602 may define an observation about the object of interest. The observation may indicate a position of the object of interest derived from the endoscopic image. Because the position is derived from the endoscopic image, the position may be referred to as an "image-based position." In certain examples, the observation may be defined to represent the position as a 2D region in 2D space or a 3D region in 3D space. For example, the observation may indicate a defined pixel area region on a projected 3D image plane (e.g., projected from the vantage point of the endoscopic camera). As another example, the observation may indicate a defined 3D region, such as a 3D region in 3D space defined by a world 3D coordinate system associated with the robotic surgical system 100.

[0059] In certain examples, in addition to representing the position of the object of interest, the observations defined by the image processing facility 602 (e.g., the output of a neural network) may indicate one or more other attributes of the object of interest. For example, the observations may indicate motion cues (e.g., velocity, acceleration, etc.) associated with the object of interest (which may be derived from a sequence of images, such as a sequence of video frames), the type of surgical instrument associated with the object of interest (e.g., the object of interest is attached to a needle driver, grasper, or cautery-type surgical tool), an additional position associated with the object of interest (e.g., the position of another component of the object of interest), the position of another object of interest relative to the position of the object of interest (e.g., the position of a surgical instrument tip, shaft, etc., relative to a distal clevis of the surgical instrument), and / or an orientation associated with the object of interest (e.g., the orientation of the distal clevis).

[0060] The image processing facility 602 may be configured to use a trained neural network to determine observations about the objects of interest. For example, the image processing facility 602 may use a convolutional neural network ("CNN") or another deep, feed-forward artificial neural network to extract the objects of interest and define observations about the objects of interest.

[0061] To this end, a neural network may be trained such that the neural network is configured to determine observations about objects of interest depicted in endoscopic images. Training may be performed in any suitable manner, for example, by using labeled (e.g., manually labeled) data as a training set. Such data may include endoscopic images labeled with observations of any objects of interest depicted in the endoscopic images. For example, training imagery may include images depicting the objects of interest and labels indicating image-based locations associated with the objects of interest. The labels may be in any suitable form, such as a bounding box defining a rectangular region (e.g., a pixel region) positionally associated with the object of interest, or any other suitable definition of a region of the image positionally associated with the object of interest. Other images in the training imagery may be similarly labeled to indicate locations associated with the object of interest. Still other images in the training imagery may be labeled (or may not be labeled) to indicate that the object of interest is not depicted in the image. In some examples, additional labels may be added to the training images to indicate other characteristics of the object of interest, such as labels indicating the type of surgical instrument and / or additional components of the surgical instrument. Examples of other labels that may be used include, but are not limited to, labels indicating key points on the surgical instrument (e.g., the instrument tip, features on the instrument surface such as a logo or other indicia on the instrument surface, the instrument shaft, the instrument shaft axis, etc.) and labels indicating per-pixel segmentations of the instrument or components of the instrument (e.g., the instrument body or portions of the instrument body). Particular images in the training images may be selected for use in training the neural network based on discrepancies between the image content and the corresponding kinematics for the surgical instrument, such as an image that does not depict a surgical instrument when the corresponding kinematics indicate that the surgical instrument should be within the field of view of the endoscope capturing the image.

[0062] After the neural network is trained, the image processing facility 602 may use the trained neural network to determine observations about objects of interest depicted in the endoscopic images. In a particular example, the image processing facility 602 may do this by providing the endoscopic images as input to the trained neural network, which may process the endoscopic images and output determined observations about the objects of interest depicted in the images and extracted by the trained neural network. The trained neural network may determine the observations about the objects of interest by selecting, from historical and / or previously generated observations for training endoscopic images (e.g., labeled during training of the neural network), the observations that best match or are most likely to match the endoscopic image being processed. In other examples, instead of outputting observations about the objects of interest, the trained neural network may output data indicative of the extracted objects of interest, and the image processing facility 602 may use this output from the trained neural network to define or supplement the observations about the objects of interest, for example, by defining a region encompassing the object of interest and / or adding additional characteristics to the observations. Described herein are examples of image processing facilities 602 that determine observations about objects of interest depicted in endoscopic images.

[0063] The observations determined by the image processing facility 602 for the object of interest may be considered to be indicative of at least a candidate image-based location for a surgical instrument because the image processing facility 602 does not associate the observations determined and output by the image processing facility 602 with any particular surgical instrument in the surgical field. When the association facility 604 determines that the object of interest does not represent a surgical instrument in the surgical field, the image processing facility 602 may provide data representing the observations for access and use by the association facility 604 to associate the observations with an appropriate surgical instrument as determined by the association facility 604 as represented by the object of interest or with a false positive designation.

[0064] The association facility 604 may access data representing observations about the object of interest from the image processing facility 602 and associate the observations about the object of interest with a particular surgical instrument or a false positive designation based on the kinematics of one or more surgical instruments in the surgical field. In a particular implementation, the association facility 604 may be configured to determine the associations to be made based on a probabilistic framework and the kinematics of one or more surgical instruments in the surgical field. As an example, the association facility 604 may determine the probability of potential associations of the observation about the object of interest to each surgical instrument in the surgical field and to a false positive designation based on the kinematics of one or more surgical instruments in the surgical field. The association facility 604 may then select the association with the highest probability and associate the observation with the surgical instrument or the false positive designation having the highest probability. As another example, the association facility 604 may determine the probability for each possible set of one-to-one associations between the observation and the surgical instrument or the false positive designation based on the kinematics of one or more surgical instruments in the surgical field. The association facility 604 may then select the most probable set of one-to-one associations and associate the observations with the respective surgical instruments or false positive designations according to the selected set of associations. Examples of the association facility 604 that use a probabilistic framework and kinematics to associate observations with surgical instruments or false positive designations are described herein.

[0065] The association facility 604 may output data representing associations made by the association facility 604, such as data representing observations and associated surgical instruments. In some instances, the association facility 604 may refrain from outputting data representing observations that are associated with false positive designations.

[0066] The position determination facility 606 may access data output by the association facility 604, such as data representing observations and associated surgical instruments. The position determination facility 606 may use the accessed data to determine the physical location of the surgical instrument in the surgical field based on the kinematics of the surgical instrument and the observations associated with the surgical instrument.

[0067] In particular examples, the position determination facility 606 may implement and use a nonlinear estimator, such as an unscented Kalman filter or other Bayesian filter, to determine the physical position of the surgical instrument in the surgical field based on the kinematics of the surgical instrument and observations related to the surgical instrument. For example, the position determination facility 606 may input the observations and kinematics into an unscented Kalman filter, which may use the observations and kinematics to determine and output the physical position of the surgical instrument in the surgical field. The physical position of the surgical instrument in the surgical field may be represented in any suitable manner, including coordinates representing the 3D positions of components of the surgical instrument (e.g., the 3D position of a distal clevis or distal tip of the surgical instrument) in a 3D space, such as a world 3D space associated with the robotic surgical system 100.

[0068] In certain embodiments, the position determination facility 606 may be configured to determine the physical position of the surgical instrument by correcting the kinematics-based position of the surgical instrument, represented by the kinematics, based on the image-based position of the object of interest represented by the observation. For example, an unscented Kalman filter may be configured to adjust the kinematics-based position of the surgical instrument based on the image-based position of the object of interest represented by the observation. The adjustment may be made in any suitable manner, such as by applying a correction transform to the kinematics-based position. In certain implementations, the position determination facility 606 may be configured to apply the correction transform to the kinematics-based position as quickly as possible, which may allow the image processing facility 602 to extract the object of interest from the endoscopic image and determine an observation about the associated object of interest as quickly as possible. By correcting the kinematics-based position with the observed image-based position, the tracking system 600 can use the kinematics when observations are unavailable, such as when the surgical instrument is not extractable from the endoscopic image (e.g., when the surgical instrument is hidden behind tissue, when there is a lot of smoke in the surgical field, when the surgical instrument is outside the field of view of the endoscope, etc.). When observations of an object of interest representing a surgical instrument are available and used to correct the kinematics-based position of the surgical instrument, tracking system 600 can track the physical position of the surgical instrument with better precision and / or accuracy than when kinematics are used alone without the observations. This can be useful in eliminating or reducing kinematics-based position errors (e.g., instrument tip position errors caused by the accumulation of joint encoder errors along the kinematic chain of a robotic system), such as stiffness errors and / or errors introduced by large movements of a surgical instrument (which may be mounted on a long kinematic chain) that may be found when determining the absolute position in global 3D world coordinates based solely on kinematics.

[0069] The physical location of the surgical instrument determined by the positioning facility 606 may be used to track the surgical instrument in the surgical field during a surgical session. By tracking the surgical instrument in this manner, the robotic surgical system 100 may provide one or more configurations that utilize the tracked physical location of the surgical instrument. Examples of such configurations are described herein.

[0070] Storage facility 608 may store and maintain any data received, generated, managed, maintained, used, and / or transmitted by facilities 602-606 in a particular implementation. For example, storage facility 608 may store program instructions (e.g., computer code) for performing operations described herein, image data, observation data, kinematic data, position data, and / or any other data useful to a particular implementation.

[0071] 7-10, an example of a system 600 for tracking surgical instruments by determining the physical location of the surgical instruments will be described. FIG. 7 illustrates a diagram 700 of surgical instruments within a surgical field. As shown, the surgical instruments may include an endoscope 702 and one or more other surgical instruments 704 in the form of one or more surgical tools (e.g., surgical instruments 704-1 through 704-3). While FIG. 7 illustrates one endoscope 702 and three surgical tools positioned within the surgical field, any number and / or combination of endoscopes and surgical tools may be present within the surgical field during a surgical session. Tissue 706 represents anatomical structures within the surgical field.

[0072] The endoscope 702 may capture an endoscopic image of the surgical field. Any surgical instruments 704 and / or tissue 706 within the field of view of the endoscope 702 may be depicted in the endoscopic image captured by the endoscope 702.

[0073] 8 illustrates exemplary images 802 (i.e., images 802-L and 802-R) of a surgical field as captured from a stereoscopic vantage point by a camera included within endoscope 702, which in this example is a stereoscopic endoscope. Images 802-L and 802-R may implement images 516-L and 516-R, respectively, described above. As shown, each image 802 depicts a portion of surgical instrument 704-1 and tissue 706 within the field of view of endoscope 702.

[0074] As shown, images 802 are relatively similar to one another. However, it will be understood that slight differences exist between images 802 due to the stereoscopic nature of the vantage point from which each image 802 is captured. Thus, images 802 may appear three-dimensional when image 802-L is presented to the left eye of a user (e.g., surgeon 110-1), while image 802-R is presented to the right eye of the user.

[0075] Similar to image 516, one or both of images 802 may be displayed or presented by system 600 in any suitable manner and / or on any suitable display screen. For example, one or both of images 802 may be displayed on display monitor 122 of vision cart 106, on stereoscopic viewer 402 on surgeon console 104, on a monoscopic display provided by endoscope 702, and / or on other display screens associated with robotic surgical system 100 or tracking system 600.

[0076] The image processing facility 602 may process one or both of the images 802 to determine an observation for any object of interest depicted in the images 802. For example, the image processing facility 602 may provide the images 802 as input to a trained neural network, which may extract from the images 802 any object of interest depicted in the images 802. For example, using the trained neural network, the image processing facility 602 may extract an object of interest 804 representing a distal clevis of the surgical instrument 704-1 in each of the images 802. The image processing facility 602 may then define an observation for the object of interest 804 based on the images 802 and the extracted object of interest 804. For example, the image processing facility 602 may define an observation that includes a bounding box 806 that represents the image-based location of the distal clevis in each image 802. While FIG. 8 illustrates a rectangular bounding box 806 to represent the image-based location indicated by the observation of the object of interest 804, in other examples, any suitable geometric region (e.g., an ellipse), coordinate point(s), or other location indicator may be used to represent the image-based location indicated by the observation of the object of interest. For example, a bounding box or other location indicator may be defined to identify any instrument portion (e.g., an instrument tip, shaft, indicia, etc.). In certain other examples, the observation may define lines forming an instrument “skeleton,” or the observation may define pixel-by-pixel segments or portions of the instrument (e.g., an image in which pixels representing instrument portions and pixels not representing instrument portions are represented as different binary values, such as black and white values). Also, while the distal clevis is represented by the object of interest 804 in each of the images 802, any suitable object of interest, such as additional or alternative components of a surgical instrument, may be defined to be an object of interest that can be extracted by a trained neural network.

[0077] In particular examples, the image processing facility 602 may use a trained neural network to extract an object of interest based on a configuration indicating that the object of interest may represent a surgical instrument, even if the object of interest does not actually represent a surgical instrument. For example, using the trained neural network, the image processing facility 602 may extract an object of interest 808 that is actually a portion of tissue 706 having a configuration indicating that the object of interest 808 may represent a surgical instrument. The image processing facility 602 may then define an observation for the object of interest 808 based on the images 802 and the extracted object of interest 808. As shown in FIG. 8 , for example, the image processing facility 602 may define an observation that includes a bounding box 810 that represents an image-based location of the object of interest 808 in each of the images 802.

[0078] The image processing facility 602 may similarly extract any other objects of interest depicted in the image 802 (e.g., objects of interest representing other surgical instruments if they are within the field of view of the endoscope 702) and define observations for those objects of interest.

[0079] The view determined by the image processing facility 602 of the object of interest may represent an image-based location of the object of interest in any suitable manner. For example, the view may represent a 2D region within the image (e.g., a pixel region or other 2D region on a projection plane projected from a viewpoint such as the viewpoint of the endoscope 702). As another example, the view may represent a 3D region within a 3D space represented by a world 3D coordinate system, which may be derived from an identified 2D region within the endoscopic image (e.g., from a 2D region within a stereoscopic endoscopic image). As another example, the view may be a single coordinate location or a set of one or more coordinate locations within a 2D region or 3D space. Examples of such views are described in more detail herein.

[0080] In certain examples, the image processing facility 602 may not have information about the surgical instrument 704 or the kinematics of the surgical instrument 704 and may determine an observation solely based on the endoscopic image using a trained neural network (e.g., by inputting the endoscopic image into a trained neural network that outputs the observation). Thus, the image processing facility 602 may not determine whether the observation represents a surgical instrument or a false positive designation, or which particular surgical instrument is represented by the observation.

[0081] The association facility 604 may associate each determined observation with a particular one of the surgical instruments 704 or a false positive designation based on the kinematics of one or more of the surgical instruments in the surgical field (e.g., the kinematics of the endoscope 702 and / or one or more of the surgical instruments 704) or a probabilistic framework. In particular implementations, this may involve the association facility 604 determining a probability of each potential association for the observation determined by the image processing facility 602 and selecting the potential association with the highest probability.

[0082] As an example, the association facility 604 may determine, based on one or more suitable probabilistic algorithms that consider the kinematics (e.g., 3D kinematic-based positions indicated by the kinematics) of one or more surgical instruments in the surgical field, the probability that the bounding box 806 represents surgical instrument 704-1, the probability that the bounding box 806 represents surgical instrument 704-2, the probability that the bounding box 806 represents surgical instrument 704-3, and the probability that the bounding box 806 represents a false positive. The association facility 604 may then select the highest determined probability and associate the bounding box 806 with the surgical instrument 704 having the highest probability or false positive designation. For the scenario illustrated in FIG. 8, the association facility 604 may determine that the association of bounding box 806 with surgical instrument 704-1 has the highest probability based at least in part on the close distance relationship between the image-based position of the observation represented by bounding box 806 and the kinematics-based position of surgical instrument 704-1 (as indicated by kinematics) (e.g., compared to the more distant relationship between the image-based position of the observation represented by bounding box 806 and the kinematics-based positions of surgical instruments 704-2 and 704-3).

[0083] 9 illustrates an exemplary probabilistic framework 900 that may be used by the association facility 604 to associate bounding box 806 with surgical instrument 704-1 in the example described above. In the illustrated probabilistic framework 900, the association facility 604 determines a probability P1 for a potential association 902-1 of bounding box 806 with surgical instrument 704-1, a probability P2 for a potential association 902-2 of bounding box 806 with surgical instrument 704-2, a probability P3 for a potential association 902-3 of bounding box 806 with surgical instrument 704-3, and a probability P4 for a potential association 902-1 of bounding box 806 with false localization 904. In FIG. 9 , box 906 includes the determined probabilities P1, P2, P3, and P4 for each potential association. The association facility 604 may select the highest probability value from P1, P2, P3, and P4 and define the association of the bounding box 806 to a particular surgical instrument 704 or false positive designation 904 according to the potential association with the highest probability.

[0084] As another example of using such a probabilistic framework, the association facility 604 may determine the probability that bounding box 810 represents surgical instrument 704-1, the probability that bounding box 810 represents surgical instrument 704-2, the probability that bounding box 810 represents surgical instrument 704-3, and the probability that bounding box 810 represents a false positive based on one or more suitable probabilistic algorithms that consider the kinematics of one or more surgical instruments in the surgical field. The association facility 604 may then select the highest determined probability and associate bounding box 810 with the surgical instrument 704 having the highest probability or false positive. For the scenario illustrated in FIG. 8 , the association facility 604 may determine that the association of bounding box 810 with the false positive designation has the highest probability based at least in part on the lack of a close distance relationship between the image-based location of the observation represented by bounding box 810 and the kinematics-based location of surgical instrument 704.

[0085] In particular implementations, the association facility 604 may determine and make a highest probability-based association individually for each observation determined by the image processing facility 602, as in the example above. In making such associations for observations, the association facility 604 may determine the probability of a potential association of the observation without regard to other observations determined from the endoscopic images. Thus, the association of an observation to a surgical instrument or a false positive designation may be made independently of the association of any other observations determined from the endoscopic images. For example, the observation represented by bounding box 806 may be associated with the surgical instrument 704 or a false positive designation independently of the observation represented by bounding box 810 and / or independently of the association of the observation represented by bounding box 810 to a surgical instrument or a false positive designation.

[0086] In other implementations, when determining the highest probability-based association of an observation, the association facility 604 may account for one or more other observations and / or associations of observations. Such other observations may include observations determined from the same endoscopic view (e.g., from the same endoscopic image), observations determined from another endoscopic view, such as a stereoscopic image, and / or observations determined from an older endoscopic view, such as a previous video frame.

[0087] In certain implementations, the association facility 604 may be configured to determine and make the highest probability-based association by collectively considering all of the observations determined from the endoscopic images by the image processing facility 602. For example, as part of determining the probability of a potential association of an observation determined from the endoscopic images by the image processing facility 602, the association facility 604 may determine and consider the probability of a potential association of one or more other observations determined from the endoscopic images by the image processing facility 602. The association facility 604 may collectively consider the probabilities to determine the highest probability association of the observations.

[0088] In particular examples, the association facility 604 may collectively determine the probabilities of the potential associations by determining a probability for each potential set of associations between the observation and a surgical instrument or a false positive. For example, for the scenario illustrated in FIG. 8 , the association facility 604 may determine a probability for each potential set of associations of bounding boxes 806 and 810 to surgical instrument 704 or a false positive, and select the set of associations with the highest probability. For example, the association facility 604 may determine that the set of associations in which bounding box 806 is associated with surgical instrument 704-1 and bounding box 810 is associated with a false positive designation has a higher probability (as a set) than the other potential sets of associations.

[0089] 10 illustrates an exemplary probabilistic framework 1000 that may be used in the above-described example to associate bounding box 806 with surgical instrument 704-1 and bounding box 810 with a false-positive designation. Based on the illustrated probabilistic framework 1000, the association facility 604 may determine a probability PR1 for a first set of potential associations 1002-1 and a probability PR2 for a second set of potential associations 1002-2. As illustrated, the first set of potential associations 1002-1 includes a potential association 1004-1 of bounding box 806 to surgical instrument 704-1 and a potential association 1004-2 of bounding box 810 to surgical instrument 704-2. The first set of potential associations 1002-1 does not include a potential association to surgical instrument 704-3 or a false-positive designation. Also as shown, the second set of potential associations 1002-2 includes a potential association 1006-1 of bounding box 806 to surgical instrument 704-1 and a potential association 1006-2 of bounding box 810 to false positive designation 1008. The second set of potential associations 1002-2 does not include a potential association to surgical instruments 704-2 and 704-3. The first set 1002-1 and second set 1002-2 of potential associations represent two possible sets of potential one-to-one associations between bounding boxes 806 and 810 and surgical instruments 704-1, 704-2, and 704-3 or false positive designations. The association facility 604 may determine a probability for each possible set of potential associations, including the probability for the first set 1002-1 and the second set 1002-2 of potential associations.

[0090] 10 , for illustrative purposes, box 1010 includes only the determined probabilities PR1 and PR2 for each set of potential associations 1002-1 and 1002-2. The association facility 604 may select the highest probability value from PR1 and PR2 (and the probabilities for any other set(s) of potential associations) and define an association 1008 of bounding box 806 and bounding box 810 to each surgical instrument 704 or false positive designation according to the set of potential associations having the highest probability. For example, the association facility 604 may select the second set of potential associations 1002-2 as having the highest probability based on the probabilistic framework and kinematics of one or more surgical instruments 704-1, 704-2, and 704-3, and associate bounding box 806 with surgical instrument 704-1 and bounding box 810 with the false positive designation 1008 according to the second set of potential associations 1002-2.

[0091] The above-described examples of probabilistic frameworks are merely illustrative. The association facility 604 may implement any suitable probabilistic algorithm and / or probabilistic framework to determine the probability of a potential association and / or set of potential associations. The probabilistic algorithm may be configured to account for the kinematics-based position of one or more surgical instruments in the probability determination. In certain implementations, the probability of each potential association or each set of potential associations may be conditionally independent given the potential association or each set of potential associations, and a suitable probabilistic algorithm for such conditional independence may be used as the probabilistic framework by the association facility 604.

[0092] The association facility 604 may use any suitable kinematic information to determine the probability of a potential association of an observation to a surgical instrument or a false positive. In certain embodiments, for example, the association facility 604 may use the kinematics and / or other characteristics (e.g., orientation, field of view, lens properties, etc.) of the endoscope 702 to determine the probability of a potential association. For example, the association facility 604 may use the kinematics-based position and orientation of the endoscope 702 and the kinematics-based position and / or orientation of the surgical instrument 704 (e.g., the position and / or orientation of particular instrument parts such as the instrument shaft, tip, clevis, etc.) to determine the probability that the surgical instrument 704 is within the field of view of the endoscope 702, which may be used by the association facility 604 to determine the probability that the bounding box 806 corresponds to the surgical instrument 704 or a false positive and / or the probability that the bounding box 810 corresponds to the surgical instrument 704 or a false positive.

[0093] The association facility 604 may be configured to use any suitable information regarding the surgical instrument to determine the probability of a potential association of an observation to a surgical instrument or a false positive. For example, the association facility 604 may use information regarding the type of surgical instrument indicated by the kinematic information, information regarding the observed type of surgical instrument indicated by the observation, and / or information regarding the position and / or orientation of a surgical instrument part (e.g., a shaft, tip, clevis, etc. of the surgical instrument) indicated by the kinematic information and / or the observation.

[0094] The probabilistic framework used by the association facility 604 may include an observation model that defines a conditional probability function that is used by the association facility 604 to determine the probability that an observation represents a surgical instrument or a false positive based on the current state of the robotic surgical system (e.g., based on the kinematics of the surgical instrument and / or any other state information about the robotic surgical system).

[0095] In certain alternative implementations, the association facility 604 may be configured to associate an observation with a surgical instrument or a false positive designation based on the kinematics of the surgical instrument in the surgical field without using a probabilistic framework. For example, the association facility 604 may be configured to associate an observation with the most proximal surgical instrument within a predetermined distance threshold, or to associate a false positive designation when the surgical instrument is not within the predetermined distance threshold.

[0096] The association facility 604 may define or otherwise establish the association in any suitable manner, for example, by associating data representing an observation with data representing a corresponding surgical instrument or false positive designation. The association facility 604 may output data representing the determined association between the observation and the surgical instrument. In some examples, the association facility 604 may refrain from outputting data representing the determined association between the observation and the false positive designation such that the location determination facility 606 accesses and uses only observations associated with surgical instruments positioned in the surgical field.

[0097] The positioning facility 606 may be configured to access and use data representing the views and associated surgical instruments to determine the physical location of the surgical instruments in the surgical field based on the views and kinematics of the associated surgical instruments. For example, the positioning facility 606 may determine the physical location of the surgical instrument 704-1 in the surgical field based on the bounding box 806 and the kinematics for the surgical instrument 704-1. Because the association facility 604 has defined an association between the views and the surgical instruments, the positioning facility 606 may select and use the kinematics for the appropriate corresponding surgical instrument for the view in combination with the view to determine the physical location of the surgical instrument.

[0098] In particular examples, the position determination facility 606 may implement a nonlinear estimator, such as an unscented Kalman filter, particle filter, or other Bayesian filter, to determine the physical position of the surgical instrument based on the kinematics of the surgical instrument and one or more observations related to the surgical instrument. The position determination facility 606 may do this by providing the kinematics and one or more observations about the surgical instrument as inputs to the nonlinear estimator, which may process the inputs and generate and output a determined physical position of the surgical instrument.

[0099] The physical location of the surgical instrument as determined by the positioning facility 606 may represent any suitable portion of the surgical instrument (e.g., the distal clevis or the instrument tip of the surgical instrument). In particular examples, the physical location of the surgical instrument as determined by the positioning facility 606 may represent a 3D location within a world 3D space for a robotic surgical system, such as robotic surgical system 100. The physical location may be represented in any suitable manner and / or using any suitable data format.

[0100] 11-14 illustrate an exemplary implementation of tracking system 600, including a trained neural network, a probabilistic framework, and a filter. In such an implementation, image processing facility 602 may implement a trained neural network, association facility 604 may implement a probabilistic framework, and location determination facility 606 may implement a filter, which may be an unscented Kalman filter or other Bayesian filter. The implementation shown in FIGS. 11-14 may be configured to use various forms of endoscopic images and observations to determine the physical location of a surgical instrument. Accordingly, FIGS. 11-14 illustrate data flows associated with an exemplary implementation of tracking system 600.

[0101] 11 illustrates an example implementation 1100 including a trained neural network 1102, a probabilistic framework 1104, and a filter 1106. As shown, an endoscopic image 1108 of a surgical field may be input to the neural network 1102, which may determine an observation 1110 based on the endoscopic image 1108 in any of the methods described herein. The observation 1110 and kinematics 1112 of one or more surgical instruments may then be input to the probabilistic framework 1104, which may be used to associate the observation 1110 with a particular surgical instrument or a false positive designation based on the kinematics. For example, the association facility 604 may use the probabilistic framework 1104 based on the kinematics 1112 to determine probabilities of potential associations for the observation 1110 and select the association with the highest probability.

[0102] When an observation 1110 is associated with a particular surgical instrument using the probabilistic framework 1104, data 1114 representing the observation 1110 and the associated surgical instrument may be output for use as input to the filter 1106. The filter 1106 may receive and use the kinematics 1116 for the associated surgical instrument and the data 1114 representing the observation 1110 associated with the surgical instrument to determine and output the physical position 1118 of the surgical instrument in the surgical field.

[0103] 12 illustrates an exemplary implementation 1200 including a trained neural network 1202, a probabilistic framework 1204, and a filter 1206. In the illustrated example, endoscopic images may include endoscopic images 1208 (which may be a pair of 2D images stereoscopic with respect to one another), e.g., a left endoscopic image 1208-L and a right endoscopic image 1208-R, which may be input to the neural network 1202. The neural network 1202 may determine observations 1210 based on the endoscopic images 1208 in any manner described herein. As illustrated, the observations 1210 may include a left endoscopic image 1212-L and a right endoscopic image 1212-R (collectively referred to as “observations 1212”). The neural network 1202 may determine the left endoscopic image 1212-L based on the left endoscopic image 1208-L and the right endoscopic image 1212-R based on the right endoscopic image 1208-R. The neural network 1202 may be configured to process the left endoscopic image 1208-L and the right endoscopic image 1208-R individually, one at a time, or in parallel to determine the left observation 1212-L and the right observation 1212-R.

[0104] The observations 1212 and kinematics 1214 of one or more surgical instruments may be input into a probabilistic framework 1204, which may be used to associate each of the observations 1212 with a particular surgical instrument or a false positive designation based on the kinematics 1214. For example, the association facility 604 may use the probabilistic framework 1204 to determine probabilities of potential associations for each of the observations 1212 based on the kinematics and select the potential association with the highest probability for each of the observations 1212.

[0105] When the observations 1212 are each associated with a particular surgical instrument using the probabilistic framework 1204, data 1216 representing the observations 1212 and the associated surgical instruments may be output for use as input to the filter 1206. For example, the data 1216 may represent a left side observation and an associated surgical instrument and a right side observation and an associated surgical instrument, and the data 1216 may be output for use as input to the filter 1206.

[0106] The filter 1206 may receive and use data representing the kinematics for the associated surgical instrument and the observations 1212 associated with the surgical instrument to determine and output the physical position 1220 of the surgical instrument in the surgical field. The filter 1206 may determine the physical position 1220 of the surgical instrument based on the observations 1212 and the kinematics 1218 for the associated surgical instrument in any suitable manner, including by using a pair of observations 1212-L and 1212-R as separate and independent input models to the filter 1206, and compensate for the kinematics 1218 (e.g., kinematic-based position) of the associated surgical instrument. This may include using the views 1212-L and 1212-R as stereoscopic images and performing one or more stereo matching operations on the views 1212-L and 1212-R to determine the depth of surfaces depicted in the views 1212-L and 1212-R and generating a 3D image-based position (e.g., 3D coordinates or a volume such as a 3D box, sphere, ellipsoid, etc.) of the surgical instrument as represented by the views 1212-L and 1212-R. The filter 1206 may use the generated image-based 3D position to correct the kinematics-based 3D position of the surgical instrument indicated by the kinematics 1218 to determine a physical position 1220 in world 3D space.

[0107] In the illustrated example, the left endoscopic image 1208-L may be processed independently of the right endoscopic image 1208-R by the neural network 1202 to determine a left observation 1212-L, which may be processed independently of the right observation 1212-R to associate a surgical instrument with the left observation 1212-L. Similarly, the right endoscopic image 1208-R may be processed independently of the left endoscopic image 1208-R to determine a right observation 1212-R, which may be processed independently of the left observation 1212-L to associate a surgical instrument with the right observation 1212-R. The left observation 1212-L and the right observation 1212-R may be input to the filter 1206 (as independent observations) and used by the filter 1206 to determine a physical position 1220 based on the observations 1212 and the kinematics 1218.

[0108] Such independent processing of the 2D endoscopic images may conserve computational resources by allowing certain operations of the tracking system 600 to be performed using 2D data instead of 3D data to determine the estimated 3D physical location 1220 of the surgical instrument. Moreover, the filter 1206 can determine the physical location 1220 even when one of the two observations 1212 is unavailable or does not depict the surgical instrument. For example, if a surgical instrument is depicted in the left endoscopic image 1208-L but not in the right endoscopic image 1208-R (e.g., because the surgical instrument is blocked from the right camera view by tissue), both the left and right endoscopic images 1208-L and 1208-R may be processed by the neural network 1202 to determine the observations 1212-L and 1212-R associated with the surgical instrument or false positives and input to the filter 1206. The filter 1206 may process both the observations 1212-L and 1212-R as inputs for generating the surgical instrument physical location 1220, even though the right observation 1212-R is not associated with the surgical instrument. Alternatively, the filter 1206 may use only the left observation 1212-L and not the right observation 1212-R as input when the right observation 1212-R is not associated with the surgical instrument by the association facility 604 (e.g., because the surgical instrument is within the field of view of the left camera view but is blocked from the right camera view by tissue). The accuracy of the physical location 1220 determined by the filter 1206 of the surgical instrument may be maintained at least over some time, even if the surgical instrument is temporarily depicted in only one of the endoscopic images 1208.

[0109] 12 illustrates an implementation 1200 in which a pair of stereoscopic images 1208-L and 1208-R are input to a neural network 1202, which determines a pair of independent observations 1212-L and 1212-R based on the respective stereoscopic images 1208-L and 1208-R. In other implementations, the tracking system 600 may be configured to process other forms of endoscopic imagery and / or determine other observations in other formats and use the other observations in other formats to determine the physical position of the surgical instrument. For example, other implementations of the system 600 may be configured to process a combined endoscopic image, which may include multiple endoscopic images combined (e.g., stacked) into an aggregate endoscopic image, such as a pair of stereoscopic images stacked into a combined image, or a sequence of multiple endoscopic images (e.g., a sequence of two, three, or more video frames) stacked into a combined image. Additionally or alternatively, other implementations of system 600 may be configured to determine views indicative of 3D image-based positions of objects of interest depicted within the endoscopic images.

[0110] 13 illustrates an exemplary implementation 1300 including a trained neural network 1302, a probabilistic framework 1304, and a filter 1306. In the illustrated example, the endoscopic images may include a left 2D endoscopic image 1308-L and a right 2D endoscopic image 1308-R, which are stereoscopic with respect to one another. The image processing equipment 602 may combine the endoscopic images 1308-L and 1308-R to form a combined endoscopic image 1308-C. This may be performed in any suitable manner, such as by the image processing equipment 602 stacking the endoscopic images 1308-L and 1308-R to form the combined endoscopic image 1308-C. If the endoscopic images 1308-L and 1308-R each include a three-channel RGB image, for example, the image processing equipment 602 may stack the three-channel RGB images to form a six-channel combined image.

[0111] The image processing facility 602 may input the combined endoscopic image 1308-C to the neural network 1302, which may be suitably configured to determine an observation 1310 based on the combined endoscopic image 1308-C in any of the manners described herein. In a particular example, the neural network 1302 may be configured to determine the observation 1310 such that the observation 1310 indicates a 3D image-based location in 3D space of the object of interest depicted in the combined endoscopic image 1308-C (e.g., by performing a stereo matching operation to determine depth values ​​of the surface of the object of interest from the combined endoscopic image 1308-C representing a combined stereoscopic image depicting the object of interest).

[0112] In an alternative implementation, the image processing facility 602 may receive the endoscopic images 1308-L and 1308-R instead of the combined endoscopic image 1308-C and perform a stereo matching operation on the endoscopic images 1308-L and 1308-R to determine depth values ​​for the surface of the object of interest from the endoscopic images 1308-L and 1308-R. Using the determined depth values, the neural network 1302 may determine an observation 1310 such that the observation 1310 indicates a 3D image-based location of the object of interest in 3D space.

[0113] The observations 1310 and kinematics 1312 of one or more surgical instruments may then be input into a probabilistic framework 1304, which may be used to associate the observations 1310 with a particular surgical instrument or a false positive designation based on the kinematics, e.g., the association facility 604 may use the probabilistic framework 1304 to determine probabilities of potential associations for the observations 1310 based on the kinematics 1312 and select the association with the highest probability. Because the observations 1310 indicate a 3D image-based position of an object of interest in 3D space, the association facility 604 may perform a particular operation using the 3D spatial data. For example, the association facility 604 may analyze the 3D image-based position of the object of interest in 3D space against one or more 3D kinematics-based positions of one or more surgical instruments in 3D space, the 3D kinematics-based positions being indicated by the kinematics 1312. The association facility 604 may use the results of the analysis (eg, the determined distances) to determine the probability of a potential association for the observation 1310 .

[0114] When an observation 1310 is associated with a particular surgical instrument using the probabilistic framework 1304, data 1314 representing the observation 1310 and the associated surgical instrument may be output for use as input to the filter 1306. The filter 1306 may access and use the kinematics 1316 for the associated surgical instrument and the data 1314 representing the observation 1310 associated with the surgical instrument to determine and output a physical position 1318 of the surgical instrument in the surgical field. Because the observation 1310 indicates a 3D image-based position of an object of interest in 3D space, the filter 1306 may perform certain operations using the 3D spatial data. For example, the filter 1306 may determine the physical position 1318 of the surgical instrument by adjusting the 3D kinematics-based position of the surgical instrument as indicated by the kinematics 1316 with the 3D image-based position of the object of interest as indicated by the observation 1310 to determine the 3D physical position of the surgical instrument in a 3D space, such as a global 3D space associated with the robotic surgical system.

[0115] By performing certain operations in three-dimensional space, the implementation 1300 may determine a very precise physical location 1318 of the surgical instrument, particularly with respect to the depth of the surgical instrument. Additionally or alternatively, by using the 3D image-based location indicated by the observation 1310, the implementation 1300 may minimize the number of translations from 2D to 3D space performed to determine the physical location 1318 of the surgical instrument.

[0116] 14 illustrates an example implementation 1400 including a trained neural network 1402, a probabilistic framework 1404, and a filter 1406. In the illustrated example, an endoscopic image may include a sequence of endoscopic images (e.g., a sequence of video frames) captured over time, such as endoscopic images 1408-T1, 1408-T2, and 1408-T3. The image processing facility 602 may combine endoscopic images 1408-T1, 1408-T2, and 1408-T3 to form a combined endoscopic image 1408-C. This may be performed in any suitable manner, such as by stacking endoscopic images 1408-T1, 1408-T2, and 1408-T3 to form combined endoscopic image 1408-C. For example, if the endoscopic images 1408-T1, 1408-T2, and 1408-T3 each include a three-channel RGB image, the image processing facility 602 may stack the three-channel RGB images to form a nine-channel combined image.

[0117] The image processing facility 602 may input the combined endoscopic image 1408-C to the neural network 1402, which may be suitably configured to determine an observation 1410 based on the combined endoscopic image 1408-C in any of the manners described herein. In a particular example, the neural network 1402 may be configured to determine the observation 1410 such that the observation 1410 indicates an image-based location of an object of interest depicted in the combined endoscopic image 1408-C. The neural network 1402 may be configured to identify motion (e.g., velocity, acceleration, etc.) of the object of interest from the combined endoscopic image 1408-C, which represents a sequence of endoscopic images captured at different times, and to determine the image-based location of the object of interest using the identified motion. In some examples, this may include using the recognized motion to determine depth points for the surface of the object of interest and to determine a 3D image-based location of the object of interest, which may be particularly useful for estimating depth from a sequence of 2D images captured from the same vantage point (e.g., when a surgical instrument is within the field of view of only one camera). The use of motion-based cues may contribute to the accuracy of the image-based location of the object of interest. The image-based location of the object of interest may include a 2D or 3D image-based location, as described herein.

[0118] The observations 1410 and kinematics 1412 of one or more surgical instruments may then be input into a probabilistic framework 1404, The probabilistic framework 1404 may be used to associate an observation 1410 with a particular surgical instrument or a false positive designation based on kinematics, for example, by the association facility 604 using the probabilistic framework 1404 to determine probabilities of potential associations for the observation 1410 and selecting the association with the highest probability.

[0119] When an observation 1410 is associated with a particular surgical instrument using the probabilistic framework 1404, data 1414 representing the observation 1410 and the associated surgical instrument may be output for use as input to the filter 1406. The filter 1406 may access and use the kinematics for the associated surgical instrument and the data 1414 representing the observation 1410 associated with the surgical instrument to determine and output the physical location 1418 of the surgical instrument in the surgical field.

[0120] 11-14 illustrate exemplary implementations of tracking system 600 configured to process various types of endoscopic images and / or determine various observations, the examples are illustrative. Other implementations may be configured to process other types of images (e.g., other types of endoscopic and / or non-endoscopic images) and / or determine other suitable observations. To this end, the neural network may be appropriately configured to suit the particular implementation.

[0121] By using a combination of image-based observation and kinematics of a surgical instrument to track the surgical instrument as described herein, certain technical benefits may be provided by tracking system 600. For example, the precision, accuracy, efficiency, and / or reliability with which a surgical instrument is tracked may be improved compared to using image-based tracking alone or kinematics-based tracking alone. For example, a kinematics-based position indicated by the kinematics of the surgical instrument may be used to determine the physical position of the surgical instrument to a certain level of accuracy (e.g., to within about one centimeter). By correcting the kinematics-based position of the surgical instrument with the image-based position indicated by observation as described herein, the physical position of the surgical instrument may be determined and may have an improved level of accuracy (e.g., to within about one millimeter).

[0122] Figure 15 illustrates an example of such improved levels of accuracy in the determined physical position of a surgical instrument. Figure 15 shows an endoscopic image 1500 of a surgical field. The endoscopic image 1500 depicts a surgical instrument 1502 positioned in the surgical field. To illustrate the level of accuracy associated with the physical position of the surgical instrument 1502 determined based solely on kinematics, an indicator 1504 (e.g., an ellipse illustrating an ellipsoid) is overlaid on the endoscopic image 1500, indicating the level of accuracy for the kinematics-based position of the surgical instrument 1502 by the size and / or area covered by the indicator 1504. To illustrate the improved level of accuracy associated with the physical position of the surgical instrument 1502, which is determined based on both kinematics and one or more image-based observations, an indicator 1506 (e.g., a circle representing a sphere) is overlaid on the endoscopic image 1500, and the size and / or area covered by the indicator 1506 indicates the level of accuracy for the physical position of the surgical instrument 1502, which is determined based on both kinematics and one or more image-based observations as described herein.

[0123] In certain embodiments, tracking system 600 may be configured to selectively capture and use endoscopic images to track a surgical instrument. For example, tracking system 600 may be configured to use kinematics for the surgical instrument to select when to capture and use endoscopic images to track the surgical instrument. By way of example, tracking system 600 may selectively use endoscopic images to track a surgical instrument in response to movement of the surgical instrument as indicated by kinematics for the surgical instrument and / or in response to changes in orientation, angle, position, or other characteristics of the endoscope as indicated by kinematics of the endoscope. Thus, tracking system 600 may be configured to switch between tracking a surgical instrument using only kinematics without image-based observation and tracking a surgical instrument using kinematics and image-based observation as described herein. Such an implementation may conserve resources (e.g., computational resources, energy, etc.) of tracking system 600 and / or the robotic surgical system.

[0124] Tracking a surgical instrument as described herein may provide one or more benefits of a robotic surgical system. For example, tracking a surgical instrument as described herein may facilitate one or more configurations of a robotic surgical system. In certain implementations, the level of accuracy of the physical position of a surgical instrument determined as described herein may facilitate certain configurations of a robotic surgical system that require a high level of accuracy.

[0125] As an example, tracking of surgical instruments as described herein may include: A robotic surgical system may use a display screen to position specific user interface content (e.g., graphics, icons, indicators, etc.) relative to the physical location of the surgical instrument or relative to the physical location of specific tissue contacted by the surgical instrument. By way of example, a robotic surgical system may juxtapose user interface content with a visual representation of a surgical instrument on a display screen in a manner that positions the user interface content proximate to the tip of the surgical instrument without obscuring the surgical instrument from the view of the surgeon or other surgical team members. For example, an indicator may be positioned to indicate the surgical mode to the surgeon (e.g., high-energy mode for cauterization) in a manner that makes the indicator easily visible to the surgeon without requiring the surgeon to take their eyes off the immediate vicinity of the surgical instrument, or an indicator may be positioned to indicate specific tissue already contacted by the probe to help the surgeon see which tissue has already been analyzed by the probe. Such precise positioning of user interface content, which may provide benefits to the surgical team and, thereby, the patient, may be facilitated by accurate determination of the physical location of the surgical instrument as described herein.

[0126] As other examples, surgical instrument tracking as described herein may be used by a robotic surgical system to warn of or prevent collisions of surgical instruments and / or robotic arms, automate the movement of surgical instruments so that the robotic surgical system may control specific movements of the surgical instruments (e.g., by automating the movement of a probe, such as an oncology probe, to systematically touch surface points of tissue, automating the movement of an endoscope based on the movement of a surgical tool, etc.), recognize specific tasks being performed during a surgical session (e.g., by recognizing when a defined segment of a surgical procedure is being performed), correct for inaccuracies in the kinematics of surgical instruments, such as inaccuracies introduced by long kinematic chains and / or flexures of flexible shafts of surgical instruments (e.g., flexible shafts used in single-port entry systems), and calibrate surgical instruments and / or robotic arms (e.g., by moving the surgical instrument into the field of view of an endoscope and calibrating the surgical instrument using information extracted from one or more images of the scene), which may improve the workflow for calibration compared to traditional calibration techniques. These examples are illustrative only. Tracking of surgical instruments as described herein may be used by robotic surgical systems in any other suitable manner.

[0127] In certain examples, the endoscopic image may be used (e.g., by tracking system 600) for camera-arm registration between a camera (e.g., an endoscope camera) and a robotic arm mounted on a separate patient side system (e.g., a separate patient side cart). In implementations such as those described above, where the endoscope and surgical instruments are mounted on the robotic arms of the same patient side cart, the kinematic chain connecting the endoscope and surgical instruments is known, and the kinematics of the endoscope and surgical instruments based on the known kinematic chain may be accessed and used by tracking system 600 to track the position of the instruments as described herein. However, in other implementations, where the endoscope and surgical instruments are mounted on the robotic arms of separate, unconnected patient side carts, the complete kinematic chain connecting the endoscope and surgical instruments is not known because the kinematic chain links between the patient side carts are not known. That is, the physical relationship between the patient side carts is not known and therefore not accessible by tracking system 600. In such implementations with separate patient side carts, tracking system 600 may be configured to perform a registration process using endoscopic images captured by the endoscopes to align (register) surgical instruments mounted on one patient side cart with endoscopes mounted on another patient side cart. Tracking system 600 may be configured to perform the registration process once (e.g., as part of a calibration, setup procedure, or other initial registration) and / or periodically or continuously after the initial registration to refine the initial registration (e.g., to account for changes in physical position that may have various kinematic errors) and / or to account for physical adjustments to the patient side cart position (e.g., intraoperative adjustments of the patient side cart).

[0128] To perform the registration process, the tracking system 600 may determine the location of the surgical instrument (e.g., the location of the surgical instrument shaft in the left and right endoscopic images) based on endoscopic images of the surgical field using a trained neural network. In certain examples, this may include using the trained neural network to determine observations of an object of interest depicted in the endoscopic images, associating the observations of the object of interest with the surgical instrument in any suitable manner (e.g., object recognition), and determining the location (e.g., image-based position) of the surgical instrument.

[0129] The tracking system 600 may then perform a constrained optimization to fit a model of the surgical instrument (e.g., a model of the surgical instrument shaft) to the determined location of the surgical instrument (e.g., the location of the surgical instrument shaft in the left and right endoscopic images). The tracking system 600 may be configured to constrain the optimization to search only for solutions that are rotations and translations on the plane of the floor on which the patient side cart is placed. This constrained optimization may yield faster and / or more accurate results than traditional optimization performed in all six degrees of freedom.

[0130] The tracking system 600 may perform the registration process for one surgical instrument at a time or for multiple instruments simultaneously. If a single surgical instrument is mounted on a single patient side cart and / or is mounted or registered alone, then associating the views to the surgical instrument may be straightforward. If two or more surgical instruments are each mounted on separate patient side carts and / or are mounted or registered simultaneously, the tracking system 600 may be configured to associate the views determined from the endoscopic images to the surgical instruments in any suitable manner, including by using a probabilistic framework as described herein.

[0131] The registration process may be used by tracking system 600 to determine missing links in the kinematic chain connecting the endoscope and surgical instruments mounted on separate patient side carts. Tracking system 600 may represent the missing links as transformations that define the estimated physical relationship between the patient side carts. The transformations may define rotations and translations that may be applied to transform data points from the frame of reference of one patient side cart to the frame of reference of another patient side cart (e.g., to transform coordinate points from the coordinate system of one patient side cart to the coordinate system of another patient side cart).

[0132] Tracking system 600 may use the missing link to complete the kinematic chain connecting the endoscope and surgical instruments that are mounted on a separate patient side cart so that the complete kinematic chain is known and accessible to tracking system 600. Tracking system 600 may then use the kinematics of the endoscope and / or surgical instruments to track the position of the surgical instruments as described herein. For example, tracking system 600 may use the endoscopic image as described herein to correct the kinematic position of the surgical instruments.

[0133] In certain instances, the use of neural networks as described herein may provide one or more advantages over traditional computer vision techniques. For example, the use of neural networks to determine observations about objects of interest depicted in endoscopic images as described herein may be significantly faster and / or more accurate than traditional computer vision techniques (e.g., traditional pure vision approaches in which algorithms that extract specific features from images are hand-crafted). Particular implementations of neural networks for determining observations about distal clevises of surgical instruments have been found to be approximately 10 times faster (e.g., 10 Hz compared to 1 Hz) than traditional marker-based computer vision techniques.

[0134] As used herein, a neural network may include an input layer, any suitable number of hidden layers, and an output layer. In example implementations in which multiple hidden layers are included in a neural network, the multiple hidden layers may be layered in any suitable manner. A neural network that includes multiple hidden layers is sometimes referred to as a "deep neural network."

[0135] Although exemplary implementations that include and / or use neural networks are described herein, alternative implementations may use other suitable machine learning models configured to learn how to extract specific features from endoscopic images for object detection and / or classification.

[0136] Figure 16 illustrates an exemplary method 1600 for tracking the position of a robotically operated surgical instrument. While Figure 16 illustrates exemplary operations according to one embodiment, other embodiments may omit, add, reorder, and / or modify any of the operations shown in Figure 16. One or more of the operations shown in Figure 16 may be performed by a tracking system such as system 600, any components included therein, and / or any implementation thereof.

[0137] In operation 1602, the tracking system may determine, based on an endoscopic image of the surgical field, observations about an object of interest depicted in the endoscopic image using a trained neural network. Operation 1602 may be performed in any manner described herein.

[0138] In operation 1604, the tracking system may associate observations of the object of interest with the robotically operated surgical instrument based on the probabilistic framework and the kinematics of the robotically operated surgical instrument located in the surgical field. Operation 1604 may be performed in any manner described herein.

[0139] In operation 1606, the tracking system may determine the physical location of the robotically manipulated surgical instrument in the surgical field based on the kinematics of the robotically manipulated surgical instrument and observations associated with the robotically manipulated surgical instrument. Operation 1606 may be performed in any of the methods described herein.

[0140] Figure 17 illustrates another exemplary method 1700 for tracking the position of a robotically operated surgical instrument. While Figure 17 illustrates exemplary operations according to one embodiment, other embodiments may omit, add, reorder, and / or modify any of the operations shown in Figure 17. One or more of the operations shown in Figure 17 may be performed by a tracking system such as system 600, any components included therein, and / or any implementation thereof.

[0141] In operation 1702, the tracking system may determine, based on an endoscopic image of the surgical field, observations about an object of interest depicted in the endoscopic image using a trained neural network. Operation 1702 may be performed in any of the methods described herein.

[0142] In operation 1704, the tracking system may associate the observation of the object of interest with the robotic surgical instrument or a false positive designation based on the kinematics of the robotic surgical instrument located in the surgical field. Operation 1704 may be performed in any of the methods described herein.

[0143] In operation 1706, the tracking system may determine a physical location of the robotically operated surgical instrument in the surgical field based on the kinematics of the robotically operated surgical instrument and the observations associated with the robotically operated surgical instrument when observations of the object of interest are associated with the robotically operated surgical instrument in operation 1704. Operation 1706 may be performed in any of the methods described herein.

[0144] In operation 1708, the tracking system may refrain from using the observation to determine the physical location of the robotically operated surgical instrument in the surgical field when the observation of the object of interest is associated with a false positive designation in the procedure 1704. Operation 1706 may be performed in any of the methods described herein.

[0145] In particular embodiments, one or more of the systems, components, and / or processes described herein may be implemented and / or performed by one or more appropriately configured computing devices. To this end, one or more of the above-described systems and / or components may include or be implemented by any computer hardware and / or computer-implemented instructions (e.g., software) embodied in at least one non-transitory computer-readable medium configured to perform one or more of the processes described herein. In particular, system components may be implemented on one physical computing device, or on more than one physical computing device. Thus, system components may include any number of computing devices and may use any of a number of computer operating systems.

[0146] In particular embodiments, one or more of the processes described herein may be implemented, at least in part, as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices. Generally, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes those instructions to thereby perform one or more processes, including one or more of the processes described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0147] Computer-readable media (also referred to as processor-readable media) include any non-transitory medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by a processor of a computer). Such media may take many forms, including, but not limited to, non-volatile media and / or volatile media. Non-volatile media may include, for example, optical or magnetic disks and other persistent memory. Volatile media may include, for example, dynamic random access memory ("DRAM"), which typically constitutes main memory. Common forms of computer-readable media include, for example, disks, hard disks, magnetic tapes, any other magnetic media, compact disc read-only memory ("CD-ROM"), digital video discs ("DVD"), any other optical media, random access memory ("RAM"), programmable read-only memory ("PROM"), electrically erasable programmable read-only memory ("EPROM"), FLASH-EEPROM, any other memory chip or cartridge, or any other tangible media from which a computer can read.

[0148] FIG. 18 illustrates an exemplary computing device 1800 that may be particularly configured to perform one or more of the processes described herein. As shown in FIG. 18, computing device 1800 may include a communication interface 1802, a processor 1804, a storage device 1806, and an input / output ("I / O") module 1808 that are communicatively connected via a communication infrastructure 1810. While an exemplary computing device 1800 is shown in FIG. 18, the components illustrated in FIG. 18 are not intended to be limiting. In other embodiments, additional or alternative components may be used. The components of computing device 1800 shown in FIG. 18 will now be described in additional detail.

[0149] The communication interface 1802 may be configured to communicate with one or more computing devices. Examples of the communication interface 1802 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0150] The processor 1804 generally represents any type or form of processing unit that can process data or that can interpret, execute, and / or direct the execution of one or more of the instructions, processes, and / or operations described herein. The processor 1804 may direct the execution of operations in accordance with one or more applications 1812 or other computer-executable instructions, such as those that may be stored on the storage device 1806 or other computer-readable medium.

[0151] Storage device(s) 1806 may include one or more data storage media, devices, or configurations, and may utilize any type, form, and combination of data storage media and / or devices. For example, storage device(s) 1806 may include, but are not limited to, a hard drive, a network drive, a flash drive, a magnetic disk, an optical disk, RAM, dynamic RAM, other non-volatile and / or volatile data storage units, or any combination or subcombination thereof. Electronic data, including the data described herein, may be stored temporarily and / or persistently on storage device(s) 1806. For example, data representing one or more executable applications 1812 configured to instruct processor 1804 to perform any of the operations described herein may be stored on storage device(s) 1806. In some examples, data may be located in one or more databases residing within storage device(s) 1806.

[0152] I / O module 1808 may include one or more I / O modules configured to receive user input and provide user output. One or more I / O modules may be used to receive input for a single virtual reality experience. I / O module 1808 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, I / O module 1808 may include hardware and / or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0153] I / O module 1808 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In particular embodiments, I / O module 1808 is configured to provide graphical data to a display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content as may be useful in a particular implementation.

[0154] In some examples, any of the facilities described herein may be implemented by or within one or more components of computing device 1800. For example, one or more applications 1812 resident in storage device 1806 may be configured to direct processor 1804 to perform one or more processes or functions associated with facilities 602-606 of system 600. Similarly, storage device 608 of system 600 may be implemented by storage device 1806 or a component thereof.

[0155] In the preceding description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the following claims. For example, particular features of one embodiment described herein may be combined with, or substituted for, features of other embodiments described herein. Accordingly, the description and drawings should be considered in an illustrative sense, and not in a limiting sense.

Claims

1. a memory for storing instructions; a processor communicatively coupled to the memory; The processor executes the instructions to determining a kinematics-based physical location of the robotically manipulated instrument in the surgical field based on the kinematics of the robotically manipulated instrument; determining an observation about the object of interest based on image data representing one or more images of the surgical field; Associating the observations with the robotically operated instruments in the surgical field; determining a corrected physical location of the robotically manipulated instrument in the surgical field by adjusting the kinematics-based physical location based on the observations associated with the robotically manipulated instrument; It is configured as follows: Surgical instrument tracking system.

2. The surgical instrument tracking system of claim 1 , wherein the observation is associated with the robotically operated instrument based on a determined probability of association of the observation to the robotically operated instrument.

3. The surgical instrument tracking system of claim 2 , wherein the determined probability is determined based on the kinematics of the robotically manipulated instrument.

4. adjusting the kinematics-based physical location based on the observations associated with the robotically operated instrument, determining an image-based physical location of the robotically operated instrument in the surgical field based on the observations; and adjusting the kinematics-based physical location based on the image-based physical location. The surgical instrument tracking system of claim 1 .

5. the kinematics-based physical location comprises a first three-dimensional (3D) position in 3D space; the image-based physical location includes a second 3D position within the 3D space; The surgical instrument tracking system of claim 4.

6. the one or more images include a sequence of video frames; the image-based physical location is determined based on a motion of the object of interest depicted within the sequence of video frames; The surgical instrument tracking system of claim 4.

7. The processor executes the instructions to determining additional observations of additional objects of interest based on the image data; Associating the additional observation about the additional object of interest with a false positive designation; refraining from using the additional observations of the additional object of interest to determine the corrected physical location of the robotically operated instrument. It is configured as follows: A surgical instrument tracking system according to any one of claims 1 to 6.

8. 8. The surgical instrument tracking system of claim 1, wherein the observation of the object of interest indicates an image-based physical location of the robotically operated instrument in the surgical field and motion cues associated with the object of interest.

9. 8. The surgical instrument tracking system of claim 1, wherein the observation of the object of interest indicates an image-based physical location of the robotically operated instrument in the surgical field and a type of instrument associated with the object of interest.

10. the processor is configured to execute the instructions to output data representing the corrected physical location of the robotically manipulated instrument in the surgical field for use by a robotic surgical system to display user interface content juxtaposed with a visual representation of the robotically manipulated instrument displayed by a display device. A surgical instrument tracking system according to any one of claims 1 to 9.

11. determining, by a surgical instrument tracking system, a kinematics-based physical location of the robotically manipulated instrument in the surgical field based on the kinematics of the robotically manipulated instrument; determining, by the surgical tool tracking system, an observation about an object of interest based on image data representing one or more images of the surgical field; associating the observations with the robotically operated instruments in the surgical field by the surgical instrument tracking system; determining, by the surgical instrument tracking system, a corrected physical location of the robotically manipulated instrument in the surgical field by adjusting the kinematics-based physical location based on the observations associated with the robotically manipulated instrument. method.

12. Associating the observations with the robotically operated instrument in the surgical field includes: determining a probability of associating the observation with the robotically operated instrument based on the kinematics of the robotically operated instrument; and associating the observation with the robotic instrument based on the probability. The method of claim 11.

13. adjusting the kinematics-based physical location based on the observations associated with the robotically operated instrument, determining an image-based physical location of the robotically operated instrument in the surgical field based on the observations; and adjusting the kinematics-based physical location based on the image-based physical location. The method of claim 11.

14. the kinematics-based physical location comprises a first three-dimensional (3D) position in 3D space; the image-based physical location includes a second 3D position within the 3D space; The method of claim 13.

15. the one or more images include a sequence of video frames; the image-based physical location is determined based on a motion of the object of interest depicted within the sequence of video frames; The method of claim 13.

Images