System and method for tracking position of robotically manipulated surgical instrument

Abstract

The invention relates to a system and method for tracking the position of a robotically manipulated surgical instrument. An exemplary surgical instrument tracking system includes at least one physical computing device that determines an observation of a subject of interest shown in an endoscopic image based on the endoscopic image of a surgical region and using a trained neural network, observations of a subject of interest are associated with a robotically manipulated surgical instrument located in a surgical region based on a probabilistic framework and motion of the robotically manipulated surgical instrument, and observation of the subject of interest is associated with the robotically manipulated surgical instrument based on motion of the robotically manipulated surgical instrument and the observations associated with the robotically manipulated surgical instrument. A physical location of a robotically manipulated surgical instrument in a surgical region is determined.

Description

[0001] This application is a divisional application of Chinese patent application 2019800589996 (PCT / US2019 / 043825), whose international application date is July 27, 2019 and whose national phase entry date is March 10, 2021, and whose name is “System and method for tracking the position of robotically manipulated surgical instruments”. Technical Field

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 712,368, filed on July 31, 2018, entitled “SYSTEM AND METHOD FOR TRACKING THE POSITION OF A ROBOTICALLY MANIPULATED SURGICAL INSTRUMENT,” the contents of which are hereby incorporated by reference in their entirety. Background Art

[0003] A robotic surgical system allows a surgeon to control robotically manipulated surgical instruments to perform a surgical procedure on a patient. For example, for minimally invasive surgery, the robotically manipulated surgical instruments are inserted into the patient through one or more catheters. The surgical instruments typically include an endoscope that captures an image of the surgical field and one or more surgical tools that are automatically manipulated by the robotic surgical system to perform the surgical procedure. The surgeon views the endoscopic image of the surgical field and controls the movement of the robotically manipulated surgical instruments using the master controls of the robotic surgical system to perform the surgical procedure.

[0004] Tracking the position of robotically manipulated surgical instruments within a surgical field is an important component of a robotic surgical system. However, such tracking of surgical instrument positions by a robotic surgical system is technically challenging, particularly when a certain level of precision, accuracy, efficiency, and / or reliability is required. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The accompanying drawings illustrate various embodiments and are part of this specification. The embodiments shown are examples only and do not limit the scope of the present disclosure. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0006] Figure 1 An exemplary robotic surgical system according to the principles described herein is shown.

[0007] Figure 2 The invention shows a method including Figure 1 An exemplary patient-side system within a robotic surgical system.

[0008] Figure 3 The invention shows a method including Figure 2 An exemplary robotic arm within the patient-side system.

[0009] Figure 4The invention shows a method according to the principles described herein including Figure 1 An exemplary surgeon's console within a robotic surgical system.

[0010] Figure 5 An exemplary stereoscopic endoscope is shown positioned in an exemplary surgical field associated with a patient according to principles described herein.

[0011] Figure 6 An exemplary surgical instrument tracking system according to the principles described herein is shown.

[0012] Figure 7 Shown are views of surgical instruments within a surgical field according to the principles described herein.

[0013] Figure 8 Shown are exemplary endoscopic images of a surgical field according to the principles described herein.

[0014] Figures 9 and 10 An exemplary probabilistic framework according to the principles described herein is shown.

[0015] Figures 11 to 14 An exemplary implementation of a surgical instrument tracking system according to the principles described herein is shown.

[0016] Fig.15 Shown is an endoscopic image of a surgical field with a position indicator superimposed thereon according to principles described herein.

[0017] Figure 16 to Figure 17 An exemplary method of tracking a surgical instrument according to the principles described herein is shown.

[0018] Fig.18 An exemplary computing system according to the principles described herein is shown. DETAILED DESCRIPTION

[0019] Systems and methods for tracking the position of a robotically manipulated surgical instrument are described herein. As will be described in greater detail below, in certain embodiments, an exemplary surgical instrument tracking system may be configured to determine observations of an object of interest shown in the endoscopic image based on an endoscopic image of a surgical area and using a trained neural network, associate the observations of the object of interest with the robotically manipulated surgical instrument based on a probabilistic framework and motion of the robotically manipulated surgical instrument located in the surgical area, and determine the physical position of the robotically manipulated surgical instrument in the surgical area based on the motion of the robotically manipulated surgical instrument and the observations associated with the robotically manipulated surgical instrument.

[0020] As will be described in greater detail herein, in certain embodiments, an exemplary surgical instrument tracking system can be configured to determine an observation of an object of interest shown in an endoscopic image of a surgical area using a trained neural network and to associate the observation of the object of interest with a robotically manipulated surgical instrument based on the movement of the robotically manipulated surgical instrument located in the surgical area or to associate the observation of the object of interest with a false positive designation. When the observation of the object of interest is associated with the robotically manipulated surgical instrument, the surgical instrument tracking system can determine the physical location of the robotically manipulated surgical instrument in the surgical area based on the movement of the robotically manipulated surgical instrument and the observation associated with the robotically manipulated surgical instrument. When the observation of the object of interest is associated with a false positive designation, the surgical instrument tracking system can avoid using the observation to determine the physical location of the robotically manipulated surgical instrument in the surgical area.

[0021] As used herein, the position of an instrument may refer to the three-dimensional ("3D") position and / or orientation of a desired portion of the instrument, such as an end effector (e.g., jaws, scissors, etc.), a distal tool tip, a wrist, a shaft, and / or another structural component of the instrument. For example, the position of an instrument may refer to the 3D position and / or orientation of a component of the instrument in a 3D space, such as a 3D position and / or orientation in a 3D space defined by a 3D coordinate system.

[0022] As described herein, by determining the physical location of a robotically manipulated surgical instrument, an exemplary surgical instrument tracking system and method can track surgical instrument locations with a level of precision, accuracy, efficiency, and / or reliability that promotes one or more features, benefits, and / or advantages of a robotic surgical system. In certain embodiments, such precision, accuracy, efficiency, and / or reliability can be provided by the exemplary surgical instrument tracking system without the use of dedicated computer vision markers (e.g., bar codes, colors, patterns, etc. integrated on or otherwise attached to the surgical instrument) on the surgical instrument, which can be referred to as "markerless" tracking of surgical instruments. In certain embodiments, such precision, accuracy, efficiency, and / or reliability can be provided by the exemplary surgical instrument tracking system in real time or near real time using limited computing resources.

[0023] Referring now to the drawings, various embodiments will be described in more detail.The systems and methods described herein may provide one or more of the benefits noted above and / or various additional and / or alternative benefits that will become apparent herein.

[0024] The surgical instrument tracking systems and methods described herein may be operated as part of or in conjunction with a robotic surgical system. Therefore, to facilitate understanding of the surgical instrument tracking systems and methods described herein, an exemplary robotic surgical system will now be described. The exemplary robotic surgical system described is illustrative and not restrictive. The surgical instrument tracking systems and methods described herein may be operated as part of or in conjunction with other suitable robotic surgical systems.

[0025] Figure 1 An exemplary robotic surgical system 100 is shown. As shown, the robotic surgical system 100 may include a patient-side system 102 (sometimes referred to as a patient-side cart), a surgeon console 104, and a vision cart 106 that are communicatively coupled to each other. The robotic surgical system 100 may be used by a surgical team to perform a robotically enabled surgical procedure on a patient 108. As shown, the surgical team may include a surgeon 110-1, an assistant 110-2, a nurse 110-3, and an anesthesiologist 110-4, all of whom are collectively referred to as "surgical team members 110." Additional or alternative surgical team members may be present during a surgical procedure as may serve a particular embodiment. While Figure 1 A minimally invasive surgical procedure is shown being performed, but it will be appreciated that the robotic surgical system 100 may similarly be used to perform open surgical procedures, or other types of surgical procedures that benefit from the robotic surgical system 100 may similarly be performed. In addition, it will be appreciated that surgical procedures that may employ the robotic surgical system 100 may include not only open surgical procedures, but also open surgical procedures. Figure 1 The operative phase of the surgical procedure shown in FIG. 1 may also include pre-operative, post-operative, and / or other suitable phases of the surgical procedure.

[0026] As shown, the patient-side system 102 may include a plurality of robotic arms 112 (e.g., robotic arms 112-1 to 112-4) that may be coupled to a plurality of robotically manipulated surgical instruments 114 (e.g., surgical instruments 114-1 to 114-4). Each surgical instrument 114 may include any suitable surgical tool, medical tool, monitoring instrument (e.g., endoscope), diagnostic instrument, etc. that may be used for a robotically enabled surgical procedure on a patient 108 (e.g., by being at least partially inserted into the body of the patient 108 to perform a robotically enabled surgical procedure on the patient 108). Note that while the patient-side system 102 is shown and described herein as a cart having a plurality of robotic arms 112 for exemplary purposes, in various other embodiments, the patient-side system 102 may include one or more carts, each having one or more robotic arms 112, the one or more robotic arms 112 being mounted on a separate structure within an operating room, such as an operating table or ceiling, and / or any other support structure. The patient-side system 102 will be described in more detail below.

[0027] The surgical instruments 114 can each be positioned in a surgical area associated with the patient. As used herein, in some examples, a "surgical area" associated with a patient can be disposed entirely within the patient's body and can include an area within the patient's body near which a surgical procedure is planned to be performed, being performed, or has been performed. For example, for a minimally invasive surgical procedure being performed on tissue outside the patient, the surgical area can include the tissue in which the surgical instruments being used to perform the surgical procedure are located and the space around the tissue. In other examples, the surgical area can be at least partially disposed outside the patient. For example, the robotic surgical system 100 can be used to perform an open surgical procedure so that a portion of the surgical area (e.g., the tissue being operated on) is inside the patient, while another portion of the surgical area (e.g., the space around the tissue in which one or more surgical instruments can be disposed) is outside the patient. When at least a portion of the surgical instrument (e.g., the distal end of the surgical instrument) is disposed within the surgical area, the surgical instrument (e.g., any surgical instrument in the surgical instrument 114) can be referred to as being "located" at the surgical area (or "located" within the surgical area).

[0028] The surgeon console 104 may be configured to facilitate control by the surgeon 110-1 of the robotic arms 112 and surgical instruments 114. For example, the surgeon console 104 may provide the surgeon 110-1 with an image (e.g., a high definition 3D image) of the surgical area associated with the patient 108 as captured by an endoscope. The surgeon 110-1 may utilize the image to perform one or more procedures with the surgical instruments 114.

[0029] To facilitate control of the surgical instruments 114, the surgeon console 104 may include a set of master controls 116 (shown in close-up view 118). The master controls 116 may be manipulated by the surgeon 110-1 to control the movement of the surgical instruments 114. The master controls 116 may be configured to detect a variety of hand movements, wrist movements, and finger movements performed by the surgeon 110-1. In this manner, the surgeon 110-1 may intuitively perform a procedure using one or more of the surgical instruments 114. For example, as shown in close-up view 120, the functional tips of the surgical instruments 114-1 and 114-4, respectively coupled to the robotic arms 112-1 and 112-4, may simulate the dexterity of the surgeon's 110-1 hands, wrists, and fingers in multiple degrees of freedom of motion in order to perform one or more surgical procedures (e.g., an incision procedure, a suturing procedure, etc.). Although the surgeon console 104 is shown and described herein as a single unit for exemplary purposes, in various other embodiments, the surgeon console 104 may include various discrete components, such as a wired or wireless main control 116, (one or more) separate display elements (e.g., a projector or head mounted display), separate data / communication processing hardware / software, and / or any other structural or functional elements of the surgeon console 104. The surgeon console 104 will be described in more detail below.

[0030] The vision cart 106 may be configured to present visual content to surgical team members 110 who do not have access to images provided to surgeon 110-1 at surgeon console 104. To this end, the vision cart 106 may include a display monitor 122 that may be configured to display one or more user interfaces, such as images of the surgical area (e.g., 2D images), information associated with the patient 108 and / or the surgical procedure, and / or any other visual content as a service-specific implementation. For example, the display monitor 122 may display an image of the surgical area and additional content (e.g., graphical content, contextual information, etc.) superimposed on the image or otherwise displayed simultaneously. In some embodiments, the display monitor 122 is implemented by a touch screen display that can interact with the surgical team members 110 (e.g., via touch gestures) to provide user input to the robotic surgery system 100.

[0031] The patient-side system 102, surgeon console 104, and vision cart 106 may be communicatively coupled to one another in any suitable manner. Figure 1As shown in , the patient-side system 102, the surgeon console 104, and the vision cart 106 can be communicatively coupled via a control line 124, which can represent any wired or wireless communication link as can serve a particular embodiment. To this end, the patient-side system 102, the surgeon console 104, and the vision cart 106 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, WiFi network interfaces, cellular interfaces, etc.

[0032] The patient-side system 102, the surgeon console 104, and the vision cart 106 may each include at least one computing device configured to control, direct, and / or facilitate the operation of the robotic surgical system 100. For example, the surgeon console 104 may include a computing device configured to transmit instructions to the patient-side system 102 via one or more of the control lines 124 to control the movement of the robotic arm 112 and / or the surgical instrument 114 in accordance with manipulations performed by the surgeon 110-1 of the master control 116. In some examples, the vision cart 106 may include one or more computing devices configured to perform the primary processing operations of the robotic surgical system 100. In such a configuration, the one or more computing devices included in the vision cart 106 may control and / or coordinate operations performed by various other components of the robotic surgical system 100 (e.g., by the patient-side system 102 and / or the surgeon console 104). For example, the computing device included in the surgeon console 104 may transmit instructions to the patient-side system 102 via one or more computing devices included in the vision cart 106.

[0033] Figure 2 A perspective view of the patient-side system 102 is shown. As shown, the patient-side system 102 includes a cart column 202 supported by a base 204. In some examples, the cart column 202 can include a protective cover 206 that protects components of the weight subsystem and the brake subsystem disposed within the cart column 202 from contamination.

[0034] The cart column 202 may support a plurality of setup arms 208 mounted thereon (e.g., setup arms 208-1 to 208-4). Each setup arm 208 may include a plurality of links and joints that allow for manual positioning of the setup arm 208, and may each be connected to one of the robotic arms 112. Figure 2 In the example of FIG. 1 , the patient-side system 102 includes four setup arms 208 and four robotic arms 112. However, it will be appreciated that the patient-side system 102 may include any other number of setup arms 208 and robotic arms 112 to serve a particular implementation.

[0035] The setup arm 208 can be non-robotically controllable and configured to statically maintain each robotic arm 112 in a respective position required by the setup or reconfiguration of the patient-side system 102. The setup arm 208 can be coupled to the carriage housing 210 and manually moved and positioned during the preoperative, intraoperative, or postoperative phases of a surgical procedure. For example, the setup arm 208 can be moved and positioned during the preoperative phase when the robotic surgical system 100 is being prepared and / or aimed for a surgical procedure to be performed. In contrast, the robotic arms 112 can be mechanically controlled (e.g., as described above, in response to manipulation of the master control 116).

[0036] As shown, each robotic arm 112 can have a surgical instrument 114 coupled thereto. In some examples, three of the four robotic arms 112 can be configured to move and / or position the surgical instruments 114 in the form of surgical tools used to manipulate patient tissue and / or other objects within the surgical field (e.g., suture material, repair material, etc.). Specifically, as shown, robotic arms 112-1, 112-3, and 112-4 can be used to move and / or position surgical instruments 114-1, 114-3, and 114-4, respectively. A fourth robotic arm 112 (e.g., Figure 2 The robotic arm 112 - 2 in the example can be used to move and / or position a monitoring instrument (e.g., a stereo endoscope), as will be described in more detail below.

[0037] The robotic arms 112 may each include one or more displacement transducers, orientation sensors, and / or position sensors (e.g., sensors 212) for generating raw (i.e., uncorrected) motion information to assist in controlling and tracking the surgical instrument 114. For example, the motion information generated by the transducers and sensors in the patient-side system 102, such as the motion-based position or other motion information for the robotic arm 112, may be transmitted to a surgical instrument tracking system (e.g., a computing device included in the vision cart 106) of the robotic surgical system 100. In certain embodiments, each surgical instrument 114 may similarly include a displacement transducer, a position sensor, and / or an orientation sensor (e.g., sensors 214), each of which may provide additional motion information to the tracking system, such as the motion-based position or other motion information for the surgical instrument 114. The tracking system may process the motion information received from the sensors included on the robotic arm 112 and / or the surgical instrument 114 to perform various operations, such as determining the physical position of the robotic arm 112 and / or the surgical instrument 114 as described herein.

[0038] Figure 3 A perspective view of an exemplary robotic arm 112 (e.g., any one of robotic arms 112-1 to 112-4) is shown. As shown, a surgical instrument 114 can be movably coupled to the robotic arm 112. Figure 3 In the example of , surgical instrument 114 is an endoscopic device (e.g., a stereo laparoscope, an arthroscope, a hysteroscope, or another type of stereoscopic or monoscopic endoscope). Alternatively, surgical instrument 114 can be a different type of imaging device (e.g., an ultrasound device, a fluoroscopic device, an MRI device, etc.), a grasping instrument (e.g., a forceps), a needle driver (e.g., a device for suturing), an energy instrument (e.g., a cauterization instrument, a laser instrument, etc.), a retractor, a clip applier, a probe holder, a cardiac stabilizer, or any other suitable instrument or tool.

[0039] In some examples, it may be desirable that the robotic arm 112 and the surgical instrument 114 coupled to the robotic arm 112 move around a single fixed center point 302 so as to limit the motion of the center point 302. For example, the center point 302 may be positioned at or near the insertion point of the surgical instrument 114 into the patient 108. For example, in certain surgical procedures (e.g., surgical procedures associated with laparoscopic surgical procedures), the center point 302 may be aligned with the incision point to an internal surgical site performed by a trocar or cannula at the abdominal wall. As shown, the center point 302 may be located on an insertion axis 304 associated with the surgical instrument 114.

[0040] The robotic arm 112 may include a plurality of links 306 (e.g., links 306-1 to 306-5) pivotally coupled in series at a plurality of joints 308 (e.g., joints 308-1 to 308-4) near respective ends of the links 306. For example, as shown, the link 306-1 is pivotally coupled to the drive mount 310 at a joint 308-1 near a first end of the link 306-1, while being pivotally coupled to the link 306-2 at a joint 308-2 near a second end of the link 306-1. The link 306-3 is pivotally coupled to the link 306-2 near a first end of the link 306-3, while being pivotally coupled to the link 306-4 at a joint 308-4 near a second end of the link 306-3. Typically, as shown, the link 306-4 may be substantially parallel to the insertion axis 304 of the surgical instrument 114. Link 306 - 5 is slidably coupled to link 306 - 4 to allow surgical instrument 114 to be mounted to and slide along link 306 - 5 as shown.

[0041] As described above, the robotic arm 112 can be configured to be mounted to the setup arm 208 (or a joint connected thereto) via the drive mount 310 so as to be supported and held in position by the setup arm 208. The drive mount 310 can be pivotally coupled to the link 306-1 and can include a first internal motor (not explicitly shown) configured to yaw the robotic arm 112 about a yaw axis of the center point 302. In a similar manner, the link 306-2 can house a second internal motor (not explicitly shown) configured to drive and pitch the linkage of the robotic arm 112 about a pitch axis of the center point 302. Similarly, the link 306-4 can include a third internal motor (not explicitly shown) configured to slide the link 306-5 and the surgical instrument 114 along the insertion axis 304. Robotic arm 112 may include a transmission system driven by one or more of these motors to control the pivoting of link 306 about joint 308 in any manner that may serve a particular implementation. Thus, if surgical instrument 114 is to be mechanically moved, one or more of the motors coupled to the transmission system may be energized to move link 306 of robotic arm 112.

[0042] Figure 4 104. As shown, the surgeon's console 104 may include a stereoscopic viewer 402, an arm support 404, a main control 116 ( Figure 4 406, foot pedals 408 and head sensor 410 (not shown).

[0043] In some examples, the stereoscopic viewer 402 has two displays, where a stereoscopic image of a surgical area associated with the patient 108 and generated by the stereoscopic endoscope can be viewed by an operator (e.g., surgeon 110-1) during a surgical procedure. When using the surgeon console 104, the operator can move his or her head to align with the stereoscopic viewer 402 to view the stereoscopic image of the surgical area. To ensure that the operator is viewing the surgical area while controlling the surgical instruments 114 of the patient-side system 102, the surgeon console 104 can use a head sensor 410 placed near the stereoscopic viewer 402. Specifically, when the operator aligns his or her eyes with the binocular eyepieces of the stereoscopic viewer 402 to view the stereoscopic image of the surgical area, the operator's head can activate the head sensor 410, which enables control of the surgical instrument 114 by the master control 116. When the operator's head moves away from the area of ​​the stereoscopic viewer 402, the head sensor 410 can be automatically deactivated, which can prevent control of the surgical instrument 114 by the master control 116. In this manner, when the robotic surgical system 100 detects that the operator is not actively engaged in attempting to control the surgical instrument 114 , the position of the surgical instrument 114 may remain stationary.

[0044] The arm supports 404 may be used to support the operator's elbows and / or forearms while the operator manipulates the main controls 116 to control the robotic arms 112 and / or surgical instruments 114. Additionally, the operator may use his or her feet to control the foot pedals 408. The foot pedals 408 may be configured to change the configuration or operating mode of the robotic surgical system 100, to generate additional control signals for controlling the surgical instruments 114, to facilitate switching control from one surgical instrument 114 to another, or to perform any other suitable operation.

[0045] Figure 5 An exemplary stereoscopic endoscope 500 is shown included within the robotic surgical system 100 and positioned in an exemplary surgical field associated with a patient. The stereoscopic endoscope 500 may be any of the surgical instruments 114 described above.

[0046] As shown, a stereo endoscope 500 may include a tube 502 having a distal tip configured to be inserted into a patient's body and a camera head 504 configured to be located outside the patient. The tube 502 may be coupled to the camera head 504 at a proximal end and may be rigid (e.g., Figure 5 ), seamed and / or flexible, as may serve a particular embodiment.

[0047] The tube 502 may include a plurality of channels 506 (e.g., a right imaging channel 506-R, a left imaging channel 506-L, and an illumination channel 506-I) configured to conduct light between a surgical field within a patient and the camera head 504. Each channel 506 may include one or more optical fibers configured to transmit light along the tube 502, such that light generated within the camera head 504 may be transmitted by the illumination channel 506-I to be output at the distal end of the tube 502, and after reflecting from the patient anatomy and / or other objects within the surgical field, may be transmitted from the distal end of the tube 502 back to the camera head 504 by the imaging channels 506-R and 506-L. Figure 5 The arrows shown within the channels 506 in FIG. 5 are shown to indicate the direction in which light can travel within each channel. Additionally, it will be appreciated that the tube 502 can be associated with (e.g., include) one or more lenses or other suitable optical devices (not explicitly shown) as may serve a particular embodiment for focusing, diffusing, or otherwise processing the light transmitted by the channels 506. In various other embodiments, there can be additional imaging channels and / or illumination channels. In other embodiments, one or more image sensors and / or illuminators can be positioned closer to the distal end of the tube 502, thereby minimizing or even eliminating the need for imaging channels and / or illumination channels through the tube 502.

[0048] In some examples, the stereo endoscope 500 can be coupled to a robotic arm of a robotic surgical system (e.g., one of the robotic arms 112 of the robotic surgical system 100) and positioned such that the distal end of the tube 502 is disposed at a surgical area of ​​the patient. In this configuration, the stereo endoscope 500 can be referred to as being positioned at or within the surgical area, even though a portion of the stereo endoscope 500 (e.g., the camera head 504 and the proximal portion of the tube 502) can be located outside the surgical area. When the stereo endoscope 500 is located at the surgical area, light reflected from the surgical area within the field of view of the stereo endoscope 500 can be captured by the distal tip of the tube 502 and transmitted to the camera head 504 through the imaging channels 506-R and 506-L.

[0049] The camera head 504 may include various components configured to facilitate the operation of the stereo endoscope 500. For example, as shown, the camera head 504 may include an image sensor 508 (e.g., an image sensor 508-R associated with the right imaging channel 506-R and an image sensor 508-L associated with the left imaging channel 506-L). The image sensor 508 may be implemented as any suitable image sensor, such as a charge coupled device ("CCD") image sensor, a complementary metal oxide semiconductor ("CMOS") image sensor, etc. In addition, one or more lenses or other optical devices may be associated with the image sensor 508 (not explicitly shown). The camera head 504 may further include an illuminator 510 that is configured to generate light to travel from the camera head 504 to the surgical field via the imaging channel 506-I so as to illuminate the surgical field.

[0050] The camera head 504 may further include a camera control unit 512 disposed therein. Specifically, the camera control unit 512-R may be communicatively coupled to the image sensor 508-R, and the camera control unit 512-L may be communicatively coupled to the image sensor 508-L. The camera control units 512 may be synchronously coupled to each other via a communication link 514, and may be implemented by software and / or hardware that is configured to control the image sensor 508 so as to generate corresponding images 516 (i.e., image 516-R associated with the right side and image 516-L associated with the left side) based on the light sensed by the image sensor 508. Therefore, each respective combination of the imaging channel 506, the image sensor 508, the camera control unit 512, and the associated optical devices may be collectively referred to as a camera included in the stereo endoscope 500. For example, the stereo endoscope 500 may include two such cameras, one for the left side and one for the right side. It can be said that such cameras are capable of capturing images 516 from a vantage point at the distal end of their respective imaging channels 506. When generated by stereo endoscope 500, image 516 can be accessed by a surgical instrument tracking system and / or used in any other manner described herein. For example, image 516 can be used by a surgical instrument tracking system to determine the position of a robotically manipulated surgical instrument located in a surgical area in any manner described herein.

[0051] Figure 6 An exemplary surgical instrument tracking system 600 ("tracking system 600" or "system 600") configured to track surgical instruments located in a surgical area is shown. As shown, system 600 may include, but is not limited to, an image processing facility 602, an association facility 604, a location determination facility 606, and a storage facility 608 selectively and communicatively coupled to one another. It will be appreciated that while facilities 602 to 608 may be used in a variety of ways, the system 600 may include, but is not limited to, an image processing facility 602, an association facility 604, a location determination facility 606, and a storage facility 608. Figure 6602 to 608 are shown as separate facilities, but facilities 602 to 608 can be combined into fewer facilities, such as a single facility, or can be divided into multiple facilities to serve a specific implementation. Each of facilities 602 to 608 can be implemented by any suitable combination of computing hardware and / or software.

[0052] The system 600 may be associated with a robotic surgical system, such as the robotic surgical system 100, in any suitable manner. For example, the system 600 may be implemented by the robotic surgical system, or may be included within the robotic surgical system. For illustration, the system 600 may be implemented by one or more computing devices included within the patient-side system 102, the surgeon console 104, and / or the vision cart 106 of the robotic surgical system 100. In some examples, the system 600 may be implemented at least in part by one or more computing devices communicatively coupled to, but not included in, the robotic surgical system (e.g., one or more servers communicatively coupled to the robotic surgical system 100 via a network).

[0053] The image processing facility 602 may be configured to access endoscopic images (sometimes referred to as "endoscopic images"), which may include images captured by an endoscope located in a surgical area associated with a patient or a set of one or more images derived from images captured by an endoscope located in a surgical area associated with a patient. The set of one or more endoscopic images may include a 2D image, a 3D image, a 3D image derived from a 2D image, one or more pairs of temporarily aligned stereo images, frames of a video, a sequence of images, an aggregated or combined image (e.g., an image formed by combining multiple images together), or an image in any other suitable form. The endoscopic images may be represented as image data in any suitable data format (e.g., data representing an RGB image having any suitable number of image channels), which may be accessed by the image processing facility 602.

[0054] For example, in some embodiments, the image processing facility 602 can access a first image (e.g., image 516-L) captured from a first vantage point by a first camera included in a stereoscopic endoscope (e.g., stereoscopic endoscope 500) and located in the surgical area and a second image (e.g., image 516-R) captured from a second vantage point that is stereoscopic with the first vantage point by a second camera included in the stereoscopic endoscope. The image processing facility 602 can access the images in any suitable manner. For example, the image processing facility 602 can access the images by receiving the images from the endoscope, or by accessing the images from the storage facility 108 or other computer memory (e.g., a storage buffer).

[0055] Although certain examples are described herein with reference to endoscopic images captured by an endoscope, in other examples, the image processing facility 602 can process any suitable image of a surgical area. For example, the image processing facility 602 can process images of a surgical area captured by one or more non-endoscopic cameras, multiple endoscopes, or a combination of at least one endoscope and at least one non-endoscopic camera.

[0056] The image processing facility 602 may process the endoscopic image to determine observations of an object of interest shown in the endoscopic image. Processing of the endoscopic image may include the image processing facility 602 extracting features representing the object of interest from the endoscopic image. As used herein, an "object of interest" refers to any object or other feature shown in the endoscopic image and is defined as being of interest to the image processing facility 602. In some examples, the object of interest may be represented by any feature shown in the endoscopic image that may be associated with a surgical instrument. For example, a feature may be a surgical instrument shown in the endoscopic image, a component of a surgical instrument shown in the endoscopic image, or a visual representation of any object shown in the endoscopic image that may be accurately or inaccurately determined by the image processing facility 602 to represent the surgical instrument or a component of a surgical instrument.

[0057] In some embodiments, the object of interest may be an unmarked component of a surgical instrument. An unmarked component of a surgical instrument may be a physical component of a surgical instrument that does not include a dedicated computer vision marker (e.g., a barcode, color, etc.). By processing the endoscopic image to extract the unmarked object of interest rather than a dedicated computer vision marker, the image processing facility 602 may overcome one or more problems associated with the use of dedicated computer vision markers, such as the cost of manufacturing the marker, the limited useful life of the marker, the inability to use the marker on all surgical instruments, obstruction of the marker from the camera's view by anatomical tissue, blood, debris, or other instruments, and computational inefficiencies associated with conventional detection of the marker.

[0058] In some embodiments, the object of interest may be a distal U-clip of a surgical instrument. The distal U-clip of a surgical instrument may be the center of the distal-most joint of the surgical instrument, and the position of the distal U-clip may be used to determine the position of the distal tip of the surgical instrument. In some embodiments, extraction of the distal U-clip of a surgical instrument from an endoscopic image may be more accurate, easier, and / or more reliable than extraction of the small distal tip of a surgical instrument from an endoscopic image. In other embodiments, an additional or alternative component of a surgical instrument (e.g., a shaft of a surgical instrument) may be defined as an object of interest.

[0059] Once the image processing facility 602 has extracted the object of interest from the endoscopic image, the image processing facility 602 can define an observation of the object of interest. The observation can indicate the location of the object of interest that has been derived from the endoscopic image. Because the location is derived from the endoscopic image, the location can be referred to as an "image-based location." In some examples, the observation can be defined to represent a location as a 2D area in a 2D space or a 3D area in a 3D space. For example, the observation can indicate a defined pixel area region on a projected 2D image plane (e.g., projected from the vantage point of the endoscopic camera). As another example, the observation can indicate a defined 3D region, such as a 3D region within a 3D space defined by a world 3D coordinate system associated with the robotic surgical system 100.

[0060] In some examples, in addition to indicating the position of an object of interest, the observations defined by the image processing facility 602 (e.g., as outputs of a neural network) can indicate one or more other properties of the object of interest. For example, the observations can indicate (which can be derived from a sequence of graphics, such as a sequence of video frames) motion cues associated with the object of interest (e.g., velocity, acceleration, etc.), the type of surgical instrument associated with the object of interest (e.g., the object of interest is attached to a needle driver, grasper, or cautery-type surgical tool), an additional position associated with the object of interest (e.g., the position of another component of the object of interest), the position of another object of interest relative to the position of the object of interest (e.g., the position of a surgical instrument tip, shaft, etc. relative to a distal clevis of a surgical instrument), and / or an orientation associated with the object of interest (e.g., the orientation of the distal clevis).

[0061] The image processing facility 602 may be configured to use a trained neural network to determine observations of objects of interest. For example, the image processing facility 602 may use a convolutional neural network ("CNN") or another deep feed-forward artificial neural network to extract objects of interest and qualify observations of objects of interest.

[0062] To this end, a neural network can be trained so that the neural network is configured to determine observations of an object of interest shown in an endoscopic image. Training can be performed in any suitable manner, such as by using labeled (e.g., hand-labeled) data as a training set. Such data can include endoscopic images that have been labeled with observations of any object of interest shown in the endoscopic image. For example, a training image can include an image showing an object of interest and a label indicating an image-based location associated with the object of interest. The label can be in any suitable form, such as a bounding box defining a rectangular area (e.g., a pixel area) associated with the object of interest, or any other suitable definition of an area of ​​the image associated with the object of interest. Other images in the training image can be similarly labeled to indicate locations associated with the object of interest. Other images in the training image can be labeled (or unlabeled) to indicate that the object of interest is not shown in the image. In some examples, additional labels can be added to the training image to indicate other attributes of the object of interest, such as a label indicating the type of surgical instrument and / or additional components of the surgical instrument. Examples of other markings that may be used include, but are not limited to, markings indicating key points on the surgical instrument (e.g., the instrument tip, features on the instrument surface such as a logo or other marking on the instrument surface, the instrument shaft, the instrument shaft axis, etc.) and per-pixel segments indicating the instrument or components of the instrument (e.g., the instrument body or portions of the instrument body). Certain of the training images may be selected for use in training the neural network based on differences between the content of the image and the corresponding motion for the surgical instrument, such as not showing an image of a surgical instrument when the corresponding motion indicates that the surgical instrument should be within the view of the endoscope that captured the image.

[0063] After the neural network has been trained, the image processing facility 602 can use the trained neural network to determine observations of objects of interest shown in the endoscopic image. In some examples, the image processing facility 602 can do this by providing the endoscopic image as an input to the trained neural network, which can process the endoscopic image and output a determined observation of the object of interest shown in the image and extracted by the trained neural network. The trained neural network can determine the observation of the object of interest by selecting the object that is most likely or more likely to match the endoscopic image being processed from previously generated (e.g., marked during training of the neural network) observations of historical and / or training endoscopic images. In other examples, instead of outputting the observation of the object of interest, the trained neural network can output data indicative of the extracted object of interest, and the image processing facility 602 can use such output from the trained neural network to qualify or supplement the observation of the object of interest, such as by qualifying a region encompassing the object of interest and / or adding another attribute to the observation. An example of an image processing facility 602 that determines an observation of an object of interest shown in an endoscopic image is shown herein.

[0064] The observation of the object of interest determined by the image processing facility 602 may be considered to be indicative of a candidate image-based location for a surgical instrument, at least because the image processing facility 602 has not yet associated the observation determined and output by the image processing facility 602 with any particular surgical instrument at the surgical area. The image processing facility 602 may provide data representing the access and use of the observation by the association facility 604 to associate the observation with an appropriate surgical instrument determined by the association facility 604 to be represented by the object of interest, or with a false positive designation when the association facility 604 determines that the object of interest does not represent a surgical instrument at the surgical area.

[0065] The association facility 604 can access data representations representing observations of an object of interest from the image processing facility 602, and associate the observations of the object of interest with a specific surgical instrument, or with a false positive designation, based on the movement of one or more surgical instruments in the surgical area. In some embodiments, the association facility 604 can be configured to determine associations made based on a probability framework and movement of one or more surgical instruments in the surgical area. As an example, the association facility 604 can determine the probability of potential associations for the observation of the object of interest with each surgical instrument in the surgical area and with a false positive designation based on the movement of one or more surgical instruments in the surgical area. Then, the association facility 604 can select the highest probability association and associate the observation with the surgical instrument or the false positive designation with the highest probability. As another example, the association facility 604 can determine the probability of each possible one-to-one association between the observation and the surgical instrument or the false positive designation based on the movement of one or more surgical instruments in the surgical area. Then, the association facility 604 can select the one-to-one association of the highest probability group, and associate the observation with the corresponding surgical instrument or the false positive designation according to the selected group of associations. An example of an association facility 604 that associates observations with surgical instruments or false positive designations using a probabilistic framework and motion is described herein.

[0066] The association facility 604 may output data representing associations made by the association facility 604, such as data representing observations and associated surgical instruments. In some examples, the association facility 604 may avoid outputting data representing observations associated with false positive designations.

[0067] Position determination facility 606 can access data output by association facility 604, such as data representing observations and associated surgical instruments. Position determination facility 606 can use the accessed data to determine the physical location of the surgical instrument in the surgical area based on the movement of the surgical instrument and the observations associated with the surgical instrument.

[0068] In some examples, the position determination facility 606 can implement and use a nonlinear estimator, such as an unscented Kalman filter or other Bayesian filter, to determine the physical position of a surgical instrument in the surgical area based on the motion of the surgical instrument and observations associated with the surgical instrument. For example, the position determination facility 606 can input the observations and motions into an unscented Kalman filter, which can use the observations and motions to determine and output the physical position of the surgical instrument in the surgical area. The physical position of the surgical instrument in the surgical area can be represented in any suitable manner, including representing the coordinates of a 3D position of a component of the surgical instrument in a 3D space, such as a world 3D space associated with the robotic surgical system 100 (e.g., the 3D position of a distal clevis or distal tip of the surgical instrument).

[0069] In some embodiments, the position determination facility 606 can be configured to determine the physical position of the surgical instrument based on the image-based position of the object of interest indicated by the observation by correcting the motion-based position of the surgical instrument indicated by the motion. For example, an unscented Kalman filter can be configured to adjust the motion-based position of the surgical instrument based on the image-based position of the object of interest indicated by the observation. The adjustment can be made in any suitable manner, such as by applying a corrective transform to the motion-based position. In some embodiments, the position determination facility 606 can be configured to apply the corrective transform to the motion-based position as quickly as possible, which can be as fast as the image processing facility 602 is able to extract the object of interest from the endoscopic image and determine the observation of the relevant object of interest. By correcting the motion-based position with the observed image-based position, the tracking system 600 is able to use motion when observation is not available, such as when the surgical instrument is not extractable from the endoscopic image (e.g., when the surgical instrument is hidden behind tissue, when there is a lot of smoke in the surgical field, when the surgical instrument is outside the field of view of the endoscope, etc.). When observation of an object of interest representing a surgical instrument is available and used to correct the motion-based position of the surgical instrument, the tracking system 600 is able to track the physical position of the surgical instrument with greater accuracy and / or precision than when motion alone is used without observation. This can help eliminate or reduce errors in motion-based position (e.g., errors in instrument tip position caused by accumulation of errors in engagement encoders along the robotic system kinematic chain), such as rigid errors that can be found when determining absolute position in a global 3D world coordinate system based on motion alone and / or errors that can be caused by large motions of a surgical instrument (which can be mounted on a long kinematic chain).

[0070] The physical location of the surgical instrument determined by the location determination facility 606 can be used to track the surgical instrument at the surgical field during the surgical procedure. By tracking the surgical instrument in this manner, the robotic surgical system 100 can provide one or more features that utilize the tracked physical location of the surgical instrument. Examples of such features are described herein.

[0071] Storage facility 608 may store and maintain any data received, generated, managed, maintained, used, and / or transmitted in a particular implementation by facilities 602 through 606. For example, storage facility 608 may store program instructions (e.g., computer code) for performing the operations described herein, image data, observation data, motion data, position data, and / or any other data as may serve a particular implementation.

[0072] Now refer to Figures 7 to 10 An example of a system 600 for tracking a surgical instrument by determining the physical location of the surgical instrument is described. Figure 7A view 700 of surgical instruments within a surgical field is shown. As shown, the surgical instruments may include an endoscope 702 and one or more other surgical instruments 704 (e.g., surgical instruments 704-1 to 704-3) in the form of one or more surgical tools. Figure 7 One endoscope 702 and three surgical tools are shown located in the surgical field, but any number and / or combination of endoscopes and surgical tools may be in the surgical field during a surgical procedure.Tissue 706 represents anatomical tissue at the surgical field.

[0073] Endoscope 702 can capture endoscopic images at the surgical area. Any surgical instruments 704 and / or tissue 706 within the field of view of endoscope 702 can be shown in the endoscopic images captured by endoscope 702.

[0074] Figure 8 Exemplary images 802 (i.e., images 802-L and 802-R) of a surgical area as captured from a stereoscopic vantage point by a camera included in an endoscope 702, which in this example is a stereo endoscope. Image 802-L and image 802-R may implement image 516-L and image 516-R, respectively, as described above. As shown, each image 802 shows a portion of tissue 706 and surgical instrument 704-1 within the field of view of endoscope 702.

[0075] As shown, images 802 are relatively similar to one another. However, it will be appreciated that due to the stereoscopic nature of the vantage point from which each image 802 is captured, there are slight differences between images 802. Thus, when image 802-L is presented to the left eye of a user (e.g., surgeon 110-1) and image 802-R is presented to the right eye of the user, images 802 may appear to be three-dimensional images.

[0076] As for the images 516, one or more of the images 802 may be presented by the system 600 in any suitable manner and / or on any suitable display screen. For example, one or both of the images 802 may be on the display monitor 122 of the vision cart 106, on the stereoscopic viewer 402 on the surgeon console 104, on the monoscopic display provided by the endoscope 702, and / or on any other display screen associated with the robotic surgery system 100 or the tracking system 600.

[0077] The image processing facility 602 may process one or both of the images 802 to determine observations of any objects of interest shown in the images 802. For example, the image processing facility 602 may provide the images 802 as inputs to a trained neural network, which may extract any objects of interest shown in the images 802 from the images 802. For example, using the trained neural network, the image processing facility 602 may extract an object of interest 804 that represents a distal U-clip of the surgical instrument 704-1 in each of the images 802. The image processing facility 602 may then define observations of the object of interest 804 based on the images 802 and the extracted objects of interest 804. For example, the image processing facility 602 may define observations including a bounding box 806 that represents an image-based location of the distal U-clip in each of the images 802. Although the rectangular bounding box 806 is Figure 8 804 to represent the image-based location indicated by the observation of the object of interest, but any suitable geometric area (e.g., an ellipse), coordinate point, or other position indicator may be used in other examples to represent the image-based location indicated by the observation of the object of interest. For example, a bounding box or other position indicator may be defined to identify any instrument component (e.g., instrument tip, shaft, marker, etc.). In certain other examples, the observation may define the lines that form the instrument "skeleton", or the observation may define the per-pixel segmentation of the instrument piece or components of the instrument (e.g., an image in which pixels representing instrument components and pixels that do not represent the instrument components are represented as different binary values ​​such as black and white color values). In addition, although the distal U-clip is represented in each image in the image 802 by the object of interest 804, any suitable object of interest, such as an additional or additional component of a surgical instrument, may be defined as an object of interest that can be extracted by a trained neural network.

[0078] In some examples, the image processing facility 602 using the trained neural network can extract an object of interest based on features indicating that the object of interest can represent a surgical instrument, even if the object of interest does not actually represent a surgical instrument. For example, using the trained neural network, the image processing facility 602 can extract an object of interest 808 that is actually a portion of the tissue 706 that has features indicating that the object of interest 808 can represent a surgical instrument. The image processing facility 602 can then qualify the observation of the object of interest 808 based on the image 802 and the extracted object of interest 808. For example, as Figure 8As shown in , the image processing facility 602 may define observations including a bounding box 810 that represents the image-based location of an object of interest 808 for each of the images 802 .

[0079] The image processing facility 602 may similarly extract any other object of interest shown in the image 802 (eg, an object of interest representing another surgical instrument if another surgical instrument is within the field of view of the endoscope 702 ) and limit the view to that object of interest.

[0080] The observations of the object of interest determined by the image processing facility 602 can represent the image-based position of the object of interest in any suitable manner. For example, the observations can represent a 2D area within the image (e.g., a pixel area or other 2D area on a projection plane projected from a viewpoint such as the viewpoint of the endoscope 702). As another example, the observations can represent a 3D area within a 3D space represented by a world 3D coordinate system, which can be derived from an identified 2D area within the endoscopic image (e.g., derived from a 2D area in a stereoscopic endoscopic image). As another example, the observations can be a single coordinate position or a set of one or more coordinate positions in a 2D area or 3D space. Examples of such observations are described in more detail herein.

[0081] In some examples, image processing facility 602 may not have information about surgical instrument 704 or the motion of surgical instrument 704, and may determine the observation based solely on the endoscopic image and using a trained neural network (e.g., by inputting the endoscopic image to the trained neural network, which outputs the observation). Therefore, image processing facility 602 may not determine whether the observation represents a surgical instrument or a false positive designation, or the specific surgical instrument represented by the observation.

[0082] The association facility 604 can associate each determined observation with a specific one of the surgical instruments 704 or a false positive designation based on the probabilistic frame and movement of one or more of the surgical instruments in the surgical field (e.g., movement of the endoscope 702 and / or one or more of the surgical instruments 704). In some embodiments, this can include the association facility 604 for the observations determined by the image processing facility 602, the association facility 604 determining the probability of each potential association and selecting the potential association with the highest probability.

[0083] As an example, association facility 604 may determine a probability that bounding box 806 represents surgical instrument 704-1, a probability that bounding box 806 represents surgical instrument 704-2, a probability that bounding box 806 represents surgical instrument 704-3, and a probability that bounding box 806 represents a false positive based on one or more suitable probability algorithms that take into account the motion of one or more surgical instruments in the surgical area (e.g., the 3D motion-based position indicated by the motion). Association facility 604 may then select the highest probability determined and associate bounding box 806 with surgical instrument 704 or the false positive designation having the highest probability. Figure 8 In the scenario shown in , the association facility 604 can determine that the association of bounding box 806 with surgical instrument 704-1 has the highest probability based at least in part on a close distance relationship between the motion-based position of surgical instrument 704-1 (as indicated by the motion) and the observed image-based position represented by bounding box 806 (e.g., compared to the more distant relationship between the observed image-based position represented by bounding box 806 and the motion-based positions of surgical instruments 704-2 and 704-3).

[0084] Fig. 9 An exemplary probabilistic framework 900 is shown that may be used by the association facility 604 in the above example to associate the bounding box 806 with the surgical instrument 704-1. In the illustrated probabilistic framework 900, the association facility 604 determines a probability P1 for a potential association 902-1 of the bounding box 806 with the surgical instrument 704-1, a probability P2 for a potential association 902-2 of the bounding box 806 with the surgical instrument 704-2, a probability P3 for a potential association 902-3 of the bounding box 806 with the surgical instrument 704-3, and a probability P4 for a potential association 902-1 of the bounding box 806 with the false positive designation 904. Fig. 9 , box 906 contains the determined probabilities P1, P2, P3, and P4 for the corresponding potential associations. The association facility 604 can select the highest probability value from P1, P2, P3, and P4, and define the association of the bounding box 806 with the specific surgical instrument 704 or the false positive designation 904 based on the potential association with the highest probability.

[0085] As another example of using such a probabilistic framework, association facility 604 may determine a probability that bounding box 810 represents surgical instrument 704-1, a probability that bounding box 810 represents surgical instrument 704-2, a probability that bounding box 810 represents surgical instrument 704-3, and a probability that bounding box 810 represents a false positive based on one or more suitable probabilistic algorithms that take into account the movement of one or more surgical instruments in the surgical area. Association facility 604 may then select the highest determined probability and associate bounding box 810 with surgical instrument 704 or the false positive with the highest probability. Figure 8In the scenario shown in , association facility 604 can determine that bounding box 810 has the highest probability of being associated with a false positive designation based at least in part on the lack of a close distance relationship between the observed image-based position represented by bounding box 810 and the motion-based position of surgical instrument 704.

[0086] In certain embodiments, the association facility 604 may determine and make a highest probability-based association, respectively, for each observation determined by the image processing facility 602, as in the example described above. In making such associations for an observation, the association facility 604 may determine the probability of a potential association for the observation without regard to other observations determined from the endoscopic image. Thus, the association of an observation with a surgical instrument or a false-positive designation may be made independently of the association of any other observation determined from the endoscopic image. For example, an observation represented by bounding box 806 may be associated with surgical instrument 704 or a false-positive designation independently of the observation represented by bounding box 810 and / or independently of the association of the observation represented by bounding box 810 with a surgical instrument or a false-positive designation.

[0087] In other embodiments, when determining the highest probability-based association of an observation, the association facility 604 may consider one or more other observations and / or associations of such observations. Such other observations may include observations determined from the same endoscopic image (e.g., from the same endoscopic image), observations determined from another endoscopic image such as a stereoscopic image, and / or observations determined from an earlier endoscopic image such as a previous video frame.

[0088] In certain embodiments, the association facility 604 may be configured to determine and make the highest probability-based association by collectively considering all observations determined from the endoscopic image by the image processing facility 602. For example, as part of determining the probability of a potential association of observations determined from the endoscopic image by the image processing facility 602, the association facility 604 may determine and consider the probability of a potential association of one or more observations determined from the endoscopic image by the image processing facility 602. The association facility 604 may consider the probabilities collectively to determine the highest probability association of the observations.

[0089] In some examples, the association facility 604 can collectively determine the probability of a potential association by determining a probability for each potential group of associations between an observation and a surgical instrument or a false positive. Figure 8, the association facility 604 can determine the probability of association of each potential group of bounding boxes 806 and 810 with surgical instrument 704 or false positives, and select the set of associations with the highest probability. For example, the association facility 604 can determine that a set of associations in which bounding box 806 is associated with surgical instrument 704-1 and bounding box 810 is associated with a false positive designation has a higher probability (as a group) than each of the other potential groups of associations.

[0090] Fig.10 An exemplary probability framework 1000 is shown that can be used to associate bounding box 806 with surgical instrument 704-1 and associate bounding box 810 with false positive designation in the above example. Based on the probability framework 1000 shown, the association facility 604 can determine the probability PR1 for the first group of potential associations 1002-1 and the probability PR2 for the second group of potential associations 1002-2. As shown, the first group of potential associations 1002-1 includes the potential association 1004-1 of bounding box 806 with surgical instrument 704-1 and the potential association 1004-2 of bounding box 810 with surgical instrument 704-2. The first group of potential associations 1002-1 includes no potential association with surgical instrument 704-3 or no potential association with false positive designation. As also shown, the second group of potential associations 1002-2 includes the potential association 1006-1 of bounding box 806 with surgical instrument 704-1 and the potential association 1006-2 of bounding box 810 with false positive designation 1008. The second set of potential associations 1002-2 includes no potential associations with surgical instruments 704-2 and 704-3. The first set of potential associations 1002-1 and the second set of potential associations 1002-2 represent two possible sets of potential one-to-one associations between bounding boxes 806 and 810 and surgical instruments 704-1, 704-2, and 704-3 or false positive designations. The association facility 604 can determine a probability for each possible set of potential associations, including the probability for the first set of potential associations 1002-1 and the second set of potential associations 1002-2.

[0091] exist Fig.10, for illustrative purposes, box 1010 contains only the determined probabilities PR1 and PR2 for the respective groups of potential associations 1002-1 and 1002-2. The association facility 604 may select the highest probability values ​​from PR1 and PR2 (and the probabilities for any other groups of potential associations), and define the association of bounding box 806 and bounding box 810 with the respective surgical instruments 704 or false positive designations 1008 according to the set of potential associations with the highest probability. For example, the association facility 604 may select the second set of potential associations 1002-2 with the highest probability based on the probabilistic frames and movements of one or more surgical instruments 704-1, 704-2, and 704-3, and may associate bounding box 806 with surgical instrument 704-1 and bounding box 810 with false positive designations 1008.

[0092] The above examples of probabilistic frameworks are illustrative only. The association facility 604 may implement any suitable probabilistic algorithm and / or probabilistic framework to determine the probabilities of potential associations and / or the probabilities of groups of potential associations. The probabilistic algorithm may be configured to take into account the motion-based position of one or more surgical instruments in the probability determination. In some embodiments, given a potential association or groups of potential associations, the probability of each potential association or the probability of each group of potential associations may be conditionally independent, and an appropriate probabilistic algorithm for such conditional independence may be used as the probabilistic framework performed by the association facility 604.

[0093] The association facility 604 may use any suitable motion information to determine the probability of observing a potential association with a surgical instrument or a false positive. For example, in some embodiments, the association facility 604 may use the motion and / or other properties of the endoscope 702 (e.g., orientation, field of view, lens properties, etc.) to determine the probability of a potential association. For example, the association facility 604 may use the motion-based position and orientation of the endoscope 702 and the motion-based position and / or orientation of the surgical instrument 704 (e.g., the position and / or orientation of specific instrument components such as the instrument shaft, tip, clevis, etc.) to determine the probability that the surgical instrument 704 is within the field of view of the endoscope 702, which probability may be used by the association facility 604 to determine the probability that the bounding box 806 corresponds to the surgical instrument 704 or a false positive and / or the probability that the bounding box 810 corresponds to the surgical instrument 704 or a false positive.

[0094] The association facility 604 can be configured to use any suitable information about the surgical instrument to determine the probability of a potential association of an observation with a surgical instrument or a false positive. For example, the association facility 604 can use information about the type of surgical instrument indicated by the motion information, information about the type of surgical instrument observed indicated by the observation, and / or information about the position and / or orientation of a surgical instrument component (e.g., a surgical instrument shaft, tip, clevis, etc.) indicated by the motion information and / or the observation.

[0095] The probabilistic framework used by the association facility 604 may include an observation model that defines a conditional probability function to be used by the association facility 604 to determine the probability that an observation represents a surgical instrument or a false positive based on the current state of the robotic surgical system (e.g., based on the motion of the surgical instrument and / or any other state information for the robotic surgical system).

[0096] In certain alternative embodiments, the association facility 604 can be configured to associate observations with surgical instruments or false positive designations based on the movement of surgical instruments in the surgical area without using a probabilistic framework. For example, the association facility 604 can be configured to associate observations with closer surgical instruments within a predefined distance threshold, or to associate observations with false positive designations when no surgical instrument is within the predefined distance threshold.

[0097] The association facility 604 may define or otherwise establish the association in any suitable manner, such as by associating data representing the observation with data representing the corresponding surgical instrument or a false-positive designation. The association facility 604 may output data representing the determined association between the observation and the surgical instrument. In some examples, the association facility 604 may refrain from outputting data representing the determined association between the observation and the false-positive designation, such that the location determination facility 606 accesses and uses only the observations that the association facility 604 has associated with the surgical instrument located in the surgical area.

[0098] The location determination facility 606 can be configured to access and use data representing observations and associated surgical instruments to determine the physical location of the surgical instrument in the surgical area based on the observations and motions for the associated surgical instruments. For example, the location determination facility 606 can determine the physical location of surgical instrument 704-1 in the surgical area based on the bounding box 806 and motions for surgical instrument 704-1. Because the association facility 604 has defined the associations between the observations and the surgical instruments, the location determination facility 606 is able to select and use the motions of the appropriate corresponding surgical instruments for the observations in conjunction with the observations to determine the physical location of the surgical instrument.

[0099] In some examples, the position determination facility 606 can implement a nonlinear estimator, such as an unscented Kalman filter, a particle filter, or other Bayesian filter, to determine the physical position of the surgical instrument based on the motion of the surgical instrument and one or more observations associated with the surgical instrument. The position determination facility 606 can do this by providing the motion of the surgical instrument and the one or more observations as inputs to the nonlinear estimator, which can process the inputs and generate and output the determined physical position of the surgical instrument.

[0100] The physical location of the surgical instrument as determined by the location determination facility 606 may represent any suitable component of the surgical instrument (e.g., a distal clevis of the surgical instrument or the instrument tip). In some examples, the physical location of the surgical instrument as determined by the location determination facility 606 may indicate a 3D location within a world 3D space for a robotic surgical system, such as the robotic surgical system 100. The physical location may be represented in any suitable manner and / or using any suitable data format.

[0101] Figures 11 to 14 An exemplary implementation of a tracking system 600 including a trained neural network, a probabilistic framework, and a filter is shown. In such an implementation, the image processing facility 602 may implement the trained neural network, the association facility 604 may implement the probabilistic framework, and the position determination facility 606 may implement the filter, which may be an unscented Kalman filter or other Bayesian filter. Figures 11 to 14 The embodiments shown in can be configured to use various forms of endoscopic images and observations to determine the physical location of surgical instruments. Figures 11 to 14 The data flow associated with an exemplary implementation of tracking system 600 is shown.

[0102] Fig.11 An exemplary implementation 1100 is shown that includes a trained neural network 1102, a probabilistic framework 1104, and a filter 1106. As shown, an endoscopic image 1108 of a surgical area can be input to the neural network 1102, which can determine observations 1110 based on the endoscopic image 1108 in any manner described herein. The observations 1110 and the motions 1112 of one or more surgical instruments can then be input to the probabilistic framework 1104, which can be used to associate the observations 1110 with a particular surgical instrument or a false positive designation based on the motions 1112. For example, the association facility 604 can determine the probabilities of potential associations for the observations 1110 based on the motions 1112 and using the probabilistic framework 1104 and select the association with the highest probability.

[0103] When observation 1110 is associated with a particular surgical instrument using probabilistic framework 1104, data 1114 representing observation 1110 and the associated surgical instrument may be output for use as input to filter 1106. Filter 1106 may receive and use motion 1116 for the associated surgical instrument and data 1114 representing observation 1110 associated with the surgical instrument to determine and output a physical location 1118 of the surgical instrument in the surgical area.

[0104] Fig.12An exemplary implementation 1200 including a trained neural network 1202, a probabilistic framework 1204, and a filter 1206 is shown. In the example shown, an endoscopic image may include an endoscopic image 1208, such as a left endoscopic image 1208-L and a right endoscopic image 1208-R (which may be a pair of 2D images that are stereoscopic to each other), which may be input to the neural network 1202. The neural network 1202 may determine observations 1210 based on the endoscopic image 1208 in any manner described herein. As shown, the observations 1210 may include a left observation 1212-L and a right observation 1212-R (collectively, "observations 1212"). The neural network 1202 may determine the left observation 1212-L based on the left endoscopic image 1208-L, and determine the right observation 1212-R based on the right endoscopic image 1208-R. Neural network 1202 may be configured to process left endoscopic image 1208-L and right endoscopic image 1208-R independently at once or in parallel to determine left observation 1212-L and right observation 1212-R.

[0105] Observations 1212 and movements 1214 of one or more surgical instruments may be input to a probabilistic framework 1204, which may be used to associate each of observations 1212 with a particular surgical instrument or a false-positive designation based on the movements 1214. For example, association facility 604 may determine a probability of a potential association for each of observations 1212 based on the movements 1214 and using probabilistic framework 1204, and for each of observations 1212, select a potential association with a highest probability.

[0106] When the observations 1212 are each associated with a particular surgical instrument using the probability framework 1204, data 1216 representing the observations 1212 and the associated surgical instruments may be output for use as input to the filter 1206. For example, the data 1216 may represent a left observation and associated surgical instrument and a right observation and associated surgical instrument, which may be output for use as input to the filter 1206.

[0107] Filter 1206 can receive and use the motion 1218 for the associated surgical instrument and the data representing the observations 1212 associated with the surgical instrument to determine and output a physical position 1220 of the surgical instrument in the surgical area. Filter 1206 can determine the physical position 1220 of the surgical instrument based on the observations 1212 and the motion 1218 for the associated surgical instrument in any suitable manner, including by using a pair of observations 1212-L and 1212-R as separate and independent input models to filter 1206 to correct for the motion 1218 of the associated surgical instrument (e.g., a position based on the motion). This may include using observations 1212-L and 1212-R as stereo images and performing one or more stereo pairing operations on observations 1212-L and 1212-R to determine the depth of the surface shown in observations 1212-L and 1212-R and generate a 3D image-based position (e.g., 3D coordinates or a volume of a 3D box, sphere, ellipsoid, etc.) of the surgical instrument as represented by observations 1212-L and 1212-R. Filter 1206 may use the generated image-based 3D image-based position to correct the motion-based 3D position of the surgical instrument indicated by motion 1218 to determine physical position 1220 in world 3D space.

[0108] In the example shown, left endoscopic image 1208-L can be processed by neural network 1202 independently of right endoscopic image 1208-R to determine left observation 1212-L, which can be processed independently of right observation 1212-R to associate a surgical instrument with left observation 1212-L. Similarly, right endoscopic image 1208-R can be processed independently of left endoscopic image 1208-R to determine right observation 1212-R, which can be processed independently of left observation 1212-L to associate a surgical instrument with right observation 1212-R. Left observation 1212-L and right observation 1212-R can be input to filter 1206 (as independent observations) and used by filter 1206 to determine physical location 1220 based on observation 1212 and motion 1218.

[0109] Such independent processing of 2D endoscopic images can save computing resources by allowing certain operations of the tracking system 600 to be performed using 2D data rather than 3D data to determine the estimated 3D physical location 1220 of the surgical instrument. In addition, the filter 1206 can determine the physical location 1220 even when one of the two observations 1212 is not available or does not show the surgical instrument. For example, if a surgical instrument is represented in the left endoscopic image 1208-L but not in the right endoscopic image 1208-R (e.g., because the surgical instrument is blocked from the view of the right camera by tissue), the left endoscopic image 1208-L and the right endoscopic image 1208-R can be processed by the neural network 1202 to determine the observations 1212-L and 1212-R associated with the surgical instrument or a false positive designation and input to the filter 1206. Filter 1206 can process observations 1212-L and 1212-R as input to generate a physical location 1220 of a surgical instrument, even if right observation 1212-R is not associated with a surgical instrument. Alternatively, when right observation 1212-R is not associated with a surgical instrument by association facility 604 (e.g., because the surgical instrument is within the field of view of the left camera field of view but is blocked from the right camera field of view by tissue), filter 1206 can use only left observation 1212-L and not right observation 1212-R as input. The accuracy of physical location 1220 determined by filter 1206 of a surgical instrument can be maintained for at least some time, even when the surgical instrument is only temporarily shown in one of endoscopic images 1208.

[0110] Fig.12 An embodiment 1200 is shown in which a pair of stereo images 1208-L and 1208-R are input into a neural network 1202, which determines a pair of independent observations 1212-L and 1212-R based on the corresponding stereo images 1208-L and 1208-R. In other embodiments, the tracking system 600 can be configured to process other forms of endoscopic images and / or determine other observations in other formats and use the other observations in other formats to determine the physical position of the surgical instrument. For example, other embodiments of the system 600 can be configured to process a combined endoscopic image, which can include multiple endoscopic images combined into (stacked into) a concentrated endoscopic image, such as a pair of stereoscopic endoscopic images stacked into a combined image, or a sequence of multiple endoscopic images (e.g., a sequence of two, three, or more video frames) stacked into a combined image. Additionally or alternatively, other embodiments of the system 600 can be configured to determine observations indicating a 3D image-based position of an object of interest shown in the endoscopic image.

[0111] Fig.13An exemplary implementation 1300 including a trained neural network 1302, a probabilistic framework 1304, and a filter 1306 is shown. In the example shown, the endoscopic image may include a left 2D endoscopic image 1308-L and a right 2D endoscopic image 1308-R that are stereoscopic with respect to each other. The image processing facility 602 may combine the endoscopic images 1308-L and 1308-R to form a combined endoscopic image 1308-C. This may be performed in any suitable manner, such as by stacking the endoscopic images 1308-L and 1308-R by the image processing facility 602 to form the combined endoscopic image 1308-C. For example, if the endoscopic images 1308-L and 1308-R each include a 3-channel RGB image, the image processing facility 602 may stack the 3-channel RGB images to form a 6-channel combined image.

[0112] The image processing facility 602 may input the combined endoscopic image 1308-C into the neural network 1302, which may be suitably configured to determine the observation 1310 based on the combined endoscopic image 1308-C in any of the methods described herein. In some examples, the neural network 1302 may be configured to determine the observation 1310 such that the observation 1310 indicates a 3D image-based position in 3D space of an object of interest shown in the combined endoscopic image 1308-C (e.g., by performing a stereo pairing operation to determine a depth value for a surface of the object of interest from the combined endoscopic image 1308-C representing a combined stereo image showing the object of interest).

[0113] In an alternative embodiment, image processing facility 602 may receive endoscopic images 1308-L and 1308-R instead of combined endoscopic image 1308-C and perform a stereo pairing operation on endoscopic images 1308-L and 1308-R to determine depth values ​​for the surface of the object of interest from endoscopic images 1308-L and 1308-R. Using the determined depth values, neural network 1302 may determine observation 1310 such that observation 1310 indicates a 3D image-based position of the object of interest in 3D space.

[0114] The observation 1310 and the motion 1312 of one or more surgical instruments may then be input to a probability framework 1304, which may be used to associate the observation 1310 with a particular surgical instrument or a false positive designation based on the motion 1312, such as by the association facility 604 determining probabilities of potential associations for the observation 1310 based on the motion 1312 and using the probability framework 1304 and selecting the association with the highest probability. Because the observation 1310 indicates a 3D image-based position of an object of interest in 3D space, the association facility 604 may use the 3D spatial data to perform certain operations. For example, the association facility 604 may analyze the 3D image-based position of the object of interest in 3D space relative to one or more 3D motion-based positions of the one or more surgical instruments in 3D space, the 3D motion-based position indicated by the motion 1312. The association facility 604 may use the results of the analysis (e.g., the determined distance) to determine the probability of a potential association for the observation 1310.

[0115] When an observation 1310 is associated with a particular surgical instrument using the probabilistic framework 1304, data 1314 representing the observation 1310 and the associated surgical instrument can be output for use as an input to a filter 1306. The filter 1306 can access and use the motion 1316 for the associated surgical instrument and the data 1314 representing the observation 1310 associated with the surgical instrument to determine and output a physical location 1318 of the surgical instrument in the surgical area. Because the observation 1310 indicates a 3D image-based location of an object of interest in 3D space, the filter 1306 can use the 3D spatial data to perform certain operations. For example, the filter 1306 can determine the physical location 1318 of the surgical instrument by adjusting the 3D motion-based location of the surgical instrument as indicated by the motion 1316 with the 3D image-based location of the object of interest as indicated by the observation 1310 to determine the 3D physical location of the surgical instrument in a 3D space, such as a global 3D space associated with the robotic surgical system.

[0116] By performing certain operations in 3D space, embodiment 1300 can determine a highly accurate physical position 1318 of a surgical instrument, particularly relative to the depth of the surgical instrument. Additionally or alternatively, by using the 3D image-based position indicated by observation 1310, embodiment 1300 can minimize the number of translations from 2D space to 3D space that are performed to determine the physical position 1318 of the surgical instrument.

[0117] Fig.14An exemplary implementation 1400 including a trained neural network 1402, a probabilistic framework 1404, and a filter 1406 is shown. In the example shown, the endoscopic image may include a sequence of endoscopic images captured over time (e.g., a sequence of video frames), such as endoscopic images 1408-T1, 1408-T2, and 1408-T3 captured at times T1, T2, and T3. The image processing facility 602 may combine the endoscopic images 1408-T1, 1408-T2, and 1408-T3 to form a combined endoscopic image 1408-C. This may be performed in any suitable manner, such as by stacking the endoscopic images 1408-T1, 1408-T2, and 1408-T3 by the image processing facility 602 to form the combined endoscopic image 1408-C. For example, if the endoscopic images 1408 - T1 , 1408 - T2 , and 1408 - T3 each include a 3-channel RGB image, the image processing facility 602 may stack the 3-channel RGB images to form a 9-channel combined image.

[0118] The image processing facility 602 may input the combined endoscopic image 1408-C into the neural network 1402, which may be suitably configured to determine an observation 1410 based on the combined endoscopic image 1408-C in any manner described herein. In certain examples, the neural network 1402 may be configured to determine the observation 1410 such that the observation 1410 indicates an image-based position of an object of interest shown in the combined endoscopic image 1408-C. The neural network 1402 may be configured to identify a motion (e.g., velocity, acceleration, etc.) of the object of interest from the combined endoscopic image 1408-C representing a sequence of endoscopic images captured at different times, and determine the image-based position of the object of interest using the identified motion. In some examples, this may include using the identified motion to determine a depth point for a surface of the object of interest, as well as determining a 3D image-based position of the object of interest, which may be particularly useful for estimating depth from a sequence of 2D images captured from the same vantage point (e.g., when a surgical instrument is within the field of view of only one camera). The use of motion-based cues may aid in the accuracy of the image-based position of the object of interest. The image-based location of the object of interest may include a 2D image or a 3D image based location, such as the locations described herein.

[0119] The observations 1410 and movements 1412 of one or more surgical instruments can then be input into a probability framework 1404, which can be used to associate the observation 1410 with a particular surgical instrument or a false positive designation based on the movement 1412, such as by the association facility 604, and determining probabilities of potential associations for the observation 1410 based on the movement 1412 using the probability framework 1404 and selecting the association with the highest probability.

[0120] When observation 1410 is associated with a particular surgical instrument using probabilistic framework 1404, data 1414 representing observation 1410 and the associated surgical instrument may be output for use as input to filter 1406. Filter 1406 may access and use motion 1416 for the associated surgical instrument and data 1414 representing observation 1410 associated with the surgical instrument to determine and output a physical location 1418 of the surgical instrument in the surgical area.

[0121] Although Figures 11 to 14 An exemplary embodiment of a tracking system 600 configured to process various forms of endoscopic images and / or determine various observations is shown, and this example is illustrative. Other embodiments may be configured to process other forms of images (e.g., other forms of endoscopic images and / or non-endoscopic images), and / or determine other suitable observations. To this end, the neural network may be appropriately configured to suit a particular embodiment.

[0122] By tracking a surgical instrument using a combination of image-based observation and motion of the surgical instrument as described herein, certain technical benefits can be provided by the tracking system 600. For example, the accuracy, precision, efficiency, and / or reliability of tracking a surgical instrument can be improved compared to using only image-based tracking or only motion-based tracking. For example, a motion-based position indicated by the motion of the surgical instrument can be used to determine the physical position of the surgical instrument to a certain level of accuracy (e.g., within a reasonable centimeter). By calibrating the motion-based position of the surgical instrument with the image-based position indicated by observation as described herein, the physical position of the surgical instrument can be determined and can have an improved level of accuracy (e.g., within a reasonable centimeter).

[0123] Fig.15 Examples of such improved levels of accuracy of the determined physical position of a surgical instrument are shown. Fig.15An endoscopic image 1500 of a surgical area is shown. Endoscopic image 1500 shows a surgical instrument 1502 located in the surgical area. To illustrate the level of accuracy associated with the physical position of surgical instrument 1502 determined based solely on motion, an indicator 1504 (e.g., an ellipse representing an ellipsoid) is overlaid on endoscopic image 1500, and the level of accuracy for the motion-based position of surgical instrument 1502 is indicated by the size of indicator 1504 and / or the area covered by indicator 1504. To illustrate the improved level of accuracy associated with the physical position of surgical instrument 1502 determined based on both motion and one or more image-based observations, an indicator 1506 (e.g., a circle representing a sphere) is overlaid on endoscopic image 1500, and the level of accuracy for the physical position of surgical instrument 1502 determined based on both motion and one or more image-based observations as described herein is indicated by the size of indicator 1506 and / or the area covered by indicator 1506.

[0124] In some embodiments, the tracking system 600 can be configured to selectively capture and use endoscopic images to track surgical instruments. For example, the tracking system 600 can be configured to use motion for surgical instruments to select tracking surgical instruments when capturing and using endoscopic images. For illustration, the tracking system 600 can selectively use endoscopic images to track the surgical instrument in response to the motion of the surgical instrument as indicated by the motion for the surgical instrument and / or in response to changes in the orientation, angle, position, or other properties of the endoscope as indicated by the motion for the endoscope. Thus, the tracking system 600 can be configured to switch between tracking the surgical instrument using motion alone without image-based observation and tracking the surgical instrument using motion and image-based observation as described herein. Such embodiments can save resources (e.g., computing resources, energy, etc.) of the tracking system 600 and / or the robotic surgical system.

[0125] Tracking of surgical instruments as described herein can provide one or more benefits of a robotic surgical system. For example, tracking of surgical instruments as described herein can facilitate one or more features of a robotic surgical system. In certain embodiments, the level of accuracy of the physical position of a surgical instrument determined as described herein can facilitate certain features of a robotic surgical system that require a high level of accuracy.

[0126] As an example, tracking of surgical instruments as described herein can be used by a robotic surgical system to place certain user interface content (e.g., images, icons, indicators, etc.) on a display screen relative to the physical location of the surgical instrument or relative to the physical location of specific tissues that have been touched by the surgical instrument. To illustrate, a robotic surgical system can juxtapose user interface content with a visual representation of a surgical instrument on a display screen in a manner that positions the user interface content close to the tip of the surgical instrument without obstructing the surgical instrument from the view of the surgeon or other surgical team members. For example, an indicator can be placed to indicate an operating mode (e.g., a high energy mode for cauterization) to the surgeon in a manner that makes the indicator easily visible to the surgeon without requiring the surgeon to look away from the vicinity of the surgical instrument, or an indicator can be placed to indicate specific tissues that have been touched by a probe to help the surgeon see which tissues have been analyzed by the probe. Such close placement of user interface content that can provide benefits to the surgical team and, therefore, the patient can be facilitated by the precise determination of the physical location of the surgical instrument, as described herein.

[0127] As other examples, tracking of surgical instruments as described herein can be used by a robotic surgical system to warn or prevent collisions of surgical instruments and / or robotic arms, automate the movement of surgical instruments so that the robotic surgical system can control certain movements of surgical instruments (e.g., by automating the movement of probes such as cancer probes to systematically touch surface points of tissue, such as automating the movement of an endoscope based on the movement of surgical tools, etc.), recognize that certain tasks are being performed during a surgical procedure (e.g., by recognizing when defined segments of a surgical procedure are being performed), correct for inaccuracies in the movement of surgical instruments, such as inaccuracies caused by long kinematic chains and / or flexibility of a flexible shaft of a surgical instrument (e.g., a flexible shaft used in a single-port access system), and calibrate surgical instruments and / or robotic arms (e.g., by moving the surgical instrument into the field of view of an endoscope and using information extracted from one or more images of the scene to calibrate the surgical instrument), which can improve the workflow for calibration compared to conventional calibration techniques. These examples are illustrative only. Tracking of surgical instruments as described herein can be used by a robotic surgical system in any other suitable manner.

[0128] In some examples, the endoscopic image can be used (e.g., by the tracking system 600) for camera arm registration between a camera (e.g., a camera of an endoscope) and a robotic arm mounted on a separate patient-side system (e.g., a separate patient-side cart). In embodiments where the endoscope and surgical instruments are mounted to the same patient-side cart's robotic arm, such as those described above, the kinematic chain connecting the endoscope and surgical instruments is known, and the motion of the endoscope and surgical instruments based on the known kinematic chain can be accessed and used by the tracking system 600 to track instrument positions as described herein. However, in other embodiments where the endoscope and surgical instruments are mounted to separate, unconnected patient-side cart's robotic arms, there is no known complete kinematic chain connecting the endoscope and surgical instruments because the links in the kinematic chain between the patient-side carts are unknown. That is, the physical relationship between the patient-side carts is unknown and therefore not accessible by the tracking system 600. In such embodiments with separate patient side carts, the tracking system 600 can be configured to use the endoscopic images captured by the endoscope to perform a registration process to register the surgical instruments mounted on one patient side cart with the endoscope mounted on the other patient side cart. The tracking system 600 can be configured to perform the registration process once and / or periodically or continuously after an initial registration (e.g., as part of a calibration, setup process, or other initial registration) to refine the initial registration (e.g., to account for changes in physical position, which may have various motion errors) and / or to account for physical adjustments to the patient side cart position (e.g., intraoperative adjustments of the patient side cart).

[0129] To perform this registration process, the tracking system 600 can determine the position of the surgical instrument (e.g., the position of the surgical instrument axis in the left endoscopic image and the right endoscopic image) based on the endoscopic image of the surgical area and using the trained neural network. In some examples, this can include the tracking system 600 using the trained neural network to determine the observation of the object of interest shown in the endoscopic image, associating the observation of the object of interest with the surgical instrument in any suitable manner (e.g., object recognition), and determining the position of the surgical instrument (e.g., based on the position of the image).

[0130] The tracking system 600 can then perform a constrained optimization to fit the model of the surgical instrument (e.g., the model of the axis of the surgical instrument) to the determined position of the surgical instrument (e.g., to the position of the axis of the surgical instrument in the left endoscopic image and the right endoscopic image). The tracking system 600 can be configured for constrained optimization to search only for solutions that are rotations and translations in the plane of the floor on which the patient cart is placed. This constrained optimization can provide faster and / or more accurate results than conventional optimization performed in all six degrees of freedom.

[0131] Tracking system 600 can perform the registration process for one surgical instrument at a time or for multiple instruments simultaneously. If a single surgical instrument is mounted on a single patient side cart and / or mounted or registered individually, the association of observations with the surgical instrument can be simple. If two or more surgical instruments are each mounted on separate patient side carts and / or mounted or registered simultaneously, tracking system 600 can be configured to associate observations determined from endoscopic images with the surgical instruments in any suitable manner, including by using a probabilistic framework as described herein.

[0132] The registration process can be used by the tracking system 600 to determine the missing link in the kinematic chain that connects the endoscope and surgical instruments mounted on separate patient side carts. The tracking system 600 can represent the missing link as a transformation that defines an estimated physical relationship between the patient side carts. The transformation can define a rotation and translation that can be applied to transform data points from a reference frame of one patient side cart to a reference frame of another patient side cart (e.g., transform coordinate points from a coordinate system of one patient side cart to a coordinate system of another patient side cart), or vice versa.

[0133] The tracking system 600 can use the missing links to complete the kinematic chain connecting the endoscope and surgical instrument mounted on a separate patient side cart so that the complete kinematic chain is known and accessible to the tracking system 600. The tracking system 600 can then use the motion of the endoscope and / or surgical instrument to track the position of the surgical instrument as described herein. For example, the tracking system 600 can use the endoscopic image to correct the kinematic position of the surgical instrument as described herein.

[0134] In certain examples, the use of neural networks as described herein can provide one or more advantages over conventional computer vision techniques. For example, as described herein, the use of a neural network that determines observations of an object of interest shown in an endoscopic image can be significantly faster and / or more accurate than conventional computer vision techniques (e.g., traditional pure vision methods where the algorithm that extracts specific features from the image is manually designed). It has been found that certain embodiments of a neural network that determines observations of a distal clevis of a surgical instrument are approximately ten times faster than conventional marker-based computer vision techniques (e.g., a speed of 10 Hz compared to 1 Hz).

[0135] As used herein, a neural network may include an input layer, any suitable number of hidden layers, and an output layer. In exemplary embodiments in which multiple hidden layers are included in the neural network, the multiple hidden layers may be layered in any suitable manner. A neural network including multiple hidden layers may be referred to as a "deep neural network."

[0136] Although exemplary embodiments that include and / or use neural networks are described herein, alternative embodiments may use other suitable machine learning models that are configured to learn to extract specific features from endoscopic images for object detection and / or classification.

[0137] Fig.16 An exemplary method 1600 for tracking the position of a robotically manipulated surgical instrument is shown. Fig.16 Exemplary operations according to one embodiment are shown, but other embodiments may omit, add to, reorder, and / or modify Fig.16 Any of the actions shown in . Fig.16 One or more of the operations shown in may be performed by a tracking system such as system 600, any components included therein, and / or any implementations thereof.

[0138] In operation 1602, a tracking system may determine, based on the endoscopic image of the surgical area and using the trained neural network, an observation of an object of interest shown in the endoscopic image. Operation 1602 may be performed in any manner described herein.

[0139] In operation 1604, the tracking system may associate observations of the object of interest with the robotically manipulated surgical instrument based on the probabilistic frame and motion of the robotically manipulated surgical instrument at the surgical area. Operation 1604 may be performed in any manner described herein.

[0140] In operation 1606, the tracking system may determine a physical location of the robotically manipulated surgical instrument in the surgical area based on the movement of the robotically manipulated surgical instrument and observations associated with the robotically manipulated surgical instrument. Operation 1606 may be performed in any manner described herein.

[0141] Fig.17 Another exemplary method 1700 for tracking the position of a robotically manipulated surgical instrument is shown. Fig.17 Exemplary operations according to one embodiment are shown, but other embodiments may omit, add to, reorder, and / or modify Fig.17 Any of the actions shown in . Fig.17 One or more of the operations shown in may be performed by a tracking system such as system 600, any components included therein, and / or any implementations thereof.

[0142] In operation 1702, the tracking system may determine, based on the endoscopic image of the surgical area and using the trained neural network, an observation of an object of interest shown in the endoscopic image. Operation 1702 may be performed in any manner described herein.

[0143] In operation 1704, the tracking system may associate observations of the object of interest with the robotically manipulated surgical instrument or false positive designations based on the movement of the robotically manipulated surgical instrument at the surgical area. Operation 1704 may be performed in any manner described herein.

[0144] In operation 1706, the tracking system may determine a physical location of the robotically manipulated surgical instrument in the surgical area based on the movement of the robotically manipulated surgical instrument and the observation associated with the robotically manipulated surgical instrument when the observation of interest was associated with the robotically manipulated surgical instrument in operation 1704. Operation 1706 may be performed in any manner described herein.

[0145] In operation 1708, the tracking system may refrain from using the observation to determine the physical position of the robotically manipulated surgical instrument in the surgical area when the observation of the object of interest is associated with a false positive designation in operation 1704. Operation 1706 may be performed in any manner described herein.

[0146] In certain embodiments, one or more of the systems, components, and / or processes described herein may be implemented and / or performed by one or more appropriately configured computing devices. To this end, one or more of the above-described systems and / or components may include or be implemented by any computer hardware and / or computer-implemented instructions (e.g., software) implemented on at least one non-transitory computer-readable medium that is configured to perform one or more of the processes described herein. In particular, the system components may be implemented on one physical computing device, or may be implemented on more than one physical computing device. Thus, the system components may include any number of computing devices, and may employ any of a large number of computer operating systems.

[0147] In some embodiments, one or more of the processes described herein may be implemented at least in part as instructions implemented in a non-transitory computer-readable medium and executable by one or more computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes those instructions to perform one or more processes, including one or more of the processes described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0148] Computer-readable media (also referred to as processor-readable media) include any non-temporary media that participate in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including but not limited to non-volatile media and / or volatile media. Non-volatile media can include, for example, optical or magnetic disks and other permanent memories. Volatile media can include, for example, dynamic random access memory ("DRAM") that typically constitutes main memory. Common forms of computer-readable media can include, for example, disks, hard disks, tapes, any other magnetic media, compact disk read-only memory ("CD-ROM"), digital video disks ("DVD"), any other optical media, random access memory ("RAM"), programmable read-only memory ("PROM"), electrically erasable programmable read-only memory ("EPROM"), flash-EEPROM, any other memory chip or cassette, or any other tangible medium that a computer can read.

[0149] Fig.18 An exemplary computing device 1800 is shown that may be specifically configured to perform one or more of the processes described herein. Fig.18 As shown in , computing device 1800 may include a communication interface 1802, a processor 1804, a storage device 1806, and an input / output ("I / O") module 1808 communicatively connected via a communication infrastructure 1810. Fig.18 An exemplary computing device 1800 is shown in FIG. Fig.18 The components shown in are not intended to be limiting. Additional or alternative components may be used in other embodiments. Fig.18 Components of computing device 1800 shown in .

[0150] The communication interface 1802 may be configured to communicate with one or more computing devices. Examples of the communication interface 1802 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0151] The processor 1804 generally represents any type or form of processing unit capable of processing data or interpreting, executing and / or directing the execution of one or more of the instructions, processes and / or operations described herein. The processor 1804 may direct the execution of operations in accordance with one or more applications 1812 or other computer-executable instructions such as may be stored in the storage device 1806 or another computer-readable medium.

[0152] The storage device 1806 may include one or more data storage media, devices or configurations, and may adopt any type, form of data storage media and / or devices and combinations of data storage media and / or devices. For example, the storage device 1806 may include, but is not limited to, a hard drive, a network drive, a flash drive, a disk, an optical disk, RAM, dynamic RAM, other non-volatile and / or volatile data storage units, or combinations or sub-combinations thereof. Electronic data including data described herein may be temporarily and / or permanently stored in the storage device 1806. For example, data representing one or more executable applications 1812 configured to direct the processor 1804 to perform any of the operations described herein may be stored in the storage device 1806. In some examples, the data may be arranged in one or more databases residing in the storage device 1806.

[0153] I / O modules 1808 may include one or more I / O modules configured to receive user input and provide user output. One or more I / O modules may be used to receive input for a single virtual reality experience. I / O modules may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, I / O modules 1808 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0154] The I / O module 1808 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 1808 is configured to provide graphical data to a display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content as may serve a particular implementation.

[0155] In some examples, any of the facilities described herein may be implemented by or within one or more components of computing device 1800. For example, one or more applications 1812 residing within storage device 1806 may be configured to direct processor 1804 to perform one or more processes or functions of associated facilities 602-606 of system 600. Likewise, storage facility 608 of system 600 may be implemented by storage device 1806 or a component thereof.

[0156] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with or replace features of another embodiment described herein. Therefore, this specification and the drawings should be considered in an illustrative rather than a restrictive sense.

Claims

1. A surgical instrument tracking system, the surgical instrument tracking system comprising: At least one physical computing device, the at least one physical computing device: determining, based on the endoscopic image of the surgical area and using the trained neural network, an observation of an object of interest shown in the endoscopic image; associating the observation of the object of interest with a robotically manipulated surgical instrument located in the surgical area based on a probabilistic frame and motion of the robotically manipulated surgical instrument; as well as Based on the movement of the robotically manipulated surgical instrument and the observations associated with the robotically manipulated surgical instrument, a physical position of the robotically manipulated surgical instrument within the surgical area is determined.

2. The system of claim 1, wherein: The endoscopic image includes a left endoscopic image and a right endoscopic image that is stereoscopic with the left endoscopic image; The observation of the object of interest includes a left observation and a right observation of the object of interest; and The at least one physical computing device: determining the observation of the object of interest by using the trained neural network to determine the left observation of the object of interest based on the left endoscopic image and to determine the right observation of the object of interest based on the right endoscopic image; associating the observation of the object of interest with the robotically manipulated surgical instrument by associating the left observation with the robotically manipulated surgical instrument and associating the right observation with the robotically manipulated surgical instrument; as well as The physical position of the robotically manipulated surgical instrument in the surgical area is determined by determining a three-dimensional ("3D") position of the object of interest in the 3D space using the left observation and the right observation and by adjusting a 3D motion-based position of the robotically manipulated surgical instrument in the 3D space based on the 3D position of the object of interest in the 3D space.

3. The system of claim 1, wherein: The endoscopic image includes a left endoscopic image and a right endoscopic image that is stereoscopic with the left endoscopic image; and The at least one physical computing device stereo matches the left and right endoscopic images and determines the observation of the object of interest by using the trained neural network to determine the observation of the object of interest based on the stereo matched left and right endoscopic images.

4. The system of claim 1, wherein: The endoscopic image comprises a combined endoscopic image; and The at least one physical computing device: generating the combined endoscopic image by combining a first endoscopic image and a second endoscopic image that is stereoscopic with the first endoscopic image; determining the observation of the object of interest by using the trained neural network to determine a three-dimensional ("3D") image-based location of the object of interest in 3D space based on the combined endoscopic image; as well as The physical position of the robotically manipulated surgical instrument in the surgical area is determined by adjusting the 3D motion-based position of the robotically manipulated surgical instrument in the 3D space based on the 3D image-based position of the object of interest in the 3D space.

5. The system of claim 1, wherein: The endoscopic image comprises a combined endoscopic image; and The at least one physical computing device: generating the combined endoscopic image by combining a plurality of endoscopic images included in a sequence of endoscopic video frames; determining the observation of the object of interest by using the trained neural network to determine a three-dimensional ("3D") image-based location of the object of interest in 3D space based on the combined endoscopic image; as well as The physical position of the robotically manipulated surgical instrument in the surgical area is determined by adjusting the 3D motion-based position of the robotically manipulated surgical instrument in the 3D space based on the 3D image-based position of the object of interest in the 3D space.

6. A system according to claim 5, wherein the at least one physical computing device determines the 3D image-based position of the object of interest in the 3D space based on the combined endoscopic image, at least in part based on the movement of the object of interest identified from the combined endoscopic image.

7. A system according to claim 1, wherein the at least one physical computing device determines the physical position of the robotically manipulated surgical instrument in the surgical area by adjusting the motion-based position of the robotically manipulated surgical instrument based on the image-based position of the object of interest indicated by the observation.

8. The system of claim 1, wherein the at least one physical computing device: determining, based on the endoscopic image of the surgical field and using the trained neural network, additional observations of additional objects of interest shown in the endoscopic image; associating the additional observation of the additional object of interest with a false positive designation based on the probabilistic framework and the motion of the robotically manipulated surgical instrument positioned in the surgical area; as well as Using the additional observations of the additional objects of interest to determine the physical position of the robotically manipulated surgical instrument is avoided.

9. The system of claim 1 , wherein the at least one physical computing device relates the observation of the object of interest to the robotically manipulated surgical instrument by: determining a first probability for a potential association of the observation of the object of interest with the robotically manipulated surgical instrument; determining a second probability for a potential association of the observation of the object of interest with a false-positive designation; determining that the first probability is higher than the second probability; as well as In response to determining that the first probability is higher than the second probability, the observation of the object of interest is associated with the robotically manipulated surgical instrument.

10. The system of claim 1, wherein the at least one physical computing device: determining, based on the endoscopic image of the surgical field and using the trained neural network, additional observations of additional objects of interest shown in the endoscopic image; associating the additional observation of the additional object of interest with a false positive designation based on the probabilistic framework and the motion of the robotically manipulated surgical instrument positioned in the surgical area; and wherein the at least one physical computing device associates the observation of the object of interest with the robotically manipulated surgical instrument and associates the additional observation of the additional object of interest with the false positive designation by: determining a first probability for a first set of potential associations, the first set of potential associations comprising a potential association of the observation of the object of interest with the robotically manipulated surgical instrument and a potential association of the additional observation of the additional object of interest with the false-positive designation; determining a second probability for a second set of potential associations, the second set of potential associations comprising a potential association of the observation of the object of interest with the false-positive designation and a potential association of the additional observation of the additional object of interest with the robotically manipulated surgical instrument; determining that the first probability is higher than the second probability; as well as In response to determining that the first probability is higher than the second probability, associating the observation of the object of interest with the robotically manipulated surgical instrument and associating the additional observation of the additional object of interest with the false positive designation.

Images