System and method for calibrating image sensor relative to robotic instrument

Abstract

An illustrative system includes a memory storing instructions and a processor communicatively coupled to the memory. A processor may be configured to execute the instructions to perform a process that includes obtaining scan data collected during a scan movement in which an image sensor physically coupled to a robotic instrument moves within an imaging space, the scan data includes an image captured by the image sensor during the scan movement and tracking data of the robotic instrument during the scan movement; determining first motion data of the image sensor based on an image captured by the image sensor during the scan movement; determining second motion data of the robotic instrument based on the tracking data of the robotic instrument; and determining a calibration of the image sensor relative to the robotic instrument in the imaging space based on the first motion data and the second motion data.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 449,253, filed on March 1, 2023, the contents of which are hereby incorporated by reference in their entirety. Background Art

[0003] Computer-assisted surgery systems that employ robotic and / or teleoperation technology typically include a stereoscopic image viewer configured to provide an image of an imaging space (e.g., a surgical space) captured by an imaging device such as an endoscope for display to a surgeon. When the surgeon's eyes are positioned in front of the viewing lens of the stereoscopic image viewer, the surgeon can view the image of the surgical space while remotely manipulating one or more surgical instruments located within the surgical space. The surgical instruments are attached to one or more manipulator arms of a surgical instrument manipulation system included as part of the computer-assisted surgery system.

[0004] In addition to the surgical instruments attached to one or more manipulator arms, additional instruments can be inserted into the surgical space to facilitate the surgeon to perform the procedure in the surgical space. For example, an image sensor such as a subsurface sensing device (e.g., an ultrasonic device) can be provided in the surgical space to improve the surgeon's perception of the surgical space and improve the results of the procedure. However, such additional instruments are generally not integrated into the module of the manipulator arm attached to the computer-assisted surgery system. In view of this, such additional instruments can only be available as drop-in instruments, which rely on, for example, a gripper surgical instrument attached to the manipulator arm of the computer-assisted surgery system to grab and move the insertion instrument in the surgical space. Operating a remote-operated gripper surgical instrument to interact with the insertion instrument requires the surgeon to perform complex operations to pick up and use the insertion instrument in the surgical space. In addition, due to the presence of a variety of ways (e.g., grabbing position, grabbing angle, etc.) in which a robotic instrument such as a remote-operated gripper surgical instrument can engage with the insertion instrument, it is difficult to determine the position and / or orientation of the insertion instrument relative to the robotic instrument. Summary of the Invention

[0005] An example system includes a memory storing instructions and a processor communicatively coupled to the memory and configured to execute the instructions to perform processing including obtaining scan data collected during a scan move in which an image sensor physically coupled to a robotic instrument moves within an imaging space, the scan data including images captured by the image sensor during the scan move and tracking data of the robotic instrument during the scan move, determining first motion data of the image sensor based on the images captured by the image sensor during the scan move, determining second motion data of the robotic instrument based on the tracking data of the robotic instrument, and determining a calibration of the image sensor relative to the robotic instrument in the imaging space based on the first motion data and the second motion data.

[0006] An example computer program product embodied in a non-transitory computer readable storage medium includes computer instructions for obtaining scan data collected during a scan move in which an image sensor physically coupled to a robotic instrument moves within an imaging space, the scan data including images captured by the image sensor during the scan move and tracking data of the robotic instrument during the scan move, determining first motion data of the image sensor based on the images captured by the image sensor during the scan move, determining second motion data of the robotic instrument based on the tracking data of the robotic instrument, and determining a calibration of the image sensor relative to the robotic instrument in the imaging space based on the first motion data and the second motion data.

[0007] An example method includes obtaining, by a calibration system, scan data collected during a scan move in which an image sensor physically coupled to a robotic instrument moves within an imaging space, the scan data including images captured by the image sensor during the scan move and tracking data of the robotic instrument during the scan move, determining, by the calibration system and based on the images captured by the image sensor during the scan move, first motion data of the image sensor, determining, by the calibration system and based on the tracking data of the robotic instrument, second motion data of the robotic instrument, and determining, by the calibration system and based on the first motion data and the second motion data, a calibration of the image sensor relative to the robotic instrument in the imaging space. BRIEF DESCRIPTION OF DRAWINGS

[0008] The accompanying drawings illustrate various implementations and are a part of the specification. The illustrated implementations are merely examples and do not limit the scope of the disclosure. Throughout the drawings, the same or like reference numerals denote similar or like elements.

[0009] Figure 1 An example computer-assisted surgical system according to the principles described herein is shown.

[0010] Figure 2 An example view of an imaging space is shown in accordance with the principles described herein.

[0011] Figure 3 Examples are shown of various different ways in which a robotic instrument in accordance with the principles described herein can interface with an example image sensor.

[0012] Figure 4 An example calibration system is shown in accordance with the principles described herein.

[0013] Figure 5 An example flowchart is shown depicting various operations that can be performed by a calibration system in accordance with the principles described herein. Figure 4

[0014] Figure 6 An example image of an imaging space is shown in accordance with the principles described herein.

[0015] Figure 7A An example image of an imaging space is shown in accordance with the principles described herein. 7B

[0016] An example image of an imaging space is shown in accordance with the principles described herein. Figure 8A Figure 8B An example method for calibrating an image sensor relative to a robotic instrument is shown in accordance with the principles described herein.

[0017] Figure 9 An example computing device is shown in accordance with the principles described herein.

[0018] DETAILED DESCRIPTION Figure 10 Systems and methods for calibrating an image sensor relative to a robotic instrument are described herein. As will be described in greater detail below, an illustrative system includes a memory storing instructions and a processor communicatively connected to the memory. The processor is configured to execute the instructions to perform processing including obtaining scan data collected during a scan movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space, the scan data including images captured by the image sensor during the scan movement and tracking data of the robotic instrument during the scan movement, determining first motion data of the image sensor based on the images captured by the image sensor during the scan movement, determining second motion data of the robotic instrument based on the tracking data of the robotic instrument, and determining a calibration of the image sensor relative to the robotic instrument in the imaging space based on the first motion data and the second motion data.

[0019] Systems and methods for calibrating an image sensor relative to a robotic instrument are described herein. As will be described in greater detail below, an illustrative system includes a memory storing instructions and a processor communicatively connected to the memory. The processor is configured to execute the instructions to perform processing including obtaining scan data collected during a scan movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space, the scan data including images captured by the image sensor during the scan movement and tracking data of the robotic instrument during the scan movement, determining first motion data of the image sensor based on the images captured by the image sensor during the scan movement, determining second motion data of the robotic instrument based on the tracking data of the robotic instrument, and determining a calibration of the image sensor relative to the robotic instrument in the imaging space based on the first motion data and the second motion data. ​​

[0020] Various advantages and benefits are associated with the systems and methods described herein. For example, systems and methods such as those described herein can facilitate fast and / or convenient calibration of an image sensor relative to a robotic instrument, regardless of the manner in which the image sensor is engaged (e.g., grasped) by the robotic instrument. Such calibration can result in improved imaging by the image sensor during operation of the image sensor in an imaging space. Additionally, the systems and methods described herein for calibration can be based on uncharacterized data associated with the imaging space and do not require fiducials and / or supplemental information associated with phantom objects that can be used in conventional calibration methods. Furthermore, systems and methods such as those described herein can reduce the mental and / or physical workload required of a user (e.g., a surgeon and / or another user associated with a computer-assisted surgical system) of the computer-assisted surgical system to use (e.g., teleoperate) a robotic instrument to interact with an image sensor located in a surgical space, such as by the systems and methods facilitating robotic instrument performance of scanning movements autonomously or semi-autonomously and / or providing guidance to the user to assist the user in performing scanning movements. In doing so, systems and methods such as those described herein can simplify processes performed within a surgical space and / or improve the usability of a computer-assisted surgical system. These and other benefits that can be achieved by the systems and methods described herein will be apparent from the following disclosure.

[0021] The example systems described herein can be configured to operate as part of or in conjunction with a variety of different types of computer-assisted surgical systems. Different types of computer-assisted surgical systems can include any type of computer-assisted surgical system as can serve a particular implementation, such as, for example, a computer-assisted surgical system designed for minimally invasive medical procedures. In certain examples, one type of computer-assisted surgical system can include a system in which one or more surgical devices (e.g., surgical instruments) are controlled manually (e.g., laparoscopically) by a user. In certain examples, one type of computer-assisted surgical system can include a robotic surgical system configured to facilitate operation of one or more intelligent instruments (e.g., intelligent subsurface imaging devices) that can be controlled manually and / or in a robotic manner by a user. In certain implementations, a plurality of different types of computer-assisted surgical systems can have different types at least because they include different types of surgical instrument manipulation systems. For example, a first computer-assisted surgical system can include a first type of surgical instrument manipulation system, a second computer-assisted surgical system can include a second type of surgical instrument manipulation system, and a third computer-assisted surgical system can include a third type of surgical instrument manipulation system.

[0022] Each type of surgical instrument manipulation system can have a different architecture (e.g., manipulator arm architecture), have different kinematic profiles, and / or operate according to different configuration parameters. Reference will now be made to Figure 1 An illustrative computer-assisted surgical system having a surgical instrument manipulation system of a first type is described. The described computer-assisted surgical system is illustrative and not limiting. Systems such as those described herein can operate as part of or in conjunction with the described computer-assisted surgical system and / or any other suitable computer-assisted surgical system.

[0023] Figure 1 An example computer-assisted surgical system 100 (“surgical system 100”) is shown. As shown, the surgical system 100 can include a surgical instrument manipulation system 102 (“manipulation system 102”), a user control system 104, and an auxiliary system 106 communicatively coupled to one another.

[0024] A surgical team can utilize the surgical system 100 to perform a computer- assisted surgical procedure on a patient 108. As shown, the surgical team can include a surgeon 110-1, an assistant 110-2, a nurse 110-3, and an anesthesiologist 110-4, all of which can be collectively referred to as “surgical team members 110.” Additional or alternative surgical team members can be present during a surgical session, as can serve a particular implementation.

[0025] While Figure 1 A minimally invasive surgical procedure is shown being performed, the surgical system 100 can similarly be used to perform an open surgical procedure or other type of surgical procedure that can similarly benefit from the accuracy and convenience of the surgical system 100. Additionally, it will be understood that a surgical session during which the surgical system 100 can be employed can include not only operating phases of a surgical procedure, as Figure 1 shown, but also pre-operative, post-operative, and / or other suitable phases of a surgical procedure. A surgical procedure can include any procedure in which manual and / or instrumented techniques (e.g., teleoperated instrumented techniques) are used on a patient to investigate, diagnose, or treat a physical condition of the patient. Additionally, a surgical procedure can include any procedure that is not performed on a living patient, such as calibration procedures, simulation training procedures, and experimental or research procedures.

[0026] As Figure 1As shown, the surgical instrument manipulation system 102 can include a plurality of manipulator arms 112 (e.g., manipulator arms 112-1 through 112-4) to which a plurality of robotic surgical instruments (“robotic instruments”) (not shown) can be coupled. As used herein, a “robotic instrument” refers to any instrument that can be directly attached to (e.g., plugged into, fixedly coupled to, mated to, etc.) a manipulator arm (e.g., manipulator arm 112-1) such that movement of the manipulator arm directly causes movement of the instrument. Each robotic instrument can be implemented by any suitable therapeutic instrument (e.g., a tool having tissue-interaction functionality), imaging device (e.g., an endoscope), diagnostic instrument, etc. that can be used in a computer-assisted surgical procedure (e.g., by being at least partially inserted into the patient 108 and manipulated to perform a computer-assisted surgical procedure on the patient 108). In some examples, one or more of the robotic instruments can include force-sensing and / or other sensing functionality.

[0027] In Figure 1 In the example shown, the manipulator arms 112 of the manipulation system 102 are attached at the distal ends of horizontally extending overhead booms. However, in certain implementations, the manipulator arms 112 can have other configurations. Additionally, while the manipulation system 102 is depicted and described herein as including four manipulator arms 112, it will be recognized that the manipulation system 102 can include only a single manipulator arm 112 or any other number of manipulator arms as can serve a particular implementation.

[0028] The manipulator arms 112 and / or the robotic instruments attached to the manipulator arms 112 can include one or more displacement transducers, orientation sensors, and / or position sensors (hereinafter “surgical system sensors”) for generating raw (e.g., uncorrected) kinematic information. One or more components of the surgical system 100 can be configured to use the kinematic information to track the robotic instruments (e.g., determine the position of the robotic instruments) and / or control the robotic instruments.

[0029] Additionally, the manipulator arms 112 can each include or otherwise be associated with a plurality of motors that control movement of the manipulator arms 112 and / or surgical instruments attached thereto. For example, the manipulator arm 112-1 can include or otherwise be associated with a first internal motor (not explicitly shown) configured to yaw the manipulator arm 112-1 about a yaw axis. In a similar manner, the manipulator arm 112-1 can be associated with a second internal motor (not explicitly shown) configured to drive and pitch the manipulator arm 112-1 about a pitch axis. Likewise, the manipulator arm 112-1 can be associated with a third internal motor (not explicitly shown) configured to slide the manipulator arm 112-1 along an insertion axis. The manipulator arms 112 can each include a drive train system driven by one or more of these motors in order to control pivoting of the manipulator arms 112 in any manner as can serve a particular implementation. Thus, if a robotic instrument attached to, for example, the manipulator arm 112-1 is to be mechanically moved, one or more of the motors coupled to the drive train can be energized to move the manipulator arm 112-1.

[0030] The robotic instruments attached to the manipulator arms 112 can each be positioned in an imaging space. As used herein, an “imaging space” can refer to any space or location in which an imaging operation can be performed by an imaging device such as described herein. In certain examples, the imaging space can correspond to a surgical space. In certain examples, a “surgical space” can be disposed entirely within a patient and can include an area within the patient at or near a location at which a surgical procedure is planned to be performed, is being performed, or has been performed. For example, for a minimally invasive surgical procedure performed on tissue inside a patient, the surgical space can include the tissue, the anatomy underneath the tissue, and the space around the tissue in which, for example, a robotic instrument and / or other instruments used to perform the surgical procedure are located. In other examples, the surgical space can be at least partially disposed outside of the patient, at or near a location at which a surgical procedure is planned to be performed, is being performed, or has been performed on the patient. For example, the surgical system 100 can be used to perform an open surgical procedure such that a portion of the surgical space (e.g., tissue being operated on) is inside the patient, while another portion of the surgical space (e.g., the space around the tissue in which one or more instruments can be disposed) is outside the patient. A robotic instrument can be said to be positioned at or within a surgical space when at least a portion of the robotic instrument (e.g., a distal portion of the robotic instrument) is located within the surgical space. Example imaging spaces and / or images of imaging spaces will be described herein.

[0031] The user control system 104 can be configured to facilitate surgeon 110-1 control of the manipulator arms 112 and robotic instruments attached thereto. For example, the surgeon 110-1 can interact with the user control system 104 to remotely move, manipulate, or otherwise remotely operate the manipulator arms 112 and robotic instruments. To this end, the user control system 104 can provide the surgeon 110-1 with one or more images (e.g., high definition three-dimensional (3D) images) of the surgical space associated with the patient 108 as captured by an imaging device. In certain examples, the user control system 104 can include a stereoscopic image viewer with two displays, where the surgeon 110-1 can view stereoscopic images (e.g., 3D images) of the surgical space associated with the patient 108 and generated by a stereoscopic imaging system. The surgeon 110-1 can utilize the images to perform one or more procedures with one or more robotic instruments attached to the manipulator arms 112.

[0032] To facilitate control of the robotic instruments, the user control system 104 can include a set of master controls (not shown). These master controls can be manipulated by the surgeon 110-1 to control movement of the robotic instruments (e.g., by utilizing robotics and / or teleoperation techniques). The master controls can be configured to detect various hand, wrist, and finger movements of the surgeon 110-1. In this manner, the surgeon 110-1 can intuitively perform surgical procedures using the one or more robotic instruments.

[0033] The user control system 104 can also be configured to facilitate surgeon 110-1 control of other components of the surgical system 100. For example, the surgeon 110-1 can interact with the user control system 104 to change a configuration or operating mode of the surgical system 100, change a display mode of the surgical system 100, generate additional control signals for controlling surgical instruments attached to the manipulator arms 112, facilitate switching control from one robotic instrument to another robotic instrument, facilitate interaction with other instruments and / or objects within the surgical space, or perform any other suitable operation. To this end, the user control system 104 can also include one or more input devices (e.g., foot pedals, buttons, switches, etc.) configured to receive input from the surgeon 110-1.

[0034] The auxiliary system 106 can include one or more computing devices configured to perform primary processing operations of the surgical system 100. The one or more computing devices included in the auxiliary system 106 can control and / or coordinate operations performed by various other components of the surgical system 100 (e.g., the manipulation system 102 and / or the user control system 104). For example, a computing device included in the user control system 104 can send instructions to the manipulation system 102 through the one or more computing devices included in the auxiliary system 106. As another example, the auxiliary system 106 can receive and process image data representing images captured by an imaging device attached to one of the manipulator arms 112 from the manipulation system 102.

[0035] In some examples, the auxiliary system 106 can be configured to present visual content to the surgical team members 110 that can not have access to images provided to the surgeon 110-1 at the user control system 104. To this end, the auxiliary system 106 can include a display monitor 114 configured to display one or more user interfaces, such as images of the surgical space (e.g., 2D images), information associated with the patient 108 and / or the surgical procedure, and / or any other visual content as can serve a particular implementation. For example, the display monitor 114 can display an image of the surgical space along with additional content displayed contemporaneously with the image (e.g., a representation of a target object, graphical content, contextual information, etc.). In some implementations, the display monitor 114 is implemented by a touch screen display with which the surgical team members 110 can interact (e.g., through touch gestures) to provide user input to the surgical system 100.

[0036] The manipulation system 102, the user control system 104, and the auxiliary system 106 can be communicatively coupled to one another in any suitable manner. For example, as shown, the manipulation system 102, the user control system 104, and the auxiliary system 106 can be communicatively coupled by control lines 116, which can represent any wired or wireless communication link as can serve a particular implementation. To this end, the manipulation system 102, the user control system 104, and the auxiliary system 106 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc. Figure 1

[0037] Figure 2 ​A view 200 of an imaging space (e.g., a surgical space) is shown in which various robotic instruments are attached to manipulator arms 112 of the surgical system 100. As shown, the robotic instruments can include an imaging device 202 and one or more other robotic instruments 204 (e.g., robotic instruments 204-1 through 204-3) in the form of one or more surgical tools. While Figure 2 One imaging device 202 and three other robotic instruments 204 are shown positioned at the imaging space, any number, type, and / or combination of robotic instruments can be present at the imaging space during a surgical procedure. For example, in some implementations, one or more of the robotic instruments 204 can be diagnostic tools, treatment tools, etc. Figure 2 In the example shown, robotic instruments 204-1 and 204-3 are shown as grasping-type robotic instruments, while robotic instrument 204-2 is shown as a cutting-type robotic instrument. It should be appreciated that other types of robotic instruments (e.g., diagnostic tools, treatment tools, etc.) different from the robotic instruments shown can additionally or alternatively be disposed within the imaging space during a surgical procedure. Tissue 206 represents anatomical tissue at the imaging space. Figure 2

[0038] The imaging device 202 can capture one or more images at the imaging space. Any of the robotic instruments 204 and / or tissue 206 within the field of view of the imaging device 202 can be depicted in the images captured by the imaging device 202.

[0039] The imaging device 202 can provide data representing visible light data of the imaging space. For example, the imaging device 202 can capture a visible light image of the surgical space representing visible light sensed by the imaging device 202. The visible light image can include an image representing a visible light-based view of the imaging space using any suitable color and / or grayscale palette.

[0040] ​The imaging device 202 can also provide data representative of depth data of the imaging space or data that can be processed to derive depth data of the imaging space. For example, the imaging device 202 can capture an image of the imaging space representative of depth sensed by the imaging device 202. Alternatively, the imaging device 202 can capture an image of the imaging space that can be processed to derive depth data of the imaging space. The depth information can be represented as a depth image (e.g., a depth map image obtained using a Z-buffer indicating a distance from the imaging device 202 to each pixel point on the image of the imaging space), which can be configured to visually indicate depths of objects in the imaging space in any suitable manner, such as by using different shades of gray to represent different depth values. Images captured by an imaging device (e.g., by the imaging device 202) and / or derived from images (e.g., visible light images and depth images) captured by an imaging device can be used to facilitate detecting robotic instruments (e.g., robotic instruments 204-1 through 204-3) and / or one or more objects within the imaging space, such as described herein.

[0041] During a procedure such as a surgical procedure, it can be desirable to use a robotic instrument (e.g., robotic instrument 204-1 or 204-3) to remotely operate an image sensor that is provided in addition to the imaging device 202 but not directly attached to one of the manipulator arms 112. In this scenario, such an image sensor can only be able to be moved within the imaging space by being manually manipulated by a user (e.g., surgeon 110-1, assistant 110-2, etc.) or by being moved by remote operation of the image sensor by a user through a robotic instrument directly attached to one of the manipulator arms. In this regard, “remote operation of an image sensor” as described herein refers to indirect remote operation of an image sensor by a robotic instrument attached to a computer-assisted surgical system. As such, such an image sensor can be referred to as an interposed image sensor. Examples of image sensors can include, but are not limited to, an interposed ultrasound probe and / or any other suitable image sensor that can be disposed in the imaging space.

[0042] Image sensors, such as those described herein, that are not directly attached to one of the manipulator arms 112 can be engaged with the computer-assisted surgery system in any suitable manner. In some examples, such image sensors can be engaged by the computer-assisted surgery system by being communicatively coupled to the computer-assisted surgery system in any suitable manner. In some examples, such image sensors can be physically coupled to a component of the computer-assisted surgery system and / or coupled by being attached to a component of the computer-assisted surgery system, such as a robotic instrument (e.g., robotic instrument 204-1 or 204-3) attached to the computer-assisted surgery system. This can be achieved in any suitable manner. For example, in some implementations, the image sensor can be configured to be grasped by a gripper robotic instrument, such as robotic instrument 204-1. To this end, in some examples, the image sensor can include one or more graspable portions (e.g., protrusions, rings, etc.) that can be grasped by the robotic instrument to facilitate user remote operation of the image sensor. Alternatively, the robotic instrument can generally grasp a housing or casing of the image sensor. Example image sensors are further described herein.

[0043] There is a great deal of variability in how an image sensor may be physically coupled to one of the robotic instruments 204 . Figure 3 A diagram 300 is shown depicting a number of different example grasping states that the robotic instrument 204-1 may assume when grasping an image sensor 302 (e.g., image sensors 302-1 and 302-2). Figure 3 As shown, the image sensor 302-1 can be grasped in a first grasping state 304-1 in which the robotic instrument 204-1 grasps the central area of ​​the image sensor 302-1. Alternatively, the image sensor 302-1 can be grasped in a second grasping state 304-2 in which the robotic instrument 204-1 grasps the image sensor 302-1 at a position closer to the distal end of the image sensor 302-1. The image sensor 302-2 differs from the image sensor 302-1 in that the image sensor 302-2 includes a protrusion 306 configured to be grasped by the robotic instrument 204-1. Figure 3 As shown, the protrusion 306 of the image sensor 302-2 can be grasped in a first grasping state 308-1 in which the protrusion 306 is deeply seated within the jaws of the robotic instrument 204-1. Alternatively, the protrusion 306 can be grasped in a second grasping state 308-2 in which the protrusion 306 is only partially seated within the jaws of the robotic instrument 204-1. Figure 3The depicted example grasp states are provided for illustrative purposes only. It should be understood that, in certain implementations, the robotic instrument 204-1 can grasp the image sensors 302-1 and 302-2 at any other suitable location and / or at any other suitable angle relative to the image sensors 302.

[0044] Due to the variability of the manner in which the image sensor can be physically coupled to the robotic instrument, it can be difficult to directly use kinematic information of the robotic instrument to perform operations associated with the image sensor in the imaging space (e.g., for three-dimensional ultrasound reconstruction, augmented reality, and / or other forms of guidance). This is because each time the robotic instrument is interfaced with the image sensor, there can be an unknown relationship between the pose of the image sensor (e.g., 3D position in 3D space and / or orientation in 3D space) and the pose of the robotic instrument that is interfaced with the image sensor. Accordingly, it is desirable to perform a calibration to determine the relationship between the position of the image sensor and the position of the robotic instrument when the image sensor is physically coupled to the robotic instrument.

[0045] Figure 4 An example calibration system 400 that can be implemented in accordance with the principles described herein to calibrate an image sensor relative to a robotic instrument is shown. As Figure 4 shown, the calibration system 400 (e.g., system 400) can include, without limitation, a memory 402 and a processor 404 that are selectively and communicatively coupled to one another. The memory 402 and the processor 404 can each include or be implemented by hardware and / or software components (e.g., processors, memories, communication interfaces, instructions stored in memory for execution by a processor, etc.). In some examples, the memory 402 and the processor 404 can be implemented by a single device (e.g., a single computing device). In certain alternative examples, the memory 402 and the processor 404 can be distributed among multiple devices and / or among multiple locations, as can serve a particular implementation.

[0046] The memory 402 can hold (e.g., store) executable data used by the processor 404 to perform any of the operations described herein. For example, the memory 402 can store instructions 406 that can be executed by the processor 404 to perform any of the operations described herein. The instructions 406 can be implemented by any suitable application, software, code, and / or other executable data instance.

[0047] The memory 402 can also hold any data received by, generated by, managed by, used by, and / or transmitted by the processor 404. For example, the memory 402 can hold any suitable data associated with calibrating an image sensor. Such data can include, but is not limited to: data associated with a scanning movement, depth map information associated with an imaging space, scan data (e.g., images captured by an image sensor, kinematic data of a robotic instrument and / or manipulator arm, etc.), pose information associated with an image sensor and / or additional objects located in a surgical space, endoscopic images of an imaging space, data defining guidance content associated with an image sensor, augmented images of an imaging space, composite images, motion path data, user interface content (e.g., graphical objects, notifications, etc.), ultrasound reconstructed volumes, and / or any other suitable data.

[0048] The processor 404 can be configured to perform (e.g., execute instructions 406 stored in the memory 402) various processing operations associated with calibrating an image sensor. For example, the processor 404 can determine a calibration of an image sensor in an imaging space relative to a robotic instrument based on first motion data of the image sensor and second motion data of the robotic instrument. These and other operations that can be performed by the processor 404 are described herein.

[0049] Figure 5 A flowchart 500 depicting various operations that can be performed by the system 400 (e.g., the processor 404) to determine a calibration, such as described herein, is shown. At operation 502, the system 400 can obtain scan data. Such scan data can be collected during a scanning movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space.

[0050] The scan data can include any suitable information that can be accessed, obtained, or generated by the system 400 during the scanning movement. For example, the scan data can include images captured by the image sensor during the scanning movement and tracking data of the robotic instrument during the scanning movement. The images can include any suitable number of images captured by the image sensor during the scanning movement. For example, the images can include a first image captured by the image sensor at a first point in time during the scanning movement, a second image captured by the image sensor at a second point in time during the scanning movement, a third image captured by the image sensor at the second point in time during the scanning movement, and so on. In examples in which the image sensor corresponds to an insertable ultrasound image sensor, the images can correspond to subsurface images of tissue captured during the scanning movement.

[0051] The tracking data of the robotic instrument can include any suitable data associated with the robotic instrument. For example, the tracking data can include kinematic data indicative of at least one of a position or an orientation of the robotic instrument during the scan movement. In certain examples, the system 400 can map the kinematic data at particular points in time during the scan movement to corresponding images captured by the image sensor at the particular points in time. For example, a first set of kinematic data of the robotic instrument at a first point in time can be mapped to a first image captured by the image sensor, a second set of kinematic data of the robotic instrument at a second point in time can be mapped to a second image captured by the image sensor, a third set of kinematic data of the robotic instrument can be mapped to a third image captured by the image sensor, and so on.

[0052] The scan movement can include any suitable movement of the robotic instrument in the imaging space. In certain examples, the scan movement can be performed by a user (e.g., the surgeon 110-1) remotely operating the robotic instrument to move the image sensor in the imaging space.

[0053] To illustrate an example, Figure 6 An image 600 is shown in accordance with the principles described herein, which can be captured by the imaging device 202 and include an example image sensor that can be remotely operated in the surgical space during a surgical procedure. As Figure 6 shown, the image 600 shows a surgical space in which the image sensor 302-2 and the robotic instruments 204-1 through 204-3 are disposed relative to a kidney 602 of a patient (e.g., the patient 108). As Figure 6 shown, the image sensor 302-2 includes a protrusion 306 that is grasped by the robotic instrument 204-1. Accordingly, remote operation of the robotic instrument 204-1 results from remote operation of the robotic instrument 204-1 by a user (e.g., manipulation of master controls of the user control system 104 by the surgeon 110-1). In Figure 6 the example shown, the arrow 604-1 represents an example motion path that the robotic instrument 204-1 can take to reach a surface of the kidney 602. The arrow 604-2 represents an example scan movement that the robotic instrument 204-1 can take to move the image sensor 302-2 along the surface of the kidney 602. Although Figure 6 the example scan movement depicted in the image 600 has a curved shape, it should be appreciated that scan movements can have any suitable shape and / or properties as can serve a particular implementation. For example, in certain examples, a scan movement can be performed in a straight line. Alternatively, a scan movement can include one or more changes in direction. For example, in certain implementations, a scan movement can have a zigzag shape and / or can back-scan over a previously traveled portion of the scan movement.

[0054] In certain implementations, the system 400 can facilitate user remote operation of an image sensor within an imaging space at various levels of autonomy. For example, in certain examples, the system 400 can assist a user in remotely operating a robotic instrument to perform a scan move. For example, the system 400 can provide haptic feedback, audible feedback, visual feedback, and / or any other suitable notification or guidance to facilitate a user moving a robotic instrument, such as the robotic instrument 204-1, along a motion path associated with a scan move.

[0055] In certain alternative examples, the system 400 can automatically perform a scan move in the imaging space. As used herein, the expression“automatically” means performing an operation (e.g., performing a scan move) or series of operations without requiring further input from a user. For example, the system 400 can analyze depth data, images, and / or any suitable information associated with the imaging space. Based on such information, the system 400 can automatically control a robotic instrument to move toward a starting position of a scan move and / or automatically move the robotic instrument along a surface in the imaging space during a scan move without requiring further input from a user.

[0056] Returning to Figure 5 At operation 504, the system 400 can determine first motion data of the image sensor based on images captured by the image sensor during the scan move. The first motion data can be indicative of a trajectory of the image sensor during the scan move (e.g., a set of positions and / or orientations of the image sensor over time during the scan move). The system 400 can determine the first motion data in any suitable manner. For example, in certain implementations, the system 400 can determine the first motion data by determining spatial relationships between captured images using temporal changes in patterns in the images captured by the image sensor.

[0057] To illustrate, Figure 7A and Figure 7B Images 700 (e.g., images 700-1 and 700-2) of a surgical space are shown in which the image sensor 302-2 is used to capture ultrasound images 702 (e.g., ultrasound images 702-1 and 702-2) at different positions within the surgical space. As shown, the image sensor 302-2 is positioned at a first position along a scan move represented by arrow 604-2. The ultrasound image 702-1 includes a first pattern representing a sub-surface image of the kidney 602. On the other hand, the ultrasound image 702-2 includes a second pattern representing a sub-surface image of the kidney 602. Figure 7A As shown, the image sensor 302-2 is positioned at a first position along a scan move represented by arrow 604-2. The ultrasound image 702-1 includes a first pattern representing a sub-surface image of the kidney 602. On the other hand, the ultrasound image 702-2 includes a second pattern representing a sub-surface image of the kidney 602. Figure 7BIn particular implementations, the image sensor 302-2 is positioned at the second location along a scan movement represented by arrow 604-2. The ultrasound 702-2 includes a second pattern representing a subsurface image of the kidney 602. The temporal change between the first pattern in the image 702-1 and the second pattern in the image 702-2 can be indicative of a spatial relationship between the images, and can be used by the system 400 in any suitable manner to determine the first motion data. This is one example of how images captured by an image sensor such as the image sensor 302-2 can be used. Other suitable ways of using such images can be utilized in other examples.

[0058] For illustrative purposes, the ultrasound images 702 are shown to the side of the image 700 in Figure 7A and Figure 7B It should be appreciated that the ultrasound images 702 can be provided for display in any suitable manner as can serve a particular implementation. In certain alternative implementations, the ultrasound images can be provided as an augmentation to an image of the surgical space (e.g., as an overlay on an endoscopic image of the surgical space). For example, in certain implementations, the ultrasound image 702-1 can be overlaid on a portion of the image 700-2 so that a user (e.g., the surgeon 110-1) can view one or more captured ultrasound images simultaneously and in situ as the teleoperational robotic instrument 204-1 is moved by the image sensor 302-2. Additionally or alternatively, the ultrasound images 700 can be provided for display at any other location relative to an image of the surgical space and / or by any other suitable display device (e.g., the display monitor 114) associated with the computer-assisted surgical system. In certain alternative examples, the ultrasound images associated with the scan movement can not be provided for display. In such examples, the determination of the first motion data based on the images captured by the image sensor can be transparent to the user (e.g., the determination can be performed as a background process).

[0059] In certain examples, determining the spatial relationship between the ultrasound images can include using speckle decorrelation. In such examples, a distance that the image sensor moved during the scan movement can be estimated based on an amount of correlation between a first speckle pattern in a first image captured by the image sensor during the scan movement and a second speckle pattern in a second image captured by the image sensor after the first image.

[0060] Additionally or alternatively, the system 400 can determine the spatial relationship by implementing a machine learning operation. Any suitable machine learning operation can be used as can serve a particular implementation. In certain examples, a pre-trained machine learning algorithm trained based on different surface profiles that can exist in the imaging space can be implemented. Additionally or alternatively, the machine learning algorithm can be trained based on one or more prior procedures performed in the imaging space. For example, in certain implementations, information obtained during one or more prior surgical procedures associated with different patients can be used to train the machine learning algorithm. In certain examples, such a machine learning algorithm can additionally or alternatively be configured to filter out portions of the images captured by the image sensor that are not suitable for speckle decorrelation.

[0061] Returning to Figure 5 At operation 506, the system 400 can determine second motion data for the robotic instrument based on the tracking data for the robotic instrument. This can be implemented in any suitable manner. For example, the system 400 can determine the second motion data based on kinematic data for the robotic instrument as the robotic instrument moves during the scan movement. The kinematic data can specify any suitable information associated with the robotic instrument as the robotic instrument moves during the scan movement. For example, the kinematic data can include information regarding the position, orientation, trajectory, etc. of the robotic instrument at any given point in time during the scan movement. In certain examples, the second motion data can also be based on endoscopic images captured by an imaging device, such as the imaging device 202. In such examples, the endoscopic images can be analyzed in any suitable manner to confirm and / or adjust the position, orientation, and / or trajectory data associated with the movement of the robotic instrument during the scan movement and / or the second motion data.

[0062] At operation 508, the system 400 can determine a calibration of the image sensor in the imaging space relative to the robotic instrument. This can be implemented in any suitable manner. For example, the system 400 can determine a calibration transform between the first motion data and the second motion data. Such a calibration transform can be computed in any suitable manner. For example, the system 400 can formulate the calibration as an AX = XB problem and estimate the calibration of the image sensor relative to the robotic instrument based on a solution to the AX = XB problem. In such examples, A, X, and B can each represent a homogeneous transform (e.g., a rigid body motion), where A represents the first motion data and B represents the second motion data. With such an equation, X can be solved based on the correspondence between the first motion data and the second motion data and determine the current spatial relationship between the robotic instrument and the physically coupled image sensor.

[0063] In certain examples, the system 400 can determine whether the calibration determined at operation 508 is within a predefined threshold confidence level. If it is determined that the calibration is not within the predefined threshold confidence level, the system 400 can perform additional scan movements and repeat operations 502-508 until the predefined threshold confidence level is reached.

[0064] At operation 510, the system 400 can use the calibration in one or more processes associated with the image sensor in the imaging space. This can be implemented in any suitable manner. For example, based on the calibration, the system 400 can facilitate the image sensor maintaining (e.g., automatically or through haptic feedback) a predetermined contact angle (e.g., a 90° contact angle) relative to the tissue during the surgical procedure. Additionally or alternatively, the system 400 can use the calibration to process images captured by the image sensor during the surgical procedure and / or generate or refine a 3D reconstructed volume of an object (e.g., the kidney 602) in the imaging space.

[0065] At operation 512, the system 400 can determine whether there has been a change in the physical coupling of the robotic instrument to the image sensor. For example, the system 400 can detect a change in the grasp state of the image sensor relative to the robotic instrument. The system 400 can determine this in any suitable manner. For example, the system 400 can determine that there has been a change in the grasp state based on kinematic information associated with the robotic instrument. For example, the kinematic information can indicate that the jaws of the robotic instrument 204-1 are open by more than a predetermined threshold amount, which can indicate a change in the grasp state. Additionally or alternatively, the system 400 can use endoscopic images, depth map images, and / or any other suitable images of the imaging space to determine whether there has been a change in the grasp position, angle, etc. of the robotic instrument relative to the image sensor. In certain additional or alternative implementations, the system 400 can detect the change based on an unexpected discrepancy between the kinematic data and the images captured by the image sensor. For example, where one of the kinematic data or the images changes while the other does not correspondingly change as expected, the system 400 can detect that there has been a change in the grasp state. For example, the system 400 can detect the change based on the kinematic data indicating movement of the robotic instrument while the images do not change in a manner expected based on the movement (e.g., this can indicate that the image sensor is not currently grasped by the robotic instrument and is thus stationary). If the answer at operation 512 is “no,” the flow returns to operation 510 and the system 400 can continue using the calibration determined at operation 508 in one or more processes associated with the image sensor. On the other hand, if the answer at operation 512 is “yes,” the flow can return to operation 502 and the system 400 can repeat operations 502-508 to determine an updated calibration of the image sensor relative to the robotic instrument.

[0066] In certain examples, the system 400 can also be configured to generate an ultrasound reconstructed volume based on the images captured by the ultrasound image sensor. The ultrasound reconstructed volume can represent a three-dimensional representation of subsurface structures of an object in the imaging space (e.g., a tissue such as a kidney or any other anatomical structure). The ultrasound reconstructed volume can be generated in any suitable manner. For example, the ultrasound reconstructed volume can be generated based on the calibration determined at operation 508.

[0067] In certain examples, the scan movements such as described herein can be performed by the system 400 in parallel with the generation of the ultrasound reconstructed volume. In such examples, the images captured during the scan movements can be used for both the calibration and for generating the ultrasound reconstructed volume.

[0068] In certain examples, the ultrasound reconstructed volume can be a previously generated ultrasound reconstructed volume. In such examples, the system 400 can be configured to adjust the previously generated ultrasound reconstructed volume based on the calibration.

[0069] In certain examples, the system 400 can be configured to generate an augmented image or a composite image of the imaging space based on the endoscopic images and the images captured by the image sensor such as the ultrasound image sensor. To do so, the system 400 can obtain the endoscopic images of the imaging space in any suitable manner. For example, the system 400 can access the images captured by the imaging device 202. The system 400 can register the images captured by the image sensor with the endoscopic images based on the calibration. Based on the registration, the system 400 can generate the augmented image or the composite image associated with the imaging space. Such augmented image or composite image can be presented to the user in any suitable manner during the procedure. For example, the augmented image or the composite image can be provided for display to the surgeon 110-1 by the display device of the user control system 104, or can be superimposed on the real-time images of the imaging space provided for display to the surgeon 110-1 by the display device of the user control system 104.

[0070] In certain examples, the system 400 can generate guidance content associated with the image sensor such as the insertable ultrasound sensor. As used herein, “guidance content” can include any content that can be used by a computer-assisted surgical system to facilitate the guided teleoperation of the image sensor in the imaging space. Generating such guidance content by the system 400 can include generating instructions and / or other guidance content for use by the computer-assisted surgical system, such as by generating computer-readable instructions for processing by the computer-assisted surgical system, and / or can include generating and / or accessing any suitable content to be presented by the computer-assisted surgical system (e.g., via a user interface associated with the computer-assisted surgical system).

[0071] Examples of guidance content may include, but are not limited to, notifications, virtual pointers, animations, instructions, auditory guidance, visual guidance, tactile feedback guidance, graphical depictions of a motion path to be followed by a robotic instrument during a scanning move, content configured to indicate the contact status of an image sensor relative to an object in the surgical space, instructions that can be used by a computer-assisted surgery system to provide guidance content, and / or any combination thereof. Examples of guidance content that can be generated by system 400 for presentation by a computer-assisted surgery system may include, but are not limited to, a motion path to be followed by a robotic instrument within an imaging space, content configured to indicate the contact status of an image sensor relative to an object in the imaging space, and / or any other generated content that can facilitate guided teleoperation of an image sensor. Specific examples of guidance content are described herein.

[0072] System 400 can generate guidance content at any suitable time.For example, system 400 can generate guidance content before a surgical procedure, during a surgical procedure, and / or at any other suitable time.

[0073] In some examples, the system 400 can generate at least some of the guidance content by accessing the guidance content from a storage device (e.g., memory 402) associated with a computer-assisted surgery system (e.g., surgical system 100). Examples of guidance content that can be accessed from the storage device can include, but are not limited to, graphical depictions of robotic instruments, graphical depictions of image sensors and / or graphical depictions of other instruments engaged by the robotic instrument (e.g., grasped by the robotic instrument), audible notifications, visual notifications, etc.

[0074] The guidance content may be generated based on any suitable parameters associated with the surgical space. For example, the guidance content may be generated based on one or more of the following: a procedural context associated with the surgical space, parameters of an image sensor (e.g., an identification type of the image sensor, a pose of the image sensor, etc.), parameters of a robotic instrument (e.g., an identification type of the robotic instrument, a pose of the robotic instrument, etc.), an indicated or predicted use or operation of the image sensor, and / or any other suitable parameter or combination of parameters.

[0075] The guidance content generated by the system 400 for presentation by a computer- assisted surgical system can be configured to be presented in any suitable manner. For example, in certain implementations, the guidance content can be configured to be presented through a user interface associated with the computer-assisted surgical system. To illustrate, the system 400 can provide guidance content for presentation through the user control system 104 of the surgical system 100, thereby facilitating remote operation of an image sensor by a user, such as the surgeon 110-1. Additionally or alternatively, the guidance content can be provided for presentation through any other suitable user interface that can be associated with the computer-assisted surgical system. For example, in certain implementations, the guidance content can be provided to a user through a user interface associated with the display monitor 114 of the assist system 106.

[0076] In certain examples, the system 400 can provide the guidance content as visual guidance to facilitate remote operation of an image sensor in an imaging space by a user of a computer-assisted surgical system, such as the surgeon 110-1. Such visual guidance can be provided in any suitable manner. For example, the system 400 can instruct the computer-assisted surgical system to provide a flashing light and / or any suitable graphical object or augmented overlay for display to the user (e.g., to the surgeon 110-1 through the user control system 104) to guide the user in remotely operating an image sensor in an imaging space.

[0077] Additionally or alternatively, the system 400 can provide the guidance content as auditory guidance to facilitate remote operation of an image sensor in a surgical space by a user of a computer-assisted surgical system. Such auditory guidance can be provided in any suitable manner. For example, the auditory notification can include a “beep,” playback of an audio clip with spoken language, and / or any other suitable auditory guidance.

[0078] Additionally or alternatively, the system 400 can provide the guidance content as haptic feedback guidance to facilitate remote operation of a robotic instrument by a user of a computer-assisted surgical system to move an image sensor. Such haptic feedback guidance can be provided in any suitable manner. For example, the system 400 can instruct the computer-assisted surgical system to vibrate one of the master controls of the user control system 104 to notify the user of a position or manner in which the image sensor is moved in the imaging space.

[0079] In certain implementations, the guidance content generated by the system 400 can facilitate the image sensor contacting the object, maintaining a predetermined amount of contact with the object, and / or maintaining a predetermined contact angle relative to a surface of the object. Thus, in such examples, the guidance content can indicate at least one of a contact pressure or a contact angle of the image sensor relative to a surface of the object, and / or can indicate one or more operations to be performed in order to obtain and / or maintain a particular contact angle and / or contact pressure between the image sensor and the object (e.g., within a particular range of contact angles and / or contact pressures).

[0080] In certain examples, the guidance content can facilitate remote operation of the image sensor to capture images during a surgical procedure prior to, during, or after calibration. For example, the system 400 can generate any suitable guidance content to be provided by the computer-assisted surgical system to facilitate remote operation of the image sensor 302-2 relative to the kidney 602 to obtain one or more ultrasound images. For example, the system 400 can obtain a first parameter indicative of a relative position between the image sensor 302-2 and the kidney 602, a second parameter indicative of a property (e.g., signal strength) of an image captured by the image sensor 302-2, a third parameter indicative of a contact pressure between the image sensor 302-2 and the kidney 602, and / or any other suitable parameter for determining whether the image sensor 302-2 is in contact with the kidney 602. If, based on such parameters, the system 400 determines that the image sensor 302-2 is not in contact with the kidney 602, the system 400 can generate any suitable guidance content indicating that the user move the image sensor 302-2 toward the kidney 602 to capture an ultrasound image. For example, the system 400 can generate a textual notification to be provided for display indicating that the user move the image sensor 302-2 toward the kidney 602. If the system 400 determines that the image sensor 302-2 is in contact with the kidney 602 but is not at a sufficient contact pressure, the system 400 can generate a visual notification in the form of, for example, a downwardly-oriented arrow icon superimposed on an image of the surgical space indicating that the user needs to increase the contact pressure of the image sensor 302-2 relative to a surface of the kidney 602. If the system 400 determines that the image sensor 302-2 is in contact with the kidney 602 but is at an excessive contact pressure, the system 400 can generate a visual notification in the form of, for example, an upwardly-oriented arrow icon indicating that the user needs to increase the contact pressure of the image sensor 302-2 relative to a surface of the kidney 602. In other implementations, the system 400 can generate any other suitable guidance content.

[0081] In certain examples, the guidance content generated by the system 400 can include a motion path to be followed by the image sensor during a scanning movement of the image sensor in the imaging space while the image sensor is engaged by the robotic instrument. In certain examples, the system 400 can be configured to generate multiple motion paths to be followed by the image sensor in the imaging space. For example, a first motion path can begin at a current position of the image sensor in the imaging space and can end at a first position on a surface of an object in the imaging space. A second motion path can begin at the first position on the surface of the object and extend to a second position on the surface of the object.

[0082] The system 400 can generate the guidance content in the form of a motion path in any suitable manner. For example, in certain implementations, the system 400 can automatically generate a motion path based on one or more parameters associated with the imaging space. For example, the system 400 can analyze Figure 6 The image 600 is shown, and based on the image 600, the presence of the kidney 602, the image sensor 302-2, and / or any other suitable parameter associated with the imaging space, the system 400 can determine that the procedural context is associated with the image sensor 302-2 that captured the image of the kidney 602. Based on such a procedural context, the system 400 can automatically generate guidance content in the form of a motion path to be followed by the robotic instrument without requiring further input from the user.

[0083] In certain alternative examples, the system 400 can generate a motion path to be followed by the image sensor for a scanning movement based on input provided by a user. To this end, the system 400 can be configured to facilitate the user in defining at least some portion of the motion path prior to the system 400 generating the motion path. The system 400 can facilitate the user in providing input to define at least a portion of the motion path in any suitable manner. For example, the system 400 can facilitate the user in defining a start position of the motion path and a stop position of the motion path. The system 400 can facilitate the user in selecting the start position and the stop position in any suitable manner. For example, the user (e.g., the surgeon 110-1) can be able to move a cursor by manipulating a master control of the user control system 104 to position a virtual pointer relative to an object in the surgical space to define the start position and the stop position. Alternatively, the user (e.g., the assistant 110-2) can define the virtual pointer by any suitable input (e.g., mouse cursor input, touch input, etc.) inputted by way of any suitable display (e.g., the display monitor 114) associated with the computer-assisted surgical system.

[0084] To illustrate an example, Figure 8A and Figure 8B8 shows images 800 (eg, images 800-1 and 800-2) that may be provided for display when a virtual pointer is used as a guide content to facilitate defining a motion path for a scanning movement. Figure 8A As shown, image 800-1 includes a first virtual pointer 802-1 indicating the starting point of the motion path and a second virtual pointer 802-2 indicating the end point of the motion path. The user can specify the position of virtual pointer 802 in any suitable manner. For example, the user can provide any suitable user input through the main controls of the user control system 104 to select the position of the first virtual pointer 802-1. The system 400 can use the depth data associated with the surgical space to project the first virtual pointer 802-1 so that it virtually appears at a depth position within the surgical space corresponding to the surface of the kidney 602. Similarly, the user can provide any suitable user input through the main controls of the user control system 104 to select the position of the second virtual pointer 802-2. The system 400 can use the depth data associated with the surgical space to project the second virtual pointer 802-2 so that it virtually appears at a depth position within the surgical space corresponding to the surface of the kidney 602.

[0085] The virtual pointer 802 may be provided for display to the user via the user interface in any suitable manner. For example, the virtual pointer may be provided for display as an overlay on a stereoscopic image displayed by the user control system 104 .

[0086] exist Figure 8A and Figure 8B In the example shown, the system 400 may use the virtual pointer 802, depth data associated with the surgical space, and / or any other suitable information as parameters to facilitate generating a path for the image sensor 302-2 to follow during the scanning movement. Figure 8B 804 is shown. This can be implemented in any suitable manner. For example, the system 400 can use the depth data to generate a smooth curve between the first virtual pointer 802-1 and the second virtual pointer 802-2 that follows the surface of the kidney 602. The system 400 can then project the generated smooth curve onto the depth map of the surgical space to generate the motion path 804. In such an example, the system 400 can be configured to discard depth outliers due to, for example, mirror reflections or other visual effects to maintain the smooth curve of the motion path 804. In doing so, the motion path 804 can facilitate optimal movement of the image sensor 302-2 relative to the kidney 602 during the scanning movement via remote operation.

[0087] exist Figure 8A and Figure 8BIn the illustrated example, the virtual pointer 802 is shown as an "X." However, it should be appreciated that in other implementations, the virtual pointer can be represented by other shapes, icons, graphical objects, etc. Additionally, for illustrative purposes, the motion path 804 is shown in the example as a dashed line. It should be appreciated that in certain implementations, guidance content provided to facilitate teleoperation of the image sensor can not include specifically displaying a motion path to the user. Figure 8B

[0088] In addition to the graphical depiction provided as part of the guidance content, the system 400 can provide additional guidance content associated with the motion path, such as by providing the additional guidance content to facilitate movement of the image sensor along the motion path at the same time. For example, in certain implementations, such additional guidance content can include providing a notification to a user of the computer-assisted surgical system requesting that the user confirm that the motion path indicated by, for example, the graphical depiction is acceptable. Such a notification can be provided to the user in any suitable manner. For example, the system 400 can access an audible notification from a storage device associated with the computer-assisted surgical system. The system 400 can instruct the computer-assisted surgical system to display the graphical depiction of the motion path and play back an audio clip with the statement "please confirm that the motion path is acceptable." The user can then visually inspect the motion path represented by the graphical depiction to determine whether the motion path is free of obstacles and / or otherwise acceptable. If the user determines that the motion path is acceptable, the user can provide any suitable response to the audio clip. For example, the user can audibly say "yes" to indicate that the motion path represented by the graphical depiction is acceptable. In such an example, the system 400 can use any suitable speech recognition algorithm to detect the user's response. Additionally or alternatively, the system 400 can access any suitable textual notification that the computer-assisted surgical system can provide for display to the user requesting that the user confirm that the motion path is acceptable.

[0089] ​Additionally or alternatively, guidance content provided by system 400 can include content that facilitates a user moving an image sensor along a motion path. For example, in certain implementations, system 400 can be configured to provide virtual guidance to facilitate a user moving an image sensor along a motion path. In certain examples, such virtual guidance can include system 400 providing haptic feedback guidance. Such haptic feedback guidance can be provided in any suitable manner. For example, such haptic feedback guidance can correspond to a virtual fixture in the surgical space, e.g., a haptic feedback tunnel, that is configured to guide an image sensor and / or control of a robotic instrument interfacing with the image sensor along a motion path in the surgical space. With such a haptic feedback tunnel, system 400 can provide haptic feedback in the form of a vibration of a master control of user control system 104 whenever an image sensor and / or a robotic instrument interfacing with the image sensor deviates from a motion path by more than a certain predefined threshold amount as a user moves the image sensor along the motion path.

[0090] In certain implementations, system 400 can provide guidance content to automatically adjust a pose of an image sensor to improve performance of a surgical procedure as a user moves a robotic instrument within a surgical space to control, e.g., a grasped image sensor. For example, in certain examples, system 400 can perform image-based visual servoing operations to automatically adjust a pose of an image sensor to improve image quality. Such image-based visual servoing operations can help ensure that an image sensor, such as an insertable ultrasound probe, maintains a desired position and / or orientation relative to an object in a surgical space. In certain examples, maintaining a desired position and / or orientation can include maintaining a pressure amount and / or a desired contact angle relative to an object in a surgical space, e.g., to capture sufficient images.

[0091] Figure 9 An example method of calibrating an image sensor relative to a robotic instrument is shown. While Figure 9 Example operations according to one implementation are shown, but other implementations can omit, add to, reorder, and / or modify Figure 9 any of the operations shown. Figure 9 One or more of the operations shown can be performed by a system such as system 400, any components included therein, and / or any implementations thereof.

[0092] At operation 902, a calibration system (e.g., calibration system 400) can obtain scan data collected during a scan movement. The scan movement can correspond to a movement in which an image sensor physically coupled to a robotic instrument moves within an imaging space. The scan data can include images captured by the image sensor during the scan movement and tracking data of the robotic instrument during the scan movement. Operation 902 can be performed in any of the ways described herein.

[0093] At operation 904, the calibration system can determine first motion data of the image sensor based on the images captured by the image sensor during the scan movement. Operation 904 can be performed in any of the ways described herein.

[0094] At operation 906, the calibration system can determine second motion data of the robotic instrument based on the tracking data of the robotic instrument. Operation 906 can be performed in any of the ways described herein.

[0095] At operation 908, the calibration system can determine a calibration of the image sensor relative to the robotic instrument in the imaging space based on the first motion data and the second motion data. Operation 908 can be performed in any of the ways described herein.

[0096] In some examples, in accordance with the principles described herein, a non-transitory computer-readable medium storing computer-readable instructions can be provided. The instructions, when executed by a processor of a computing device, can direct the processor and / or computing device to perform one or more operations including one or more of the operations described herein. Such instructions can be stored and / or transmitted using any of a variety of known computer-readable media.

[0097] A non-transitory computer-readable medium as referred to herein can include any non-transitory storage medium that participates in providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of the computing device). For example, a non-transitory computer-readable medium can include, but is not limited to, any combination of non-volatile storage media and / or volatile storage media. Illustrative non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random-access memory (“RAM”), and optical discs (e.g., compact discs, digital video discs, Blu-ray discs, etc.). Illustrative volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0098] Figure 10 An example computing device 1000 that can be specifically configured to perform one or more of the processes described herein is shown. As Figure 10As shown, computing device 1000 can include a communication interface 1002, a processor 1004, a storage 1006, and an input / output (“I / O”) module 1008 communicatively connected to each other via a communication infrastructure 1010. While Figure 10 An example computing device 1000 is shown in FIG. 10, but Figure 10 The components shown are not intended to be limiting. Additional or alternative components in other embodiments can be used. The components of the computing device 1000 shown will now be described in greater detail. Figure 10 The components of the computing device 1000 shown.

[0099] The communication interface 1002 can be configured to communicate with one or more computing devices. Examples of communication interface 1002 include, but are not limited to, a wired network interface (e.g., a network interface card), a wireless network interface (e.g., a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0100] The processor 1004 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing execution of one or more of the instructions, processes, and / or operations described herein. The processor 1004 can perform operations by executing computer-executable instructions 1012 (e.g., applications, software, code, and / or other executable data) stored in the storage 1006.

[0101] The storage 1006 can include one or more data storage media, devices, or configurations and can employ any type, form, and combination of data storage media and / or device. For example, the storage 1006 can include, but is not limited to, any combination of the non-volatile media and / or volatile media described herein. Electronic data, including the data described herein, can be temporarily and / or permanently stored in the storage 1006. For example, data representative of the computer-executable instructions 1012 configured to direct the processor 1004 to perform any of the operations described herein can be stored within the storage 1006. In some examples, the data can be arranged in one or more databases residing in the storage 1006.

[0102] The I / O module 1008 can include one or more I / O modules configured to receive user input and provide user output. The one or more I / O modules can be used to receive input for a single virtual experience. The I / O module 1008 can include any hardware, firmware, software, or combination thereof for supporting input and output functions. For example, the I / O module 1008 can include hardware and / or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0103] The I / O module 1008 can include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, the I / O module 1008 is configured to provide graphical data to a display for presentation to a user. The graphical data can be representative of one or more graphical user interfaces and / or any other graphical content as can serve a particular implementation.

[0104] In some examples, any of the systems, computing devices, and / or other components described herein can be implemented by the computing device 1000. For example, the memory 402 can be implemented by the storage device 1006, and the processor 404 can be implemented by the processor 1004.

[0105] In the foregoing description, various example implementations have been described. It is to be understood, however, that the example implementations can be practiced with variation of the specific implementations described and / or in combinations with implementation(s) not specifically described. For example, certain features of one implementation described herein can be combined with or substituted for features of another implementation described herein. Therefore, it will be appreciated that the description and drawings are illustrative and not restrictive, and that changes can be made to the description and drawings without departing from the scope of the application as claimed.

Claims

1. A system comprising: a memory for storing instructions; as well as one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: obtaining scan data collected during a scanning movement in which an image sensor physically coupled to a robotic instrument moves within an imaging space, the scan data comprising images captured by the image sensor during the scanning movement and tracking data of the robotic instrument during the scanning movement; determining first motion data of the image sensor based on the images captured by the image sensor during the scanning movement; determining second motion data of the robotic apparatus based on the tracking data of the robotic apparatus; and A calibration of the image sensor relative to the robotic instrument in the imaging space is determined based on the first motion data and the second motion data.

2. The system according to claim 1, wherein: The tracking data includes kinematic data indicating at least one of a position or an orientation of the robotic instrument during the scanning movement.

3. The system according to claim 1, wherein: Determining the first motion data includes using temporal variations of patterns in images captured by the image sensor to determine spatial relationships between the captured images.

4. The system according to claim 3, wherein: Determining the spatial relationship includes using at least one of speckle decorrelation or a machine learning operation.

5. The system according to claim 1, wherein Determining the calibration includes determining a calibration transformation between the first motion data and the second motion data.

6. The system according to claim 1, wherein: The image sensor is physically coupled to the robotic instrument by being grasped by the robotic instrument.

7. The system according to claim 6, wherein: The processing also includes: detecting a change in a grasping state of the image sensor relative to the robotic apparatus; obtaining, based on the change, additional scanning data collected during additional scanning movements in which the image sensor, physically coupled to the robotic instrument, moves within the imaging volume, the additional scanning data comprising additional images captured by the image sensor during the additional scanning movements and additional tracking data of the robotic instrument during the additional scanning movements; determining third motion data of the image sensor based on the additional images captured by the image sensor during the additional scanning movement; determining fourth motion data for the robotic apparatus based on the additional tracking data for the robotic apparatus; and An updated calibration of the image sensor relative to the robotic instrument in the imaging space is determined based on the third motion data and the fourth motion data.

8. The system of claim 1 , wherein: The image sensor is an insertable ultrasound probe; and The images include ultrasound images captured by the insertable ultrasound probe during the scanning movement.

9. The system according to claim 8, wherein: The processing also includes generating an ultrasound reconstruction volume.

10. The system according to claim 9, wherein: The ultrasound reconstruction volume is generated based on the calibration.

11. The system according to claim 9, wherein: The scanning movement is performed in parallel with the generation of the ultrasound reconstruction volume.

12. The system of claim 9, wherein: The ultrasound reconstruction volume is a previously generated ultrasound reconstruction volume; and The processing also includes adjusting the previously generated ultrasound reconstruction volume based on the calibration.

13. The system of claim 1, wherein: The processing also includes generating a motion path for the robotic instrument to follow during the scanning movement.

14. The system of claim 13, wherein: The robotic instrument is coupled to a computer-assisted surgery system; and The process also includes assisting a user of the computer-assisted surgery system in moving the robotic instrument along the motion path.

15. The system according to claim 14, wherein: The process also includes providing at least one of tactile feedback, auditory feedback, or visual feedback to assist the user in moving the robotic apparatus along the motion path.

16. The system of claim 13, wherein: The process also includes facilitating a user of the computer-assisted surgery system to define a starting location of the motion path and a stopping location of the motion path; and The generation of the motion path is based on the start position and the stop position defined by the user.

17. The system of claim 13, wherein: The process also includes directing a computer-assisted surgery system to automatically move the robotic instrument along the motion path.

18. The system of claim 1, wherein: The processing also includes: obtaining an endoscopic image of the imaging space; registering the image captured by the image sensor with the endoscopic image based on the calibration; and An enhanced image or a composite image associated with the imaging space is generated based on the registration.

19. A computer program product embodied in a non-transitory computer-readable storage medium and comprising computer instructions for performing a process comprising: obtaining scan data collected during a scanning movement in which an image sensor physically coupled to a robotic instrument moves within an imaging space, the scan data comprising images captured by the image sensor during the scanning movement and tracking data of the robotic instrument during the scanning movement; determining first motion data of the image sensor based on the images captured by the image sensor during the scanning movement; determining second motion data for the robotic apparatus based on the tracking data of the robotic apparatus; as well as A calibration of the image sensor relative to the robotic instrument in the imaging space is determined based on the first motion data and the second motion data.

20. The computer program product of claim 19, wherein: The tracking data includes kinematic data indicating at least one of a position or an orientation of the robotic instrument during the scanning movement.

21. The computer program product of claim 19, wherein: Determining the first motion data includes using temporal variations of patterns in images captured by the image sensor to determine spatial relationships between the captured images.

22. The computer program product of claim 21, wherein: Determining the spatial relationship includes using at least one of speckle decorrelation or a machine learning operation.

23. The computer program product of claim 19, wherein: Determining the calibration includes determining a calibration transformation between the first motion data and the second motion data.

24. The computer program product of claim 19, wherein: The image sensor is physically coupled to the robotic instrument by being grasped by the robotic instrument.

25. The computer program product of claim 24, wherein: The processing also includes: detecting a change in a grasping state of the image sensor relative to the robotic apparatus; obtaining, based on the change, additional scanning data collected during additional scanning movements in which the image sensor, physically coupled to the robotic instrument, moves within the imaging volume, the additional scanning data comprising additional images captured by the image sensor during the additional scanning movements and additional tracking data of the robotic instrument during the additional scanning movements; determining third motion data of the image sensor based on the additional images captured by the image sensor during the additional scanning movement; determining fourth motion data for the robotic apparatus based on the additional tracking data for the robotic apparatus; and An updated calibration of the image sensor relative to the robotic instrument in the imaging space is determined based on the third motion data and the fourth motion data.

26. The computer program product of claim 19, wherein: The image sensor is an insertable ultrasound probe; and The images include ultrasound images captured by the insertable ultrasound probe during the scanning movement.

27. The computer program product of claim 26, wherein: The processing also includes generating an ultrasound reconstruction volume.

28. The computer program product of claim 27, wherein: The ultrasound reconstruction volume is generated based on the calibration.

29. The computer program product of claim 27, wherein: The scanning movement is performed in parallel with the generation of the ultrasound reconstruction volume.

30. The computer program product of claim 27, wherein: The ultrasound reconstruction volume is a previously generated ultrasound reconstruction volume; and The processing also includes adjusting the previously generated ultrasound reconstruction volume based on the calibration.

31. The computer program product of claim 19, wherein: The processing also includes generating a motion path for the robotic instrument to follow during the scanning movement.

32. The computer program product of claim 31 , wherein: The robotic instrument is coupled to a computer-assisted surgery system; and The process also includes assisting a user of the computer-assisted surgery system in moving the robotic instrument along the motion path.

33. The computer program product of claim 32, wherein: The process also includes providing at least one of tactile feedback, auditory feedback, or visual feedback to assist the user in moving the robotic apparatus along the motion path.

34. The computer program product of claim 31 , wherein: The process also includes facilitating a user of the computer-assisted surgery system to define a starting location of the motion path and a stopping location of the motion path; and The generation of the motion path is based on the start position and the stop position defined by the user.

35. The computer program product of claim 31 , wherein: The process also includes directing a computer-assisted surgery system to automatically move the robotic instrument along the motion path.

36. The computer program product of claim 19, wherein: The processing also includes: obtaining an endoscopic image of the imaging space; registering the image captured by the image sensor with the endoscopic image based on the calibration; and An enhanced image or a composite image associated with the imaging space is generated based on the registration.

37. A method comprising: obtaining, by a calibration system, scan data collected during a scanning movement in which an image sensor physically coupled to a robotic instrument moves within an imaging space, the scan data comprising images captured by the image sensor during the scanning movement and tracking data of the robotic instrument during the scanning movement; determining, by the calibration system and based on the images captured by the image sensor during the scanning movement, first motion data for the image sensor; determining, by the calibration system and based on the tracking data of the robotic instrument, second motion data for the robotic instrument; as well as A calibration of the image sensor relative to the robotic instrument in the imaging space is determined by the calibration system and based on the first motion data and the second motion data.

38. The method according to claim 37, wherein The tracking data includes kinematic data indicating at least one of a position or an orientation of the robotic instrument during the scanning movement.

39. The method of claim 37, wherein: Determining the first motion data includes using temporal variations of patterns in images captured by the image sensor to determine spatial relationships between the captured images.

40. The method of claim 39, wherein Determining the spatial relationship includes using at least one of speckle decorrelation or a machine learning operation.

41. The method of claim 37, wherein: Determining the calibration includes determining a calibration transformation between the first motion data and the second motion data.

42. The method of claim 37, wherein: The image sensor is physically coupled to the robotic instrument by being grasped by the robotic instrument.

43. The method of claim 42, further comprising: detecting, by the calibration system, a change in a grasping state of the image sensor relative to the robotic instrument; obtaining, by the calibration system and based on the change, additional scanning data collected during additional scanning movements in which the image sensor, physically coupled to the robotic instrument, moves within the imaging volume, the additional scanning data comprising additional images captured by the image sensor during the additional scanning movements and additional tracking data of the robotic instrument during the additional scanning movements; determining, by the calibration system and based on the additional images captured by the image sensor during the additional scanning movement, third motion data for the image sensor; determining, by the calibration system and based on the additional tracking data of the robotic instrument, fourth motion data for the robotic instrument; as well as An updated calibration of the image sensor relative to the robotic instrument in the imaging space is determined by the calibration system and based on the third motion data and the fourth motion data.

44. The method of claim 37, wherein: The image sensor is an insertable ultrasound probe; and The images include ultrasound images captured by the insertable ultrasound probe during the scanning movement.

45. The method of claim 44, further comprising generating an ultrasound reconstruction volume by the calibration system.

46. ​​The method of claim 45, wherein The ultrasound reconstruction volume is generated based on the calibration.

47. The method of claim 45, wherein The scanning movement is performed in parallel with the generation of the ultrasound reconstruction volume.

48. The system of claim 45, wherein: The ultrasound reconstruction volume is a previously generated ultrasound reconstruction volume; and The method also includes adjusting the previously generated ultrasound reconstruction volume based on the calibration.

49. The method of claim 37, further comprising generating a motion path for the robotic instrument to follow during the scanning movement.

50. The method of claim 49, wherein: The robotic instrument is coupled to a computer-assisted surgery system; and The method also includes assisting a user of the computer-assisted surgery system in moving the robotic instrument along the motion path.

51. The method of claim 50, further comprising providing at least one of tactile feedback, auditory feedback, or visual feedback to assist the user in moving the robotic machine along the motion path.

52. The method of claim 49, wherein: The method further includes facilitating a user of the computer-assisted surgery system to define a starting location of the motion path and a stopping location of the motion path; and The generation of the motion path is based on the start position and the stop position defined by the user.

53. The method of claim 49, further comprising directing a computer-assisted surgery system to automatically move the robotic instrument along the motion path.

54. The method of claim 37, further comprising: obtaining an endoscopic image of the imaging space; registering the image captured by the image sensor with the endoscopic image based on the calibration; as well as An enhanced image or a composite image associated with the imaging space is generated based on the registration.

Images