Systems and methods for robotic surgical control and navigation - Patents.com

Abstract

Systems and methods are disclosed for controlling and navigating a robot in a surgical environment. The systems and methods described herein provide techniques for adjusting the position of a robot, such as a surgical robot, in response to detecting patient movement using image-based tracking techniques. Techniques are provided that allow a robotic control system to adjust the position of a surgical robot in real-time or near real-time in response to measurements from sensors coupled to the robot or the patient within a surgical environment. Techniques are disclosed for initiating a cooperative control state of a surgical robot in response to detecting image alignment errors, sensor measurements, or other conditions.

Description

[Technical field]

[0001] The present disclosure relates generally to the field of surgical robotic navigation and control for invasive and non-invasive surgical procedures. (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 241,285, filed September 7, 2021, and claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 355,497, filed June 24, 2022, the contents of each of which are incorporated by reference herein in their entirety for all purposes. [Background technology]

[0002] Positioning surgical tools within a patient's body can be difficult. Surgical robots can be provided in the surgical environment to assist in performing procedures. Summary of the Invention

[0003] The present disclosure generally relates to the field of surgical robot navigation and control for invasive and non-invasive surgical procedures. The present solution provides techniques for tracking patient movement within a surgical environment and adjusting or navigating a surgical robot to perform a given procedure while compensating for patient movement. The techniques described herein can be implemented using various movement detection techniques, such as patient tracking or torque sensing techniques. The present disclosure further provides techniques for initiating cooperative control of a surgical robot by switching to manual control over a surgical tool during a surgical procedure in response to various conditions. The present solution can further be used for non-invasive surgical navigation, such as transcranial magnetic stimulation (TMS) and focused ultrasound (FUS), by combining the image guidance of the present solution with surgical instruments. The present solution enables robotic control of surgical instruments for both invasive and non-invasive cranial procedures and utilizes real-time alignment to target highlighted locations of interest.

[0004] At least one aspect of the present disclosure is directed to a method for controlling a robot using image-based tracking technology. The method may include accessing a 3D (three-dimensional) point cloud corresponding to a surgical environment and a patient, the 3D point cloud having a reference coordinate system. The method may include determining a position of the surgical robot within the reference coordinate system of the 3D point cloud. The method may include detecting a change in the position of the patient based on a corresponding change in the position of one or more points in the 3D point cloud. The method may include generating instructions to modify the position of the surgical robot based on the change in the position of the one or more points in response to detecting the change in the position of the patient.

[0005] In some implementations, determining the position of the surgical robot within the reference coordinate system may include calibrating the surgical robot using a calibration technique. In some implementations, the surgical robot further comprises a display positioned over the surgical site in the surgical environment. In some implementations, the method may include presenting an image captured by a capture device mounted on the surgical robot. In some implementations, the surgical robot may include an attachment that receives a surgical tool. In some implementations, determining the position of the surgical robot may include determining a position of the surgical tool.

[0006] In some implementations, the method may include navigating the surgical robot along a predetermined path in a reference coordinate system. In some implementations, navigating the surgical robot may include adjusting a position of the surgical robot according to a predetermined trajectory in the reference coordinate system. In some implementations, navigating the surgical robot may include periodically determining whether a change in a patient position satisfies a threshold. In some implementations, navigating the surgical robot may include adjusting a position of the surgical robot according to the predetermined trajectory and the change in the patient position in response to determining that the change in the patient position satisfies a threshold.

[0007] In some implementations, determining the position of the surgical robot is based on infrared tracking technology. In some implementations, the surgical robot comprises one or more markers. In some implementations, determining the position of the surgical robot based on infrared tracking technology includes detecting a respective position of each of the one or more markers. In some implementations, detecting a change in the patient's position includes comparing a point of the 3D point cloud to a second point of a second 3D point cloud captured after the 3D point cloud. In some implementations, detecting a change in the patient's position includes determining that a distance between the point and the second point exceeds a predetermined threshold.

[0008] At least one aspect of the present disclosure is directed to a system for controlling a robot using image-based tracking techniques. The system may include one or more processors coupled to a non-transitory memory. The system may have access to a 3D point cloud corresponding to a surgical environment and a patient. The 3D point cloud may have a reference coordinate system. The system may determine a position of the surgical robot within the reference coordinate system of the 3D point cloud. The system may detect a change in the position of the patient based on a corresponding change in the position of one or more points in the 3D point cloud. The system may generate instructions to modify the position of the surgical robot based on the change in the position of the one or more points in response to detecting the change in the position of the patient.

[0009] In some implementations, the system can determine a position of the surgical robot within a reference coordinate system by performing operations including calibrating the surgical robot using a calibration technique. In some implementations, the surgical robot further comprises a display positioned over a surgical site within the surgical environment. In some implementations, the system can present images captured by a capture device mounted on the surgical robot. In some implementations, the surgical robot can include an attachment that receives a surgical tool. In some implementations, the system can determine a position of the surgical tool. In some implementations, the system can navigate the surgical robot along a predetermined path within the reference coordinate system.

[0010] In some implementations, to navigate the surgical robot, the system can adjust the position of the surgical robot according to a predetermined trajectory in a reference coordinate system. In some implementations, to navigate the surgical robot, the system can periodically determine whether a change in the patient's position meets a threshold. In some implementations, to navigate the surgical robot, the system can adjust the position of the surgical robot according to the predetermined trajectory and the change in the patient's position in response to determining that the change in the patient's position meets a threshold.

[0011] In some implementations, the system can determine the position of the surgical robot based on infrared tracking technology. In some implementations, the surgical robot includes one or more markers. In some implementations, the system can detect a respective position of each of the one or more markers. In some implementations, the system can detect a change in the patient's position by performing an operation including comparing a point of the 3D point cloud with a second point of a second 3D point cloud captured after the 3D point cloud. In some implementations, the system can detect a change in the patient's position by performing an operation including determining that a distance between the point and the second point exceeds a predetermined threshold.

[0012] At least one other aspect of the present disclosure is directed to a method of controlling a robot based on torque sensing technology. The method may include identifying a set of measurements captured by one or more torque sensors in a surgical environment including a patient. The method may include determining a position of the surgical robot within the surgical environment. The method may include detecting a position modification condition based on the set of measurements captured by the one or more torque sensors. The method may include generating instructions to modify a position of the surgical robot based on the set of measurements in response to detecting the position modification condition.

[0013] In some implementations, the one or more torque sensors are coupled to a patient. In some implementations, the method may include detecting a position correction condition and further includes determining that patient movement meets a predetermined threshold. In some implementations, the one or more torque sensors are coupled to a surgical robot. In some implementations, the method may include detecting a position correction condition and further includes determining that a collision with the surgical robot has occurred based on the set of measurements. In some implementations, the one or more torque sensors are coupled to a surgical robot.

[0014] In some implementations, the method may include detecting a position correction condition and further includes determining that a position of the surgical robot has deviated from a predetermined trajectory based on the set of measurements. In some implementations, the surgical robot may include a display positioned over the surgical site in the surgical environment. In some implementations, the method may include presenting a view of the patient in the surgical environment on the display. In some implementations, the surgical robot may include an attachment that receives a surgical tool. In some implementations, determining a position of the surgical robot may include determining a position of the surgical tool.

[0015] In some implementations, the one or more torque sensors comprise at least one of an accelerometer, a gyroscope, or an inertial measurement unit (IMU). In some implementations, determining the position of the surgical robot is based on infrared tracking technology. In some implementations, the surgical robot comprises one or more markers. In some implementations, the method may include determining a respective position of each of the one or more markers. In some implementations, generating instructions to modify the position of the surgical robot may include generating instructions to move the surgical robot according to the movement of the patient.

[0016] At least one other aspect of the present disclosure is directed to a system for controlling a robot based on torque sensing techniques. The system may include one or more processors coupled to a non-transitory memory. The system may identify a set of measurements captured by one or more torque sensors in a surgical environment including a patient. The system may determine a position of the surgical robot within the surgical environment. The system may detect a position modification condition based on the set of measurements captured by the one or more torque sensors. The system may generate instructions to modify the position of the surgical robot based on the set of measurements in response to detecting the position modification condition.

[0017] In some implementations, the one or more torque sensors are coupled to a patient. In some implementations, the system can detect a position correction condition by performing an operation including determining that a patient movement meets a predetermined threshold. In some implementations, the one or more torque sensors are coupled to a surgical robot. In some implementations, the system can detect a position correction condition by performing an operation including determining that a collision with the surgical robot has occurred based on a set of measurements. In some implementations, the one or more torque sensors are coupled to a surgical robot. In some implementations, the system can detect a position correction condition by performing an operation including determining that a position of the surgical robot has deviated from a predetermined trajectory based on a set of measurements.

[0018] In some implementations, the surgical robot further comprises a display positioned over the surgical site in the surgical environment. In some implementations, the system can present a view of the patient in the surgical environment on the display. In some implementations, the surgical robot comprises an attachment that receives a surgical tool. In some implementations, the system can determine a position of the surgical robot by performing an operation that includes determining a position of the surgical tool. In some implementations, the one or more torque sensors can include at least one of an accelerometer, a gyroscope, or an inertial measurement unit (IMU).

[0019] In some implementations, the system can determine the position of the surgical robot based on infrared tracking technology. In some implementations, the surgical robot includes one or more markers. In some implementations, the system can determine a respective position of each of the one or more markers. In some implementations, the system can generate instructions to modify the position of the surgical robot by performing operations including generating instructions to move the surgical robot according to patient movements.

[0020] At least one other aspect of the present disclosure is directed to a method of initiating cooperative control of a robot in response to a detected condition. The method may include controlling a position of the surgical robot within a surgical environment including a patient. The method may include detecting a cooperative control condition of the surgical robot based on the condition of the surgical environment. The method may include generating instructions for providing manual control of the surgical robot in response to detecting the cooperative control condition.

[0021] In some implementations, the method may include accessing a 3D (three-dimensional) point cloud corresponding to a patient in a surgical environment. In some implementations, detecting the cooperative control condition may include determining that one or more points of the 3D point cloud satisfy a movement condition. In some implementations, the method may include identifying a set of torque measurements captured from one or more torque sensors coupled to the patient. In some implementations, detecting the cooperative control condition further includes determining that the set of torque measurements satisfy a patient movement condition. In some implementations, detecting the cooperative control condition may include detecting an error condition in an image-to-patient registration process. In some implementations, detecting the cooperative control condition may include receiving an interaction with a button corresponding to the cooperative control condition.

[0022] In some implementations, controlling the position of the surgical robot includes identifying one or more pre-determined trajectories for an instrument to perform a surgical procedure. In some implementations, identifying one or more pre-determined trajectories for the instrument includes receiving a selection of the one or more pre-determined trajectories via user input. In some implementations, controlling the position of the surgical robot includes navigating the surgical robot along the one or more pre-determined trajectories. In some implementations, controlling the position of the surgical robot includes navigating the surgical robot according to patient movement. In some implementations, detecting the cooperative control condition includes identifying a set of torque measurements captured from one or more torque sensors coupled to the surgical robot. In some implementations, detecting the cooperative control condition includes determining that the set of torque measurements satisfies a patient movement condition.

[0023] At least one other aspect of the present disclosure is directed to a method of initiating cooperative control of a robot in response to a detected condition. The system may include one or more processors coupled to a non-transitory memory. The system may control a position of the surgical robot within a surgical environment including a patient. The system may detect a cooperative control condition of the surgical robot based on the condition of the surgical environment. The system may generate instructions for providing manual control of the surgical robot in response to detecting the cooperative control condition.

[0024] In some implementations, the system can access a 3D (three-dimensional) point cloud corresponding to a patient in a surgical environment. In some implementations, the system can detect the cooperative control condition by performing an operation including determining that one or more points of the 3D point cloud satisfy a movement condition. In some implementations, the system can identify a set of torque measurements captured from one or more torque sensors coupled to the patient. In some implementations, the system can detect the cooperative control condition by performing an operation including determining that the set of torque measurements satisfy a patient movement condition.

[0025] In some implementations, the system can detect the cooperative control condition by performing an operation including detecting an error condition in an image-to-patient registration process. In some implementations, the system can detect the cooperative control condition by performing an operation including receiving an interaction with a button corresponding to the cooperative control condition. In some implementations, the system can control the position of a surgical robot by performing an operation including identifying one or more predetermined trajectories for an instrument to perform a surgical procedure.

[0026] In some implementations, the system can identify one or more pre-defined trajectories for the instrument by performing an operation including receiving, via a user input, a selection of one or more pre-defined trajectories. In some implementations, the system can control a position of the surgical robot by performing an operation including navigating the surgical robot along one or more pre-defined trajectories. In some implementations, the system can control a position of the surgical robot by performing an operation including navigating the surgical robot according to patient movement. In some implementations, the system can detect a cooperative control condition by performing an operation including identifying a set of torque measurements captured from one or more torque sensors coupled to the surgical robot. In some implementations, the system can detect a cooperative control condition by performing an operation including determining that the set of torque measurements satisfies a patient movement condition.

[0027] Various aspects generally relate to systems and methods for real-time multi-modality image alignment using 3D (three-dimensional) image data, which may be implemented with sub-millimeter accuracy without markers. 3D images, including scans such as CT or MRI, may be directly registered onto an object, such as a patient's body, captured in real-time using one or more capture devices. This allows certain scan information, such as internal tissue information, to be displayed in real-time along with a point cloud representation of the object. This may be beneficial for surgical procedures that otherwise utilize manual processes to orient instruments within the same reference coordinate system as the CT scan. Instruments may be tracked, instrument trajectories may be plotted, and targets may be highlighted on the scan. The solution can provide real-time sub-millimeter registration for a variety of applications, such as aligning depth capture information with medical scans (e.g., for surgical navigation), aligning depth capture information with CAD models (e.g., for manufacturing and troubleshooting), aligning and fusing multiple medical imaging modalities (e.g., MRI and CT, CT and 3D ultrasound, MRI and 3D ultrasound), aligning multiple CAD models (e.g., to determine differences between the models), and fusing depth capture data from multiple image capture devices.

[0028] The solution may be implemented for image-guided procedures in a variety of environments, including operating rooms, outpatient settings, CT suites, ICUs, and emergency rooms. The solution may be used for neurosurgical applications such as CSF diversion procedures (e.g., external ventricular placement and VP shunt placement), brain tumor resection and biopsy, and electrode placement. The solution may be used for interventional radiology such as abdominal and lung biopsies, ablation, aspiration, and drainage. The solution may be used for orthopedic procedures such as spinal fusion. The solution may be used for non-invasive surgical navigation such as transcranial magnetic stimulation (TMS) and focused ultrasound (FUS) by combining the image guidance of the solution with surgical instruments. The solution enables robotic control of surgical instruments for non-invasive cranial procedures and utilizes real-time registration to target highlighted locations of interest.

[0029] At least one aspect of the present disclosure relates to a method of delivering a treatment to a location of interest through a surgical instrument. The method can be executed by one or more processors of a data processing system. The method can include registering, by the one or more processors, a 3D medical image positioned relative to a reference coordinate system. The method can include receiving tracking data of a surgical instrument being used to perform the treatment. The method can include determining a relative position of the surgical instrument to the location of interest in a reference coordinate system associated with the first point cloud and the 3D medical image. The method can include tracking target movement and adjusting the surgical instrument to remain aligned with the location of interest. The method can include delivering the treatment to the location of interest through the surgical instrument. The method can include receiving a threshold for the treatment and a parameter detected during the treatment. The method can include having the surgical instrument terminate the treatment in response to the parameter meeting the threshold. In some implementations of the method, the location of interest is on a surface of the subject's head.

[0030] In some implementations of the method, transforming the tracking data from the surgical instrument can include generating the transformed tracking data using the first reference coordinate system. In some implementations of the method, rendering the transformed tracking data can be included in rendering the first point cloud and the 3D medical image.

[0031] In some implementations of the method, generating movement instructions for the surgical instrument may be based on the first point cloud, the 3D medical image, and the location of interest. In some implementations of the method, transmitting the movement instructions may include the surgical instrument. In some implementations of the method, displaying a highlighted area of ​​the location of interest may be included within the rendering of the 3D medical image and the first point cloud. In some implementations of the method, determining a distance of an object represented in the 3D medical image from the capture device may be responsible, at least in part, for generating the first point cloud.

[0032] In some implementations of the method, causing the surgical instrument to terminate energy emission can include the location of interest not being within a reference coordinate system. In some implementations of the method, causing the surgical instrument to terminate energy emission can include the target movement exceeding the surgical instrument movement for the procedure to the location of interest.

[0033] In some implementations of the method, allowing the surgical instrument to contact the target may include responding to target movement by combining the registered 3D medical image and the first point cloud with torque sensing. In some implementations of the method, receiving tracking data from the surgical instrument may include applying a force to keep the surgical instrument in contact with the surface. In some implementations of the method, transforming the tracking data from the procedure may be relative to the detected target movement and may include maintaining the force initially applied to the surface.

[0034] At least one other aspect of the present disclosure relates to a system for delivering a treatment to a location of interest through a surgical instrument. The system can register, by one or more processors, a 3D medical image positioned relative to a reference coordinate system. The system can receive, by one or more processors, tracking data of the surgical instrument and determine a relative position of the surgical instrument relative to the location of interest in a reference coordinate system associated with the first point cloud and the 3D medical image. The system can track target movement and adjust the surgical instrument to remain aligned with the location of interest by one or more processors based on the relative position. The system can deliver, by one or more processors, a treatment to a location of the instrument through the surgical instrument and receive thresholds for the treatment and parameters detected during the treatment. The system can cause the surgical instrument to terminate the treatment responsive to the parameters meeting the thresholds by one or more processors. In some implementations of the system, the location of interest can be on a surface of a subject's head.

[0035] In some implementations of the system, the system can transform tracking data from the surgical instrument to a first reference coordinate system to generate transformed tracking data, In some implementations of the system, the system can render the transformed tracking data in a rendering of the first point cloud and the 3D medical image.

[0036] In some implementations of the system, the system can generate movement instructions for the surgical instrument based on the first point cloud, the 3D medical image, and the location of interest. In some implementations of the system, the system can transmit the movement instructions to the surgical instrument. In some implementations of the system, the system can display a highlighted region in a rendering of the 3D medical image and the first point cloud corresponding to the location of interest. In some implementations of the system, the system can be represented in the 3D medical image from a capture device that is at least partially responsible for determining the distance of the object and generating the first point cloud.

[0037] In some implementations of the system, the system can cause the surgical instrument to terminate energy emission if the location of interest is not within the reference coordinate system. In some implementations of the system, the system can cause the surgical instrument to terminate energy emission if the target movement exceeds the surgical instrument movement for the procedure to the location of interest.

[0038] In some implementations of the system, the system can allow the surgical instrument to contact the target and can also respond to target movement. In some implementations of the system, the system can combine the registered 3D medical image and the first point cloud with torque sensing. In some implementations of the system, the system can receive tracking data from the surgical instrument and apply a force to keep the surgical instrument in contact with the surface. In some implementations of the system, the system can translate the tracking data from the surgical instrument to the detected target movement and maintain the force originally applied to the surface.

[0039] These and other aspects and implementations are described in detail below. The above information and the following detailed description include illustrative examples of the various aspects and implementations and provide an overview or framework for understanding the nature and characteristics of the claimed aspects and implementations. The drawings provide an illustration and further understanding of the various aspects and implementations and are incorporated in and constitute a part of this specification. It will be readily understood that the aspects can be combined and that features described in the context of one aspect of the invention can be combined with other aspects. The aspects can be implemented in any convenient form. For example, by a suitable computer program that can be carried on a suitable carrier medium (computer readable medium), which can be a tangible carrier medium (e.g., disk) or an intangible carrier medium (e.g., communication signal). The aspects can also be implemented using a suitable apparatus, which can take the form of a programmable computer executing a computer program configured to implement the aspect. As used in this specification and claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0040] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For purposes of clarity, not every component may be labeled in every drawing. [Brief description of the drawings]

[0041] [Figure 1A] FIG. 1 is a perspective view of an exemplary image processing system according to one or more implementations. [Figure 1B] FIG. 1 is a perspective view of an exemplary image processing system according to one or more implementations.

[0042] [Diagram 2] FIG. 1 is a block diagram of an image processing system capable of monitoring the position of a patient and a robot, according to one or more implementations.

[0043] [Diagram 3] FIG. 1 is a perspective view of an exemplary robotic control system according to one or more implementations.

[0044] [Figure 4] FIG. 1 is a block diagram of a robotic control system capable of controlling a surgical robot based on patient tracking, according to one or more implementations.

[0045] [Diagram 5] FIG. 1 is a flow diagram of an exemplary method for controlling a surgical robot based on patient tracking, according to one or more implementations.

[0046] [Figure 6] FIG. 1 is a block diagram of a robotic control system capable of controlling a surgical robot based on torque sensing technology, according to one or more implementations.

[0047] [Figure 7] FIG. 1 is a flow diagram of an exemplary method for controlling a surgical robot based on torque sensing technology, according to one or more implementations.

[0048] [Figure 8] FIG. 1 is a block diagram of a robotic control system capable of initiating cooperative control of a surgical robot in response to a detected condition, according to one or more implementations.

[0049] [Figure 9] FIG. 1 is a flow diagram of an example method for initiating cooperative control of a surgical robot in response to a detected condition, according to one or more implementations.

[0050] [Figure 10] FIG. 1 is a block diagram of an image processing system including a surgical instrument, according to one or more implementations.

[0051] [Figure 11]FIG. 1 is a flow diagram of a method for real-time non-invasive surgical navigation, according to one or more implementations.

[0052] [Figure 12A] FIG. 1 is a block diagram of an exemplary computing environment in accordance with one or more implementations. [Figure 12B] FIG. 1 is a block diagram of an exemplary computing environment in accordance with one or more implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0053] Below are detailed descriptions of various concepts related to techniques, approaches, methods, apparatus, and systems for managing surgical tools with integrated display devices, as well as implementations thereof. The various concepts introduced above and described in more detail below may be implemented in any of a number of ways, as the concepts described are not limited to any particular manner of implementation. Examples of specific implementations and applications are provided primarily for purposes of illustration.

[0054] For purposes of reading the following description of the various implementations, the following descriptions of sections of this specification and their respective contents may be useful.

[0055] Section A describes hardware components that may implement the robotic control techniques described herein.

[0056] Section B describes techniques for controlling a surgical robot based on patient tracking techniques.

[0057] Section C describes techniques for controlling a surgical robot based on torque sensing techniques.

[0058] Section D describes techniques for initiating cooperative control of a surgical robot in response to detected conditions.

[0059] Section E describes techniques for real-time non-invasive navigation.

[0060] Section F describes computing environments that may be useful for practicing the implementations described herein.

[0061] A. Hardware Components and System Architecture The image tracking, torque sensing, and robotic control techniques described herein can be performed in real time within a surgical environment, for example, during a cranial surgical procedure. Before discussing in detail the specific techniques for initiating image registration based surgical robotic control, torque sensing based surgical robotic control, and coordinated control responses to conditions detected in the surgical environment, it is useful to describe certain components located within the surgical environment in which such techniques may operate.

[0062] 1A, 1B, and 2 show an image processing system 100. The image processing system 100 may include one or more image capture devices 104, such as a 3D (three-dimensional) camera. The camera may be a visible light camera (e.g., color or black and white), an infrared camera (e.g., IR sensor 220, etc.), or a combination thereof. Each image capture device 104 may include one or more lenses 204. In some implementations, the image capture device 104 may include a camera for each lens 204. The image capture device 104 may be selected or designed to be at a predetermined resolution and / or to have a predetermined field of view. The image capture device 104 may have a resolution and field of view for detecting and tracking objects. The image capture device 104 may have a pan, tilt, or zoom mechanism. The image capture device 104 may have a pose that corresponds to the position and orientation of the image capture device 104. The image capture device 104 may be a depth camera. The image capture device 104 may be a KINECT manufactured by Microsoft Corporation.

[0063] Light for an image captured by image capture device 104 is received through one or more lenses 204. Image capture device 104 may include sensor circuitry, including but not limited to, charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) circuitry, that can detect the light received through the one or more lenses 204 and generate an image 208 based on the received light.

[0064] The image capture device 104 can provide the image 208 to the processing circuit 212, for example, via a communication bus. The image capture device 104 can provide the image 208 with a corresponding timestamp that can facilitate synchronization of the image 208 when image processing is performed on the image 208. The image capture device 104 can output a 3D image (e.g., an image with depth information). The image 208 can include a number of pixels, with each pixel assigned spatial position data (e.g., horizontal, vertical, and depth data), intensity or brightness data, and / or color data. For example, when captured within a surgical environment including a surgical robot operating on a patient, the image 208 can include pixels that represent portions of the surgical robot, such as the tool end of the surgical robot, or markers positioned on the surgical robot, among others. In an implementation in which the image capture device 104 is a 3D camera, the surgical robot and the patient can be mapped to corresponding 3D point clouds in the reference coordinate system of the image capture device 104. The 3D point cloud may be stored, for example, in a memory of the processing circuit 212 and provided to a robotic controller system described herein. In some implementations, the processing circuit 212 can perform image-to-patient registration (e.g., registration between a CT image of the patient and a 3D point cloud representing the patient) in addition to tracking the 3D point cloud corresponding to the patient.

[0065] Each image capture device 104 can be coupled to the platform 112, such as via one or more arms or other support structures, and can be communicatively coupled to the processing circuit 212. The platform 112 can be a cart, which can include wheels for mobility and various support surfaces for supporting devices used with the platform 112. In some implementations, the platform 112 is a fixed structure without wheels, such as a table. In some implementations, the components coupled to the platform 112 can be modular and removable such that they can be replaced with other tracking or computing devices as needed.

[0066] The platform 112 can support processing hardware 116 (described in more detail below in conjunction with FIG. 2), including at least a portion of the processing circuit 212, as well as a user interface 120. The user interface 120 can be any type of display or screen as described herein and can be used, for example, to display a three-dimensional rendering of an environment captured by the image capture device 104. The images 208 can be processed by the processing circuit 212 for presentation via the user interface 120. As described above, the images 208 can include representations of a patient or a surgical tool positioned within a surgical environment captured by the image capture device 104. In some implementations, the processing circuit 212 can utilize one or more image classification techniques (e.g., deep neural networks, light detection, color detection, etc.) to determine the location (e.g., pixel location, 3D point location, etc.) of a surgical robot, a surgical tool, or a patient, as described herein.

[0067] The processing circuitry 212 can incorporate features of the computing device 1000 described with reference to Figures 12A and 12B. For example, the processing circuitry 212 can include a processor(s) and a memory. The processor can be implemented as a dedicated processor, an application specific integrated circuit (ASIC), one or more field programmable gate arrays (FPGAs), a group of processing components, or other suitable electronic processing components. The memory is one or more devices (e.g., RAM, ROM, flash memory, hard disk storage, etc.) for storing data and computer code for completing and facilitating the various user or client processes, layers, and modules described in this disclosure. The memory may be or include volatile or non-volatile memory and may include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures of the features described herein. The memory can be communicatively connected to the processor and may include computer code or instruction modules for performing one or more operations described herein. The memory may include various circuits, software engines, and / or modules that cause the processor to perform the operations described herein.

[0068] Some portions of the processing circuitry 212 may be provided by one or more devices separate from the platform 112. For example, one or more servers (e.g., as described with reference to Figures 12A and 12B), cloud computing systems, or mobile devices may be used to perform the various image processing techniques described herein.

[0069] The image processing system 100 may include a communication circuit 216. The communication circuit 216 may implement features of the computing device 1000 described with reference to FIGS. 12A and 12B, such as a network interface 1218. The communication circuit 216 may be used to communicate information regarding the location of a 3D point cloud corresponding to a patient within a surgical environment, which may be used within the processing components described herein to navigate a surgical robot. In some implementations, the communication circuit 216 may be used to communicate with a robot control system 405, 605, or 805. In some implementations, the image processing system 100 may implement one or more of the functions of any of the robot control systems 405, 605, or 805 described herein. The communication circuit 216 may be any type of input / output interface capable of communicating information between the image processing system 100 (or a component thereof) and one or more components, devices, or systems, including any component, device, or system described herein.

[0070] The image processing system 100 may include one or more infrared (IR) sensors 220. The IR sensor 220 may detect IR signals from various devices in the environment surrounding the image processing system 100. For example, the IR sensor 220 may be used to detect IR signals from an IR emitter that may be coupled to a tracked feature in the surgical environment, such as a portion of a patient, a portion of a surgical robot, or a tool end coupled to a surgical robot, among others. The IR sensor 220 may be communicatively coupled to other components of the image processing system 100 such that the components of the image processing system 100 may utilize the IR signals in appropriate operations in the techniques described herein.

[0071] 3, an exemplary robotic control system 300 is shown, according to one or more implementations. The robotic system 300 may include a cart 305, which may include one or more of the robotic control systems 405, 605, or 805 as described herein in connection with FIGS. 4, 6, and 8. Mounted on the cart is a robotic arm 310, which may be in communication with or controlled by the robotic control system 405, 605, or 805. A close-up view of the tracked end effector 315 is shown in close-up 312. As shown in close-up 312, the tracked end effector 315 of the robotic arm 310 may include a screen 325, an instrument holder 330, one or more buttons 335, and a tracked instrument 340 (e.g., shown here as a catheter guide including one or more markers) connected to the instrument holder 330. The instrument holder 330 may be a generic attachment or connector on the robotic arm 310 for the tracked instrument 340. For example, the instrument holder can allow the robotic arm 310 to be used with any type of tracked instrument 340.

[0072] The cart 305 may be similar to the platform 112 and may include any of its structure or functionality. The cart 305 may include wheels for mobility and may support other devices shown in the robotic system 300. The robotic arm 310 may be any type of robotic arm that can be navigated in 3D space according to instructions provided by the robotic control systems 405, 605, and 805 described herein. The robotic arm 310 may be automatically controlled according to a predefined path in 3D space, or may be controlled in a partially automated state where a surgeon or other medical professional can move the robotic arm 310 within predefined boundaries established through software. In some implementations, in response to various conditions described herein, the robotic arm 310 can enter a "collaborative mode" in which the surgeon is provided with manual control over the position and orientation of the robotic arm 310 (e.g., tracked instrument 340, etc.). Such techniques are described in more detail herein in connection with Section D. In some implementations, the robotic arm 310 may be an M0609 robotic arm manufactured by DOOSAN ROBOTICS. The robotic arm 310 may be a computer-controlled electromechanical articulated arm.

[0073] The tracked end effector 315 includes a screen 325, an instrument holder 330, and a tracked instrument 340. In some implementations, the tracked end effector 315 includes one or more buttons 335 that, when interacted with, can enable the surgeon to navigate different points or sequences of a 3D path to perform a surgical procedure. In some implementations, one or more of the buttons 335 can initiate a collaborative control mode, allowing the surgeon to have full control over the position and orientation of the robotic arm 310. The robotic arm 310 can receive instructions, including movement instructions, from, for example, a computing device described herein via a communication interface. For example, the movement instructions can be instructions to cause the robotic arm 310 to modify the position of the tracked instrument (e.g., by actuating one or more joints according to its internal programming, etc.) to a desired position or orientation within the surgical environment. In addition, the surgical robot may transmit messages to a computing device described herein to provide information related to the status of the robot arm 310 (e.g., whether the robot arm is in an automatic navigation mode, whether the robot arm is in a collaborative mode, etc.).

[0074] The tracked instrument 340 may include any type of surgical instrument, and is shown here as a catheter guide that can be used in neurosurgery. In some implementations, the tracked instrument 340 can be coupled to one or more markers, allowing the position and orientation of the tracked instrument 340 within the surgical environment to be determined. For example, in some implementations, the processing circuit 212 of the image processing system 100 or the image processing system 1000 of FIG. 10 can track the position of the instrument relative to a 3D point cloud representing the patient. In some implementations, tracking markers can be coupled to the robotic arm 310, for example, to track the position of the robotic arm 310 or its various joints.

[0075] The systems and methods described herein may be utilized at the bedside in a surgical environment. For example, as described herein, both the platform 112 including the image processing system 100 and the robotic system 300, which may include any of the robotic control systems 405, 605, or 805, or the image processing system 1000, may be positioned in a surgical environment including a patient. The image processing system 100 may perform an image-to-patient registration process to align a 3D image from a computed tomography (CT) scan or a magnetic resonance imaging (MRI) scan with a 3D point cloud of the patient captured by the image capture device 104. In addition, the position of the patient's face may be determined based on the position of the 3D point cloud representing the patient. The processing circuitry 212 may also capture and track the position of the robotic arm 310 or the tracked instrument 340 in the same reference coordinate system as the 3D point cloud representing the patient, allowing the processing circuitry 212 to determine the distance of the tracked instrument 340 from the patient or a predetermined surgical path. Techniques for updating the position of the surgical robot 310 based on various attributes of the surgical environment (eg, detected patient movement, torque sensing, etc.) are described in further detail in the following sections.

[0076] (B. Control of surgical robots based on patient tracking technology) The systems and methods described herein provide various techniques for controlling a surgical robot in a surgical environment. In particular, the techniques described herein provide improved movement tracking and coordination for the surgical robot based on a real-time 3D image of the patient in the surgical environment. Using the image processing system 100 to perform the patient tracking techniques, the systems and methods described herein can determine precise patient movement in near real-time while navigating the surgical robot along a predefined path in the surgical environment. The surgical robot can be controlled to align a tracked instrument (e.g., tracked instrument 340) with a predefined path or target. For example, the target can be an intracranial target. The location of the intracranial target can be determined based on the location of a point of interest in a CT scan image or an MRI scan image that is aligned with the real-time 3D image using, for example, image-to-patient registration techniques. When patient movement is detected, the trajectory or position of the tracked instrument used in the surgical procedure can be adjusted to maintain this alignment. The systems and methods described herein improve surgical robot navigation techniques by using real-time patient tracking to enable real-time correction of the surgical path. The techniques described herein improve patient safety during surgical procedures.

[0077] 4, shown is an exemplary system 400 for controlling a surgical robot (such as robot 310) based on patient tracking technology, according to one or more implementations. System 400 may include at least one robot control system 405, at least one robot 420, and at least one image processing system 100. Robot control system 405 may include at least one point cloud accessor 435, at least one robot tracker 440, at least one image registration component 445, at least one movement detector 450, and at least one robot navigator 455. Robot 420 may include an instrument 430.

[0078] Each of the components of system 400 (e.g., robot control system 405, image processing system 100, robot 420, etc.) may be implemented using hardware components or a combination of hardware components and software of a computing system (e.g., computing system 1000, any other computing system described herein, etc.) detailed herein in conjunction with Figures 12A and 12B. Each of the components of robot control system 405 (e.g., point cloud accessor 435, robot tracker 440, image registration component 445, movement detector 450, robot navigator 455, etc.) may perform functions detailed herein. Although image processing system 100 and robot control system 405 are shown as separate systems, it should be understood that robot control system 405 may be a part of image processing system 100 (e.g., implemented at least in part by processing circuitry 212, etc.) or vice versa (e.g., processing circuitry 212 of image processing system 100 implemented on one or more processors of robot control system 405). Similarly, the robot control system 405 may be implemented with or include the image processing system 1000 described in connection with FIG. 10, or vice versa. In implementations in which the image processing system 100 and the robot control system 405 are implemented as separate computing systems, the image processing system 100 and the robot control system 405 may exchange information via a communication interface, as described herein. Similarly, the robot control system 405 and the robot 420 may communicate via one or more communication interfaces. The robot control system 405 may communicate any generated instructions to the robot 420 for execution.

[0079] The robotic control system 405 may be or form part of the image processing system 100 described herein in conjunction with FIGS. 1A, 1B, and 2, and may perform any of the functions of the image processing system 100 as described herein. The robotic control system 405 may include at least one processor and memory (e.g., processing circuitry). The memory may store processor-executable instructions that, when executed by the processor, cause the processor to perform one or more of the operations described herein. The processor may include a microprocessor, an ASIC, an FPGA, a graphics processing unit (GPU), or the like, or a combination thereof. The memory may include, but is not limited to, an electronic, optical, magnetic, or any other storage or transmission device capable of providing program instructions to the processor. The memory may further include a floppy disk, a memory chip, an ASIC, an FPGA, a read only memory (ROM), a random access memory (RAM), an electrically erasable programmable ROM (EEPROM), an erasable programmable ROM (EPROM), a flash memory, an optical media, or any other suitable memory from which a processor can read instructions. The instructions may include code from any suitable computer programming language. The robotic control system 405 may include one or more computing devices or servers capable of performing various functions described herein. The robotic control system 405 may include any or all of the components and perform any or all of the functions of the computer system 1000 described herein in connection with FIG. 12A and FIG. 12B.

[0080] The robot 420 can be or include any of the functions or structures of the robot arm 310 described herein above in connection with FIG. 3. In some implementations, the robot 420 can be a different type of surgical robot capable of maneuvering a surgical instrument within a surgical environment. The robot 420 can include a computing device that executes instructions to move the robot to a desired position or orientation within a surgical environment. The robot control system 405 (or components thereof) can generate instructions for the robot 420 that cause the robot 420 to change its position, orientation, or status (e.g., automatic or cooperative, etc.). The robot 420 can operate in an automatic mode, where the position of the robot 420 (and the instrument 430 coupled thereto) is controlled by software (e.g., the robot navigator 455 described in connection with FIG. 4, the robot navigator 650 described in connection with FIG. 6, the robot navigator 835 described in connection with FIG. 8, any other components of the robot control system 405, 605, or 805 as described herein, etc.). When the robot 420 is in the appropriate status, it can operate in a collaborative mode in which the robot 420 can be fully or partially controlled by the surgeon's manual input. For example, the surgeon may hold parts of the robot to position the robot 420 or the instrument 430 towards a desired target.

[0081] In some implementations, as described herein above in connection with FIG. 4, the robot 420 may include a display positioned over the patient in the surgical environment to allow the surgeon to view information about the patient (e.g., a close-up view of the surgical site with annotations, information about the target of the surgical procedure or other target locations, etc.) as the surgical procedure is performed. In some implementations, the robot may include a capture device similar to one of the image capture devices 104 described herein above. Images 208 captured by an image capture device coupled to the robot 420 may be displayed on a display coupled to the robot 420 and positioned over the surgical site.

[0082] The instrument 430 may be any type of instrument that may be used in a surgical environment for a surgical procedure on a patient. The instrument 430 may be coupled to the robot 420 such that the robot 420 can control the position and orientation of the instrument 430 in space. The instrument 430 may be, for example, a punch tool, a cannula needle, a biopsy needle, a catheter device, or any other type of surgical instrument. The instrument 430 may be a tracked instrument 340 and may include any of its structures and functions. For example, the instrument 430 (or a bracket coupling the instrument 430 to the robot 420) may be coupled to one or more tracking indicators. The tracking indicators may be, for example, IR light emitting diodes (LEDs), LEDs that emit colors in the visible spectrum, a tracking ball colored in a predetermined color or having a predetermined detectable shape, or other tracking features such as a QR code. The tracking indicators may be positioned at predetermined locations on the instrument 430 or robot 420 and may form a matrix or array of sensors that, when detected by a computing device (e.g., image processing system 100, robot control system 405, etc.), may be used to determine the position and orientation of the instrument 430 or robot 420. In some implementations, the instrument 430 may include or be coupled to one or more position sensors, such as an accelerometer, gyroscope, or inertial measurement unit (IMU), among others.

[0083] The point cloud accessor 435 can access a 3D (three-dimensional) point cloud corresponding to the surgical environment and the patient. The 3D point cloud can have a reference coordinate system corresponding to the surgical environment. As described above in this specification, the image processing system 100 can utilize one or more image capture devices 104, which can be 3D cameras, to capture real-time (or near real-time) 3D images of the patient during a surgical procedure. The 3D point cloud can correspond to, for example, the patient's head, the patient's body, or any other part of the patient on which surgery can be performed. In some implementations, the image capture device 104 can be positioned in the surgical environment to capture images of the patient's face. In some implementations, the point cloud accessor 435 can apply an image segmentation model to the 3D point cloud captured by the image capture device 104. In some implementations, the point cloud accessor 435 can receive the point cloud from the processing circuit 212 of the image processing system 100 of FIGS. 1 and 2 or the processing circuit 1014 of the image processing system 1000 of FIG. 10 via, for example, one or more communication interfaces. In some implementations, the point cloud accessor 435 can capture an indication of the global environment (e.g., stationary points in the surgical environment from which a 3D point cloud corresponding to the patient can be accessed). The point cloud accessor 435 can repeatedly receive 3D images including a point cloud representing the patient, for example at a predetermined frame rate of the image capture device 104. The point cloud accessor 435 can access or otherwise retrieve a 3D point cloud representing the user and store the 3D point cloud in memory of the robotic control system 405 such that the 3D point cloud can be accessed by other components of the robotic control system. The 3D point clouds can be stored in association with respective timestamps such that the position and orientation of the patient can be determined in real time or near real time.

[0084] In addition to tracking the position of the patient, the robot tracker 440 can determine the position of the surgical robot in the same reference coordinate system as the 3D point cloud. As described herein above, the image capture device 104 can capture images of the patient in the surgical environment. The image capture device 104 can have a predefined pose in the surgical environment relative to other sensors, such as the IR sensor 220. As described herein above, the IR sensor 220 can be an IR camera that captures IR wavelengths of light. The robot tracker 440 can determine the position of the robot by utilizing one or more tracking techniques, such as IR tracking techniques, to determine the position of the robot 420. The robot 420 can include one or more markers or indicators, such as IR indicators, on its surface. The robot tracker 440 can utilize the IR sensor 220 to determine the relative position of the robot. In some implementations, the robot tracker 440 can determine the orientation (e.g., pose) of the robot 420 by performing a similar technique. Because the sensors used to track the position of the robot 420 are at a known distance from the image capture device 104, the positions of the markers detected by the IR sensor 220 can be mapped to the same reference coordinate system as the 3D point cloud captured by the image capture device 104.

[0085] In some implementations, the image capture device 104 may be used to determine the position of the robot 420. For example, the robot 420 may include one or more graphical indicators, such as bright distinct colors, patterns, or QR codes, among others. In addition to capturing the 3D point cloud of the patient, the image capture device 104 may capture images of the surgical environment, and the robot tracker 440 may perform image analysis techniques to determine the position of the surgical robot in the image 208 based on the detected positions of the indicators in the image 208. The position and orientation of the surgical robot may be calculated periodically, for example, in real-time or near real-time, allowing the robot tracker 440 to track the movement of the robot 420 over time. In some implementations, the robot tracker 440 may perform a calibration procedure, for example, to establish a unified reference frame between the 3D point cloud and the indicators coupled to the surgical robot. The calibration procedure may include identifying the position of the robot 420 relative to a global indicator positioned in the surgical environment. The robot tracker 440 can store the position of the robot 420 in a memory of the robot tracker 440 such that the real-time or near real-time position of the robot 420 can be accessed by other components of the robot control system 405. The calibration procedure can include using predefined markers or patterns (e.g., a checkerboard pattern, etc.) within the surgical environment to calibrate the posture (e.g., position and orientation) of the robot 420 relative to the patient.

[0086] Tracking the position of the robot 420 may include tracking the position of the instrument 430. As described herein above, the instrument 430 may include its own indicator coupled to the instrument 430 or a bracket / connector coupling the instrument 430 to the robot 420. An example of an indicator coupled to the instrument 430 is shown as tracked instrument 340 in the close-up 312 shown in FIG. 3. The robot tracker 440 may track the position and orientation of the instrument 430 and the robot in real-time or near real-time using techniques similar to those described above. The position and orientation of the instrument 430 determined by the robot tracker 440 may be stored in the memory of the robot control system 405. In some implementations, the robot tracker 440 may store the position and orientation of the instrument 430 in association with an individual timestamp.

[0087] The image registration component 445 can perform image-to-patient registration techniques to align a 3D image of the patient, which may include indicators of potential targets, with a 3D point cloud captured from the patient. For example, in the case of intracranial surgery, the target may be a biopsy site within the patient's skull. A 3D image, such as a CT scan of the patient's head, may include both a 3D representation of the patient's face and an indication of the biopsy site within the patient's brain. By registering the 3D image of the patient with the real-time 3D point cloud of the patient within the surgical environment, the image registration component 445 can map the 3D image of the patient and any target indicators to the same reference coordinate system as both the 3D point cloud, the tracked position of the robot 420, and the tracked position of the instrument 430. Doing so allows the robot tracker 440 to track the position of the instrument 430 relative to both the patient and the target area shown in the 3D image of the patient. To register the 3D image to the 3D point cloud captured by the image capture device 104, the image registration component 445 can perform an iterative fitting process, such as a random sample consensus (RANSAC) algorithm and an iterative closest point (ICP) algorithm. The image registration component 445 can continuously register the 3D image to the patient's 3D point cloud. If the registration fails (e.g., if a fitting algorithm fails to fit the 3D image to the 3D point cloud within a predetermined error threshold), the image registration component 445 can generate a signal indicating the failure.

[0088] As described herein above, the point cloud accessor 435 can continuously track the position of the patient over time in the surgical environment (e.g., each time an image is captured by the image capture device 104). The movement detector 450 can detect changes in the patient's position by comparing the positions of the 3D point clouds over time. For example, when the point cloud accessor 435 receives or accesses a new 3D point cloud based on a new image captured by the image capture device 104, the point cloud accessor 435 can store the position of each point in the 3D point cloud in a memory of the robotic control system 405, for example, in one or more data structures. The data structures can be time-stamped or an index or other indication of the order can be otherwise encoded in the data structures such that the order in which the 3D point clouds were captured can be determined by components of the robotic control system 405. The point cloud accessor 435 may store the new 3D point cloud representing the patient in a rolling queue, such that a predetermined number of the most recent frames captured by the image capture device 104 are stored in the memory of the robotic control system 405.

[0089] The motion detector 450 can access the 3D point cloud stored in the memory of the robot control system 405 to determine the amount of patient movement within the surgical environment. One improvement of the systems and methods described herein when used during a surgical procedure is that the technique provides accurate patient tracking when the patient is unconstrained. Certain surgical procedures, including certain intracranial procedures, can be performed without fixing the patient to a surgical table or device. The motion detector 450 can accurately detect patient movement in such scenarios and generate signals for components of the robot control system 405 to adjust the position of the robot 420 to accommodate the patient movement. The motion detector 450 can compare a previous position of the 3D point cloud representing the patient to the position of the current or new 3D point cloud captured by the image capture device 104. In some implementations, the motion detector 450 can maintain two sets of 3D point clouds: one set from a previously captured frame and another set from a currently captured (e.g., most recently captured) frame. The comparison may be of distance (eg, Euclidean distance, etc.) in a reference coordinate system of the 3D point cloud.

[0090] The motion detector 450 may perform an iterative calculation to determine which 3D points in a first 3D point cloud (e.g., a previous frame) correspond to 3D points in a second 3D point cloud (e.g., a current frame). For example, the motion detector 450 may perform an iterative ICP algorithm or a RANSAC fitting technique to approximate which points in the current frame correspond to points in the previous frame. The distance between the corresponding points may then be determined. In some implementations, to improve computational performance, the motion detector 450 may compare subsets of points (e.g., by performing a downsampling technique on the point clouds, etc.). For example, after determining point correspondence, the motion detector 450 may select a subset of matching point pairs between each point cloud for comparison (e.g., determining the distance between the points in space). In some implementations, to determine the patient's movement in the surgical environment, the motion detector 450 may calculate the average movement of the 3D points between frames. When a new frame is captured by the image capture device 104, the motion detector 450 can overwrite (e.g., in memory) the 3D point cloud representing the previous frame with the current frame, overwrite the current frame with the 3D point cloud of the new frame, and calculate the patient's movement between the current frame and the previous frame. The patient's movement within the surgical environment over time can be stored in one or more data structures in the memory of the robotic control system 405 such that the movement values ​​(e.g., change in position over time, absolute patient position, etc.) are accessible to components of the robotic control system 405. In some implementations, the motion detector 450 can detect the patient's movement in response to detecting that the movement exceeds a predetermined threshold (e.g., more than a few millimeters, etc.) from a previous frame or from a predetermined starting position (e.g., the patient's position at the start of the procedure, etc.). In some implementations, the predetermined threshold can be defined as part of a target path or surgical procedure associated with a target position within the surgical environment. For example, the surgeon can provide user input to define one or more thresholds as small if the robot is navigating an area requiring high precision movement.Thus, the robotic navigator 455 can maintain a data structure with multiple predefined thresholds for movement, each predefined threshold being assigned to a particular portion of multiple portions of image data corresponding to a patient, allowing the robotic navigator 455 to control navigation in a manner that is more dynamic and responsive to underlying anatomical and surgical considerations.

[0091] The robot navigator 455 can generate instructions to modify the position of the surgical robot based on the change in the position of one or more points in response to detecting a change in the patient's position. As described herein above, the robot 420 can change its position by executing or interpreting instructions or signals indicative of a target position for the instrument 430. For example, the robot 420 can execute such instructions indicative of a target position for the instrument 430 and actuate various movable components within the robot 420 to move the instrument 430 to the target position. The robot navigator 455 can navigate the robot 420 according to the patient's movements such that the robot 420 or instrument 430 is aligned with a target position or target path within the surgical environment even when the patient is not immobilized. The robot navigator 455 can use the computer instructions to generate commands that cause the robot 420 to move, for example, until a predetermined process or procedure is completed (e.g., the robot 420 or instrument 430 is navigated to one or more predetermined targets within the surgical environment) or an exit condition is triggered (e.g., a collaborative mode condition). In some implementations, the target location or target path may be defined as part of a 3D image (e.g., a CT scan or an MRI scan, etc.) that is registered to a real-time 3D point cloud representing the patient in the reference coordinate system of the image capture device 104. Because the positions of the instrument 430 and the robot 420 are also mapped into the same reference coordinate system, an accurate measurement of the distance between the instrument 430 and the robot 420 can be determined. For example, an offset may be added to this distance based on known attributes of the instrument to approximate the location of a given portion of the instrument 430 (e.g., the tip or tool end) within the surgical environment.

[0092] When patient movement is detected by the movement detector 450, the robot navigator 455 can generate corresponding instructions to move the robot 420 according to the patient movement. For example, from a CT scan or from user input, the instrument 430 may be aligned with a target path or target location in the surgical environment to perform a surgical procedure. If the patient moves during the surgical procedure, the robot navigator 455 can generate instructions to move the instrument 420 in synchronization with the detected patient movement, for example, using one or more application programming interfaces (APIs) of the robot 430. For example, if a patient movement of 2 cm to the left of the target path is detected, the robot navigator 455 can adjust the position of the robot 430 the same 2 cm to the left according to the target path. In some implementations, the robot navigator 455 can make adjustments while navigating the robot 420 along a predetermined trajectory (e.g., a path into the patient's skull to reach a target location in the brain). For example, the robot 420 can navigate left or right following the patient's movements while also navigating the instrument 430 down into the patient's skull along a predetermined trajectory. This provides an improvement over other robotic implementations that do not track and compensate for patient movement since any patient movement during a procedure can cause harm to the patient. By moving the instrument following the patient's movements, unintended collisions or interference with other parts of the patient are mitigated as a predetermined target path can be more accurately followed.

[0093] The robot navigator 455 may perform these adjustments iteratively or periodically multiple times per second to compensate for sudden patient movement. As described herein above, the movement detector 450 may calculate the patient movement periodically (e.g., according to the capture rate of the image capture device 104, etc.). Each time a new frame is captured by the image capture device 104, the movement detector 450 may determine whether the change in the patient position meets a threshold (e.g., a predetermined amount of movement relative to the previous frame, the patient position at the start of the procedure, or the position of the instrument 430, etc.). The robot navigator 455 may then adjust the position of the robot 420 such that the instrument 430 is aligned with a predetermined path (e.g., a predetermined trajectory) relative to the detected change in the patient position. The change in the position of the trajectory or path may be determined based on a change in the position or orientation of one or more target indicators in a 3D image (e.g., a CT scan or an MRI scan, etc.) that is registered to the patient in real time.

[0094] In some implementations, the 3D image may be modified to show a path or trajectory to a selected target. Similarly, in implementations where there are multiple targets or paths (or path segments), the surgeon may select, via user input (e.g., button selection, touch screen selection, etc.), one or more paths along which the robot 420 should navigate the instrument 430. The robot navigator 455 can navigate the robot 420 along the selected path while compensating for patient movement in real time, as described herein. The robot navigator 455 can navigate the robot 420 in multiple scenarios. For example, the robot navigator 455 can adjust the position of the robot 420 while the robot is still in space (e.g., providing a rigid port to the patient). In some implementations, the robot navigator 455 can adjust the position of the robot along one or more axes while the operator of the robot 420 moves the robot 420 or instrument 430 downward through a predetermined trajectory to the target location.

[0095] 5, an exemplary method 500 of controlling a surgical robot based on patient tracking techniques is shown, according to one or more implementations. Method 500 may be performed, for example, by robot control system 405, 605, or 805, or any other computing device described herein, including computing system 1000 described herein in connection with FIGS. 12A and 12B. In overview of method 500, in step 502, a robot control system (e.g., robot control system 405, etc.) may access a 3D point cloud corresponding to a patient. In step 504, the robot control system may determine a position of a surgical robot (e.g., robot 420). In step 506, the robot control system may monitor the position of the patient over time. In step 508, the robot control system may determine whether patient movement is detected. In step 510, the robot control system may generate instructions to move the robot according to the detected movement.

[0096] In further detail of method 500, in step 502, a robotic control system (e.g., robotic control system 405, etc.) can access a 3D point cloud corresponding to a patient. As described herein above, an image processing system (e.g., image processing system 100) can utilize one or more image capture devices (e.g., image capture device 104), which can be 3D cameras, to capture real-time (or near real-time) 3D images of a patient during a surgical procedure. The 3D point cloud can correspond to, for example, the patient's head, the patient's body, or any other part of the patient on which surgery can be performed. In some implementations, the image capture device can be positioned within the surgical environment to capture images of the patient's face. In some implementations, the robotic control system can apply an image segmentation model to the 3D point cloud captured by the image capture device. In some implementations, the robotic control system can receive the point cloud from processing circuitry of the image processing system, for example, via one or more communication interfaces. In some implementations, the robotic control system can capture an indication of a global environment (e.g., static points within the surgical environment at which a 3D point cloud corresponding to a patient can be accessed, etc.). The robotic control system may receive 3D images including a point cloud representing the patient repetitively, for example, at a predetermined frame rate of the image capture device. The robotic control system may access or otherwise retrieve the 3D point cloud representing the user and store the 3D point cloud in memory of the robotic control system so that it may be accessed by other components of the robotic control system. The 3D point clouds may be stored in association with respective timestamps so that the position and orientation of the patient may be determined in real time or near real time.

[0097] In step 504, the robotic control system can determine the position of the surgical robot (e.g., robot 420). In addition to tracking the position of the patient, the robotic control system can determine the position of the surgical robot (e.g., robot 420) in the same reference coordinate system as the 3D point cloud. As described herein above, the image capture device can capture an image of the patient in the surgical environment. The image capture device can have a predefined pose in the surgical environment relative to other sensors, such as an IR sensor (e.g., IR sensor 220). As described herein above, the IR sensor can be an IR camera that captures IR wavelengths of light. The robotic control system can determine the position of the robot by utilizing one or more tracking techniques, such as IR tracking techniques, to determine the position of the robot. The robot can include one or more markers or indicators, such as IR indicators, on its surface. The robotic control system can utilize the IR sensor to determine the relative position of the robot. In some implementations, the robotic control system can determine the orientation (e.g., pose) of the robot by performing a similar technique. The sensors used to track the robot's position are at a known distance from the image capture device so that the positions of the markers detected by the IR sensors can be mapped into the same reference coordinate system as the 3D point cloud captured by the image capture device.

[0098] In some implementations, the image capture device may be used to determine the position of the robot. For example, the robot may include one or more graphical indicators, such as bright distinct colors or QR codes, among others. In addition to capturing the 3D point cloud of the patient, the image capture device may capture images of the surgical environment, and the robotic control system may perform image analysis techniques to determine the position of the surgical robot in the images based on the detected positions of the indicators in the images. The position and orientation of the surgical robot may be calculated periodically, e.g., in real-time or near real-time, allowing the robotic control system to track the movement of the robot over time. In some implementations, the robotic control system may perform a calibration procedure, e.g., to establish a unified reference frame between the 3D point cloud and the indicators coupled to the surgical robot. The calibration procedure may include identifying the position of the robot relative to a global indicator positioned in the surgical environment. The robotic control system may store the position of the robot in a memory of the robotic control system, such that the real-time or near real-time position of the robot may be accessed by other components of the robotic control system.

[0099] Tracking the position of the robot may include tracking the position of an instrument (e.g., instrument 430) coupled to the robot. As described herein above, the instrument may include its own indicator coupled to the instrument or a bracket / connector coupling the instrument to the robot. One example of an indicator coupled to an instrument is shown as tracked instrument 340 in the close-up 312 shown in FIG. 3. The robotic control system may track the position and orientation of the instrument and robot in real-time or near real-time using techniques similar to those described above. The instrument positions and orientations determined by the robotic control system may be stored in a memory of the robotic control system. In some implementations, the robotic control system may store the instrument positions and orientations in association with their respective timestamps.

[0100] In step 506, the robot control system can monitor the position of the patient over time. The robot control system can accurately detect the patient's movement and generate signals to adjust the position of the robot 420 to accommodate the patient's movement. The robot control system can compare previous positions of the 3D point cloud representing the patient with the position of the current or new 3D point cloud captured by the image capture device 104. In some implementations, the robot control system can maintain two sets of 3D point clouds, one from a previously captured frame and another from a currently captured (e.g., most recently captured) frame. The comparison can be a distance (e.g., Euclidean distance, etc.) in a reference coordinate system of the 3D point clouds. The robot control system can perform an iterative calculation to determine which 3D points in a first 3D point cloud (e.g., a previous frame) correspond to 3D points in a second 3D point cloud (e.g., a current frame). For example, the robot control system can perform an iterative ICP algorithm or a RANSAC fitting technique to approximate which points in the current frame correspond to points in the previous frame. The distance between corresponding points can then be determined. In some implementations, to improve computational performance, the robotic control system can compare subsets of points (e.g., by performing downsampling techniques on the point clouds). For example, after determining point correspondences, the robotic control system can select a subset of matching point pairs between each point cloud for comparison (e.g., determining the distance between the points in space). In some implementations, to determine the patient's movement within the surgical environment, the robotic control system can calculate the average movement of the 3D points between frames.

[0101] In step 508, the robotic control system can determine whether patient movement is detected. When a new frame is captured by the image capture device 104, the robotic control system can overwrite (e.g., in memory) the 3D point cloud representing the previous frame with the current frame, overwrite the current frame with the 3D point cloud of the new frame, and calculate the patient movement between the current frame and the previous frame. The patient movement in the surgical environment over time can be stored in one or more data structures in the memory of the robotic control system such that the movement values ​​(e.g., change in position over time, absolute patient position, etc.) are accessible to components of the robotic control system. In some implementations, the robotic control system can detect patient movement if the movement exceeds a predetermined threshold (e.g., more than a few millimeters, etc.) from the previous frame or from a predetermined starting position (e.g., the patient's position at the start of the procedure, etc.). If the threshold is exceeded, the robotic control system can generate an instruction to move the robot in step 510. If the threshold is not exceeded, the robotic control system can continue to monitor the patient movement in step 506.

[0102] In step 510, the robot control system can generate instructions to move the robot according to the detected movements. As described herein above, the robot can change its position by executing or interpreting instructions or signals that indicate a target position for the instrument. For example, the robot can execute such instructions that indicate a target position for the instrument and actuate various moving components in the robot to move the instrument to the target position. The robot control system can navigate the robot according to the patient's movements such that the robot or instrument is aligned with a target position or target path in the surgical environment even when the patient is not immobilized. In some implementations, the target position or target path can be defined as part of a 3D image (e.g., a CT scan or an MRI scan, etc.) that is registered to a real-time 3D point cloud that represents the patient in the reference coordinate system of the image capture device. The positions of the instrument and the robot are also mapped in the same reference coordinate system, so that an accurate measurement of the distance between the instrument and the robot can be determined. For example, an offset may be added to this distance based on known attributes of the instrument to approximate the position of a given part of the instrument (e.g., tip or tool end) in the surgical environment.

[0103] When patient movement is detected by the robotic control system, the robotic control system can generate corresponding instructions to move the robot according to the patient movement. For example, from a CT scan or from user input, the instrument may be aligned with a target path or target location in the surgical environment to perform the surgical procedure. If the patient moves during the surgical procedure, the robotic control system can generate instructions to move the instrument in sync with the detected patient movement, for example, using one or more application programming interfaces (APIs) of the robot. For example, if a patient movement of 2 cm to the left of the target path is detected, the robotic control system can adjust the position of the robot the same 2 cm to the left, following the target path. In some implementations, the robotic control system can make adjustments while navigating the robot along a predefined trajectory (such as a path into the patient's skull to reach a target location in the brain). For example, the robot can navigate left or right according to the patient movement, while also navigating the instrument down into the patient's skull along a predefined trajectory. This provides an improvement over other robotic implementations that do not track and compensate for patient movement, as any patient movement during the procedure may result in harm to the patient. By moving the instrument following the patient's movements, a predetermined target path may be followed more accurately, reducing unintended collisions or interference with other parts of the patient.

[0104] The robotic control system can perform these adjustments iteratively or periodically, multiple times per second, to compensate for sudden patient movement. As described above herein, the robotic control system can calculate the patient movement periodically (e.g., according to the capture speed of the image capture device, etc.). Each time a new frame is captured by the image capture device, the robotic control system can determine whether the change in the patient's position meets a threshold (e.g., a predetermined amount of movement relative to the previous frame, the patient position at the start of the procedure, or the position of the instrument, etc.). The robotic control system can then adjust the position of the robot such that the instrument is aligned with a predetermined path (e.g., a predetermined trajectory) relative to the detected change in the patient's position. The change in the trajectory or path position can be determined based on a change in the position or orientation of one or more target indicators in a 3D image (e.g., a CT scan or an MRI scan, etc.) that is aligned to the patient in real time. In some implementations, the 3D image can be modified to show a path or trajectory to the selected target. Similarly, in implementations where there are multiple targets or paths (or path segments), the surgeon may select, via user input (e.g., button selection, touch screen selection, etc.), one or more paths along which the robot should navigate the instrument. The robotic control system can navigate the robot along the selected paths while compensating for patient movement in real time, as described herein.

[0105] (C. Control of surgical robots based on torque sensing technology) The systems and methods described herein provide various techniques for controlling a surgical robot in a surgical environment. This section describes such techniques operating in conjunction with a torque sensor, which may be positioned, for example, on the patient or on the surgical robot operating on the patient. The techniques described herein provide improved movement tracking and adjustments for the surgical robot based on real-time torque measurements of forces applied by the patient. The torque measurements detected by the systems and methods described herein can be used to modify the position or orientation of a surgical tool during a surgical procedure, which provides improved patient safety during a surgical procedure. The torque sensors described herein may be positioned at various locations on the patient's body or around the patient's surgical site. In some implementations, other types of sensors, such as accelerometers, magnetometers, gyroscopes, or all-in-one sensors such as IMUs, may be utilized in conjunction with or instead of the torque sensors.

[0106] 6, an exemplary system 600 for controlling a surgical robot (e.g., robot 310, robot 420, etc.) based on torque sensing technology, according to one or more implementations, is shown. System 600 may include at least one robot control system 605, at least one robot 620, at least one image processing system 100, and one or more sensors 655. Robot control system 605 may include at least one measurement identifier 635, at least one robot tracker 640, at least one movement detector 645, and at least one robot navigator 650. Robot 620 may include an instrument 630.

[0107] Each of the components of system 600 (e.g., robot control system 605, image processing system 100, robot 620, etc.) may be implemented using hardware components or a combination of hardware components and software of a computing system (e.g., computing system 1000, any other computing system described herein, etc.) detailed herein in conjunction with Figures 12A and 12B. Each of the components of robot control system 605 (e.g., measurement identifier 635, robot tracker 640, motion detector 645, robot navigator 650, etc.) may perform functions detailed herein. Although image processing system 100 and robot control system 605 are shown as separate systems, it should be understood that robot control system 605 may be a part of image processing system 100 (e.g., implemented at least in part by processing circuitry 212, etc.) or vice versa (e.g., processing circuitry 212 of image processing system 100 implemented on one or more processors of robot control system 605). Similarly, the robot control system 605 may be implemented with or include the image processing system 1000 described in connection with FIG. 10, or vice versa. In implementations in which the image processing system 100 and the robot control system 605 are implemented as separate computing systems, the image processing system 100 and the robot control system 605 may exchange information via a communication interface, as described herein. Similarly, the robot control system 605 and the robot 620 may communicate via one or more communication interfaces. The robot control system 605 may communicate any generated instructions to the robot 620 for execution.

[0108] The robot 620 and the tool 630 may be similar to the robot 420 and the tool 430, respectively, and may include any of their structures and functions. In addition, the robot control system 605 may include any of the structures or functions of the robot control system 605. The robot control system 605 may be or form part of the image processing system 100 described herein in conjunction with FIGS. 1A, 1B, and 2, and may perform any of the functions of the image processing system 100 as described herein. The robot control system 605 may include at least one processor and memory (e.g., processing circuitry). The memory may store processor-executable instructions that, when executed by the processor, cause the processor to perform one or more of the operations described herein. The processor may include a microprocessor, an ASIC, an FPGA, a GPU, or the like, or a combination thereof. The memory may include, but is not limited to, an electronic, optical, magnetic, or any other storage or transmission device capable of providing program instructions to the processor. The memory may further include memory chips, ASICs, FPGAs, ROMs, RAMs, EEPROMs, EPROMs, flash memory, optical media, or any other suitable memory from which a processor can read instructions. The instructions may include code from any suitable computer programming language. The robotic control system 605 may include one or more computing devices or servers capable of performing various functions described herein. The robotic control system 605 may include any or all of the components and perform any or all of the functions of the computer system 1000 described herein in connection with FIGS. 12A and 12B.

[0109] The sensor 655 can be any type of sensor capable of detecting movement from a patient in a surgical environment, for example. The sensor 655 can be a force transducer that converts a detected mechanical force, such as torque, into an analog or digital signal. The signal can be communicated to the robotic control system 605 via one or more communication interfaces. In general, the sensor 655 can monitor the conditions of the surgical environment continuously (e.g., in real time or near real time, etc.) or at predetermined periodic intervals. In some implementations, the sensor 655 may provide a signal to the robotic control system 605 in response to detecting a torque or movement exceeding a predetermined threshold. In addition to measuring torque, the sensor 655 can include any type of movement sensor, such as a torque sensor, an accelerometer, a gyroscope, a magnetometer, or an all-in-one sensor such as an IMU. Measurements from the sensor can be communicated to the robotic control system 605 in response to a request, at a predetermined time period, or in response to detecting a signal exceeding a predetermined threshold. In some implementations, one or more of the sensors 655 can be positioned on one or more portions of the robot 620 or the instrument 630.

[0110] Referring now to the operation of the robotic control system 605, the measurement identifier 635 can identify measurements captured by the sensors 655 and store the sensor measurements in a memory of the robotic control system 605. In some implementations, the measurement identifier 635 can periodically transmit requests to the sensors 655, for example at a predetermined time period. In response to the requests, the sensors 655 can transmit each sensor 655 measurement to the measurement identifier 635. The measurement identifier 635 can store the sensor measurements, for example, chronologically or otherwise associated with a timestamp corresponding to when the sensor measurement was captured. In some implementations, the measurement identifier 635 can store each sensor measurement in association with the identifier of the sensor 655 from which the measurement was captured. In some implementations, the measurement identifier 635 can store an association between each measurement and the object (e.g., the patient, the robot 620, the instrument 630, etc.) to which the respective sensor 655 is coupled. This allows components of the robotic control system 605 to measure or detect forces experienced or generated by each object in the surgical environment.

[0111] The robot tracker 640 can determine the position of the robot 620 within the surgical environment. As described herein above, the image capture device 104 can capture an image of the patient within the surgical environment. The image capture device 104 can have a predefined orientation within the surgical environment relative to other sensors, such as the IR sensor 220. As described herein above, the IR sensor 220 can be an IR camera that captures IR wavelengths of light. The robot tracker 640 can determine the position of the robot by utilizing one or more tracking techniques, such as IR tracking techniques or computer vision tracking techniques, to determine the position of the robot 620. The robot 620 can include one or more markers or indicators, such as IR indicators, on its surface. The robot tracker 640 can utilize the IR sensor 220 to determine the relative position of the robot 620 within the surgical environment. In some implementations, the robot tracker 640 can determine the orientation (e.g., pose) of the robot 620 by performing a similar technique. Because the IR sensor 220 (or other sensor) used to track the position of the robot 620 is at a known distance from the image capture device 104, the positions of the markers detected by the IR sensor 220 can be mapped to the same reference coordinate system as the 3D point cloud captured by the image capture device 104.

[0112] In some implementations, the image capture device 104 may be used to determine the position of the robot 620. For example, the robot 620 may include one or more graphical indicators, such as bright distinct colors, patterns, or QR codes, among others. The image capture device 104 may capture images of the surgical environment, and the robot tracker 640 may perform image analysis techniques to determine the position of the surgical robot in the image 208 based on the detected positions of the indicators in the image 208. The position and orientation of the surgical robot may be calculated periodically, for example, in real-time or near real-time, allowing the robot tracker 620 to track the movement of the robot 620 over time. In some implementations, the robot tracker 620 may perform a calibration procedure, for example, to establish a unified reference frame between the 3D point cloud and the indicators coupled to the surgical robot. The calibration procedure may include identifying the position of the robot 620 relative to a global indicator positioned in the surgical environment. The robot tracker 640 can store the position of the robot 620 in a memory of the robot tracker 640 such that the real-time or near real-time position of the robot 620 can be accessed by other components of the robot control system 605. The calibration procedure can include using predefined markers or patterns (e.g., a checkerboard pattern, etc.) within the surgical environment to calibrate the posture (e.g., position and orientation) of the robot 620 relative to the patient.

[0113] Tracking the position of the robot 620 may include tracking the position of the instrument 630. As described herein above, the instrument 630 may include its own indicator coupled to the instrument 630 or a bracket / connector coupling the instrument 630 to the robot 620. An example of an indicator coupled to the instrument 630 is shown as tracked instrument 340 in the close-up 312 shown in FIG. 3. The robot tracker 640 may track the position and orientation of the instrument 630 and the robot 620 in real-time or near real-time using techniques similar to those described above. The position and orientation of the instrument 630 determined by the robot tracker 640 may be stored in the memory of the robot control system 605. In some implementations, the robot tracker 640 may store the position and orientation of the instrument 630 in association with an individual timestamp. In some implementations, tracking the position of the robot 620 or instrument 630 may also be performed based on measurements captured from one or more sensors 655 coupled to the robot 620 or instrument 630. For example, the robot tracker 640 may determine the position of the robot 620 using acceleration or velocity values ​​(captured from the sensor 655), e.g., to interpolate the position of the robot 620 or the instrument 630 over time.

[0114] The movement detector 645 can detect a position modification condition based on a set of measurements captured by one or more sensors 655. As described herein above, in some implementations, one or more of the sensors 655 can be positioned on one or more portions of the patient. Measurements captured from these sensors 655 can indicate to the movement detector 645 that the patient's position has changed over time (e.g., has undergone acceleration, exerted a force or torque since the beginning of the surgical procedure). The movement detector 645 can detect the patient's movement by integrating (e.g., in the case of acceleration or velocity measurements) or interpolating (e.g., in the case of position measurements) the patient's position over time. The movement detector 645 can store the detected positions of the patient in association with respective timestamps corresponding to when the corresponding sensor measurements were captured. In this manner, the movement detector 645 can determine and record the patient's position over time within the surgical environment.

[0115] The movement detector 645 can detect whether the patient's position meets a position correction condition. The position correction condition can be a predetermined amount of movement or deviation from an initial condition of the surgical procedure. Using the recorded position values ​​of the patient determined from the sensor 655 measurements, the movement detector 645 can determine the amount of movement over time by comparing the patient's current position (e.g., from the most recent sensor measurement) to the patient's initial condition. If the position correction condition is met, the robot navigator 650 can adjust the position of the robot 620 or the instrument 630 to minimize the occurrence of injury to the patient.

[0116] In some implementations, the movement detector 645 can determine that the position correction condition is met when the determined movement of the patient over time exceeds or equals a predetermined threshold movement amount. In some implementations, the patient movement can be measured from the patient's position following a previous position adjustment of the robot 620 such that the adjusted position becomes a new baseline for subsequent adjustments of the position of the robot 620 or the instrument 630. As described herein above, the robot 620 can be navigated along a predetermined trajectory such that the instrument 630 can interact with a target location and perform a portion of a surgical procedure. In some implementations, the movement detector 645 can detect that the position correction condition is met by determining that the position of the surgical robot has deviated from the predetermined trajectory. For example, measurements of one or more sensors 655 positioned on the robot 620 can indicate that the robot 620 has experienced an external force (e.g., from a collision with another object) that has caused or will cause the robot 620 to deviate from the predetermined trajectory. In response to the detected force, the movement detector 645 can generate a signal that causes the robot control system 605 to navigate the robot 620 back to the predetermined trajectory.

[0117] In some implementations, one or more of the measurements from the sensor 655 can indicate that a collision has occurred between the robot 620 or the instrument 630 and another object in the surgical environment. For example, one or more force sensors can indicate that a collision has occurred against a surface on the patient. In some implementations, the movement detector 645 can detect that a position correction condition has been met by determining that a collision with the robot 620 (or the instrument 630) has occurred based on a set of measurements. If a collision occurs, the movement detector 645 can generate a signal that can include information about the nature of the collision (e.g., the direction of the force received, the direction in which the robot 620 should be moved to avoid further or harmful collisions, etc.). This information can be used by the robot control system 605 to navigate the robot 620 to avoid further collisions.

[0118] The robot navigator 650 can generate instructions to modify the position of the surgical robot based on the set of measurements. As described herein, when a position modification condition is met, the robot navigator 650 can generate instructions for the robot 620 corresponding to the sensor measurements that triggered the position modification condition. As described herein above, the robot 620 can change its position by executing or interpreting instructions or signals that indicate a target position for the instrument 630. For example, the robot 620 can execute such instructions that indicate a target position for the instrument 630 and actuate various movable components within the robot 620 to move the instrument 630 to the target position. The robot navigator 650 can navigate the robot 620 according to the movement of the patient such that the robot 620 or the instrument 630 is aligned with a target position or target path in the surgical environment even when the patient is not immobilized. In some implementations, the target position or target path can be defined as part of a 3D image (e.g., a CT scan or an MRI scan, etc.), as described herein above in connection with FIG. 4.

[0119] When patient movement is detected by the movement detector 645, the robot navigator 650 can generate corresponding instructions to move the robot 620 according to the patient movement. For example, sensor measurements monitored by the robot tracker 640 can indicate that the patient has moved a predetermined amount in a predetermined direction. The robot navigator 650 can generate instructions for the robot to move the instrument 630 according to the patient movement so that the instrument 630 stays on a predetermined path or trajectory that leads to a selected target location on or within the patient. In some implementations, the robot navigator 650 can generate instructions to compensate from external forces experienced by the robot 620. For example, the instructions can move the robot 620 against the detected forces to stay in the proper position relative to the patient.

[0120] The robot navigator 650 can perform these adjustments iteratively or periodically multiple times per second to compensate for sudden patient movement or detected forces. As described herein above, the movement detector 645 can calculate the patient movement periodically (e.g., according to the rate at which the sensor 655 captures measurements, etc.). Each time a new measurement from the sensor 655 is captured, the movement detector 645 can determine whether the change in the position of the patient (or robot 620) satisfies a position correction condition. The robot navigator 650 can then adjust the position of the robot 620 such that the instrument 630 is aligned with a predefined path (e.g., a predefined trajectory) based on the condition (e.g., the measurement from the sensor 655) that triggered the position correction condition. In addition, the robot navigator 650 can navigate the robot 620 along a target path while compensating for patient movement in real time, as described herein. The robot navigator 650 can navigate the robot 620 in multiple scenarios. For example, the robot navigator 650 can adjust the position of the robot 620 while it is still in space (e.g., providing a rigid port to the patient). In some implementations, the robot navigator 650 can adjust the position of the robot along one or more axes while an operator of the robot 620 is moving the robot 620 or instrument 630 down a predetermined trajectory to a target location.

[0121] 7, an exemplary method 700 for controlling a surgical robot based on patient tracking techniques is shown, according to one or more implementations. Method 700 may be performed, for example, by robot control system 405, 605, or 805, or any other computing device described herein, including computing system 1000 described herein in connection with FIGS. 12A and 12B. In overview of method 700, in step 702, a robot control system (e.g., robot control system 605, etc.) may identify measurements from a sensor (e.g., sensor 655, etc.). In step 704, the robot control system may determine a position of a surgical robot (e.g., robot 620). In step 706, the robot control system may monitor the sensor measurements. In step 708, the robot control system may determine whether a position correction condition is met. In step 710, the robot control system may generate instructions to move the robot.

[0122] In further detail of method 700, in step 702, a robotic control system (e.g., robotic control system 605, etc.) can identify measurements from sensors (e.g., sensor 655, etc.). The robotic control system can store the sensor measurements in a memory of the robotic control system. In some implementations, the robotic control system can periodically transmit requests to the sensors, e.g., at a predetermined time period. In response to the requests, the sensors can transmit each sensor's measurement to the robotic control system. The robotic control system can store the sensor measurements, e.g., over time or otherwise in association with a timestamp corresponding to when the sensor measurement was captured. In some implementations, the robotic control system can store each sensor measurement in association with an identifier for the sensor from which the measurement was captured. In some implementations, the robotic control system can store an association between each measurement and an object to which the respective sensor is coupled (e.g., a patient, a robot, an instrument such as instrument 630, etc.). This allows components of the robotic control system to measure or detect forces experienced or generated by each object in the surgical environment.

[0123] In step 704, the robotic control system can determine a position of the surgical robot (e.g., robot 620). As described herein above, an image capture device (e.g., image capture device 104) can capture an image of a patient within a surgical environment. The image capture device can have a predefined orientation within the surgical environment relative to other sensors, such as an IR sensor (e.g., IR sensor 220). As described herein above, the IR sensor can be an IR camera that captures IR wavelengths of light. The robotic control system can determine the position of the robot by utilizing one or more tracking techniques, such as IR tracking techniques or computer vision tracking techniques, to determine the position of the robot. The robot can include one or more markers or indicators, such as IR indicators, on its surface. The robotic control system can utilize the IR sensor to determine the relative position of the robot within the surgical environment. In some implementations, the robotic control system can determine the orientation (e.g., pose) of the robot by performing a similar technique. The IR sensor (or other sensor) used to track the robot's position is a known distance from the image capture device, so that the positions of the markers detected by the IR sensor can be mapped to the same reference coordinate system as the 3D point cloud captured by the image capture device.

[0124] In some implementations, the image capture device may be used to determine the position of the robot. For example, the robot may include one or more graphical indicators, such as bright distinct colors, patterns, or QR codes, among others. The image capture device may capture an image of the surgical environment, and the robotic control system may perform image analysis techniques to determine the position of the surgical robot in the image 208 based on the detected positions of the indicators in the image. The position and orientation of the surgical robot may be calculated periodically, for example, in real-time or near real-time, allowing the robotic control system to track the movement of the robot over time. In some implementations, the robotic control system may perform a calibration procedure, for example, to establish a unified reference frame between the 3D point cloud and the indicators coupled to the surgical robot. The calibration procedure may include identifying the position of the robot relative to a global indicator positioned in the surgical environment. The robotic control system may store the position of the robot in a memory of the robotic control system, such that the real-time or near real-time position of the robot may be accessed by other components of the robotic control system. The calibration procedure may include using predefined markers or patterns (eg, a checkerboard pattern, etc.) within the surgical environment to calibrate the posture (eg, position and orientation) of the robot relative to the patient.

[0125] Tracking the position of the robot may include tracking the position of an instrument (e.g., instrument 630). As described herein above, the instrument may include its own indicator coupled to the instrument or a bracket / connector coupling the instrument to the robot. An example of an indicator coupled to an instrument is shown as tracked instrument 340 in the close-up 312 shown in FIG. 3. The robot control system may track the position and orientation of the instrument and robot in real-time or near real-time using techniques similar to those described above. The instrument position and orientation determined by the robot control system may be stored in a memory of the robot control system. In some implementations, the robot control system may store the instrument position and orientation in association with a respective timestamp. In some implementations, tracking the position of the robot or instrument may also be performed based on measurements captured from one or more sensors coupled to the robot or instrument. For example, the robot control system may determine the position of the robot using, for example, acceleration or velocity values ​​(captured from the sensors) and interpolate the position of the robot or instrument over time.

[0126] In step 706, the robotic control system can monitor sensor measurements. As described herein above, in some implementations, one or more of the sensors can be positioned on one or more parts of the patient. Measurements captured from these sensors can indicate to the robotic control system that the patient's position has changed over time (e.g., exerted accelerations, forces or torques since the beginning of the surgical procedure, etc.). The robotic control system can detect the patient's movement by integrating (e.g., in the case of acceleration or velocity measurements) or interpolating (e.g., in the case of position measurements, etc.) the patient's position over time. The robotic control system can store the detected positions of the patient in association with respective timestamps corresponding to when the corresponding sensor measurements were captured. In this manner, the robotic control system can determine and record the patient's position over time within the surgical environment.

[0127] In step 708, the robotic control system can determine whether a position correction condition is met. The robotic control system can detect whether the patient's position has met the position correction condition. The position correction condition can be a predetermined amount of movement or deviation from an initial condition of the surgical procedure. Using the recorded position values ​​of the patient determined from the sensor measurements, the robotic control system can determine the amount of movement over time by comparing the patient's current position (e.g., from the most recent sensor measurements) to the patient's initial condition. If the position correction condition is met, the robotic navigator can adjust the position of the robot or instrument to minimize the occurrence of injury to the patient.

[0128] In some implementations, the robotic control system can determine that a position correction condition is met if the determined movement of the patient over time exceeds or equals a predetermined threshold movement amount. In some implementations, the patient movement can be measured from the patient's position following a previous position adjustment of the robot such that the adjusted position becomes a new baseline for subsequent adjustments of the robot or instrument position. As described above herein, the robot can navigate along a predetermined trajectory such that the instrument can interact with a target location to perform a portion of a surgical procedure. In some implementations, the robotic control system can detect that a position correction condition is met by determining that the position of the surgical robot has deviated from the predetermined trajectory. For example, measurements of one or more sensors positioned on the robot can indicate that the robot has received an external force (e.g., from a collision with another object) that has caused or will cause the robot to deviate from the predetermined trajectory. In response to the detected force, the robotic control system can generate a signal that causes the robotic control system to navigate the robot back to the predetermined trajectory.

[0129] In some implementations, one or more of the measurements from the sensors may indicate that a collision has occurred between the robot or instrument and another object in the surgical environment. For example, one or more force sensors may indicate that a collision has occurred against a surface on the patient. In some implementations, the robotic control system may detect that a position correction condition has been met by determining that a collision with the robot (or instrument) has occurred based on a set of measurements. If a collision has occurred, the robotic control system may generate a signal that may include information about the nature of the collision (e.g., the direction of the force received, the direction in which the robot should move to avoid further collisions or harmful collisions, etc.). This information may be used by the robotic control system to navigate the robot to avoid further collisions. If the position correction condition is met, the robotic control system may perform step 710 of method 700. If the position correction condition is not met, the robotic control system may continue to monitor the sensor measurements at step 706 of method 700.

[0130] In step 710, the robot control system can generate instructions to move the robot. As described herein, when a position correction condition is met, the robot control system can generate instructions for the robot corresponding to the sensor measurements that triggered the position correction condition. As described above herein, the robot can change its position by executing or interpreting instructions or signals that indicate a target position for the instrument. For example, the robot can execute such instructions that indicate a target position for the instrument and actuate various moving components within the robot to move the instrument to the target position. The robot control system can navigate the robot according to the patient's movements such that the robot or instrument is aligned with a target position or target path within the surgical environment even when the patient is not immobilized.

[0131] When patient movement is detected by the robotic control system, the robotic control system can generate corresponding instructions to move the robot according to the patient movement. For example, sensor measurements monitored by the robot tracker 640 can indicate that the patient has moved a predetermined amount in a predetermined direction. The robotic control system can generate instructions for the robot to move the instrument according to the patient movement such that the instrument remains on a predetermined path or trajectory that leads to a selected target location on or within the patient. In some implementations, the robotic control system can generate instructions to compensate from external forces experienced by the robot. For example, the instructions can move the robot against the detected forces to remain in the proper position relative to the patient.

[0132] The robotic control system can perform these adjustments iteratively or periodically multiple times per second to compensate for sudden patient movement or detected forces. As described above herein, the robotic control system can calculate the patient movement periodically (e.g., according to the rate at which the sensors capture measurements, etc.). Each time a new measurement from the sensor is captured, the robotic control system can determine whether the change in the patient's (or robot's) position satisfies a position correction condition. The robotic control system can then adjust the position of the robot such that the instrument is aligned with a predefined path (e.g., a predefined trajectory) based on the condition (e.g., the measurement from the sensor) that triggered the position correction condition. In addition, the robotic control system can navigate the robot along a target path while compensating for patient movement in real time, as described herein. The robotic control system can navigate the robot in multiple scenarios. For example, the robotic control system can adjust the position of the robot while the robot is still in space (e.g., providing a rigid port for the patient). In some implementations, the robotic control system can adjust the position of the robot along one or more axes while the robot operator is moving the robot or instrument down through a predefined trajectory to a target position.

[0133] (D. Initiating Cooperative Control in Response to Detected Conditions) The robotic systems described herein may operate autonomously or in conjunction with input from a surgeon. For example, the surgical robots described herein may assist in guiding a surgical instrument along a predetermined path, while the surgeon can manually insert or remove the instrument from the patient, as well as actuate the instrument to perform the surgical procedure. However, in some cases, it is desirable for the surgeon to have full manual control over the position and trajectory of the surgical instrument. The systems and methods described herein provide improved techniques for detecting and managing such conditions. The techniques described herein may be used to detect conditions in which manual control over the surgical instrument should be established, and to generate commands for the surgical robot to initiate manual control. Manual control may be referred to as "collaborative control" because the surgical robot may still support the weight of the instrument while the position of the instrument may be manually guided by the surgeon. These and other improvements are described in detail below.

[0134] 8, an exemplary system 800 for initiating coordinated control of a surgical robot (e.g., robotic system 300 or components thereof, such as robot arm 310, system 400 or components thereof, such as robot 420, system 600 or components thereof, such as robot 620, etc.) in response to a detected condition, according to one or more implementations. System 800 may include at least one robot control system 805, at least one robot 820, at least one image processing system 100, and one or more sensors 855. As with the various robots described herein, robot 820 may include one or more members that may be manually and / or automatically manipulated using various actuators in response to external forces and / or control signals. Robot control system 805 may include at least one robot navigator 835, at least one control condition detector 840, and at least one manual control initiator 845. Robot 820 may include an instrument 830.

[0135] Each of the components of the system 800 (e.g., the robot control system 805, the image processing system 100, the robot 820, etc.) may be implemented using hardware components or a combination of software with hardware components of a computing system (e.g., the computing system 1200 described in connection with FIGS. 12A and 12B). Each of the components of the robot control system 805 (e.g., the robot navigator 835, the control condition detector 840, the manual control initiator 845, etc.) may perform functions detailed herein. Although the image processing system 100 and the robot control system 805 are shown as separate systems, it should be understood that the robot control system 805 may be a part of the image processing system 100 (e.g., implemented at least in part by the processing circuitry 212, etc.) or vice versa (e.g., the processing circuitry 212 of the image processing system 100 implemented on one or more processors of the robot control system 805). Similarly, the robot control system 805 may be implemented with or include the image processing system 1000 described in connection with FIG. 10. In implementations in which the image processing system 100 and the robot control system 805 are implemented as separate computing systems, the image processing system 100 and the robot control system 805 may exchange information via a communication interface, as described herein. Similarly, the robot control system 805 and the robot 820 may communicate via one or more communication interfaces. The robot control system 805 may communicate any generated instructions to the robot 820 for execution.

[0136] The robot 820 and the tool 830 may be similar to and include any of the structures and functions of any of the robots or tools described herein (e.g., robot 420 or 620, tool 430 or 630, etc.). The sensor 855 may be similar to and include any of the structures and functions of the sensor 655 described herein above in connection with FIG. 6. Additionally, the robot control system 805 may include any of the structures or functions of the robot control system 405 or 605. The robot control system 805 may be or form part of the image processing system 100 described herein in conjunction with FIGS. 1A, 1B, and 2, and may perform any of the functions of the image processing system 100 as described herein. The robot control system 805 may include at least one processor and a memory (e.g., processing circuitry). The memory may store processor-executable instructions that, when executed by the processor, cause the processor to perform one or more of the operations described herein. The processor may include a microprocessor, an ASIC, an FPGA, a GPU, or the like, or a combination thereof. The memory may include, but is not limited to, electronic, optical, magnetic, or any other storage or transmission device capable of providing program instructions to the processor. The memory may further include memory chips, ASICs, FPGAs, ROM, RAM, EEPROM, EPROM, flash memory, optical media, or any other suitable memory from which the processor can read instructions. The instructions may include code from any suitable computer programming language. The robotic control system 805 may include one or more computing devices or servers capable of performing various functions described herein. The robotic control system 805 may include any or all of the components and perform any or all of the functions of the computer system 1000 described herein in connection with FIG. 12A and FIG. 12B.

[0137] Referring now to the operation of the robot control system 805, the robot navigator 835 can control the position of the robot 820 and the instrument 830 within a surgical environment, including the patient. For example, the robot navigator 835 can identify one or more predefined trajectories along which the instrument 830 should be guided to perform a surgical procedure. In some implementations, the surgeon can pre-plan one or more trajectories and map the trajectories to one or more points of interest in a 3D image (e.g., a CT scan or an MRI scan) of the patient as described herein. When performing a procedure, in implementations where there are multiple targets or paths (or segments of a path), the surgeon may select, via user input (e.g., button selection, touch screen selection, etc.), one or more paths along which the robot 820 should navigate the instrument 830. The robot navigator 835 can receive the user input and identify one or more paths by processing the user input. The robot navigator 835 can navigate the robot 820 along the selected path while compensating for patient movement in real time as described herein.

[0138] The robot navigator 835 can navigate the robot 820 in multiple scenarios. For example, the robot navigator 835 can adjust the position of the robot 820 while the robot is still in space (e.g., providing a rigid port to the patient). In some implementations, the robot navigator 835 can adjust the position of the robot along one or more axes while the operator of the robot 820 moves the robot 820 or the instrument 830 down through a predetermined trajectory to a target location. The robot navigator 835 can further navigate the robot 820 using the techniques described above in this specification in connection with FIG. 4 and FIG. 6. For example, the robot navigator 835 can generate instructions to adjust the position of the robot 820 in response to the 3D point cloud representing the patient indicating that the patient has moved. Additionally, the robot navigator 835 can generate instructions to adjust the position of the robot in response to a condition of a measured signal from the sensor 855. In some implementations, the robot navigator 835 can navigate the robot 820 continuously or in near real-time to closely match the conditions of the surgical environment (e.g., sudden patient movements, instantaneous forces encountered, etc.).

[0139] The control condition detector 840 can detect a cooperative control condition of the robot 820 based on one or more conditions of the surgical environment. As described herein above, the robot 820 can operate autonomously, semi-autonomously, or manually, with the surgeon having full control over the position and orientation of the robot 820 and the instruments 830. Under certain conditions, it is advantageous to initiate cooperative control of the robot such that the surgeon has full manual control over the position and orientation of the surgical instruments. The control condition detector 840 can monitor the conditions of the surgical environment, for example, using information from the image processing system 100 or the sensor 855, and detect whether a cooperative control condition has been met.

[0140] In some implementations, the control condition detector 840 can access and monitor a 3D point cloud corresponding to the patient via the image processing system 100. As described in more detail herein above in connection with FIG. 4, the robot control system 805 (which may include any of the functions of the robot control system 405) can receive 3D images corresponding to the patient and identify one or more points in the 3D point cloud over time. By monitoring the positions of the points in the 3D point cloud, the control condition detector 840 can determine changes in the patient's position over time. In some cases, it is advantageous to allow the surgeon to have full manual control of the robot 820 and the instruments 830 when patient movement beyond one or more thresholds is detected. For example, in some implementations, if the patient moves by a predetermined displacement (e.g., a relatively long distance in a relatively short time) for a predetermined time, the control condition detector 840 can determine that a control condition is met and generate a signal for the robot control system 805 to initiate coordinated control of the robot 820.

[0141] In some implementations, the control condition detector 840 can identify measurements generated by the sensors 855. For example, to identify measurements from the sensors 855, the control condition detector 840 can perform operations as described herein above in connection with the measurement identifier 635 of FIG. 6. For example, in some implementations, one or more of the sensors 855 can be positioned on or coupled to the patient during a surgical procedure. The control condition detector 840 can identify measurements from the sensors 855, for example, to determine the amount of patient movement over time. Using the patient's recorded position values ​​determined from the sensor 855 measurements, the control condition detector 840 can determine the amount of movement over time by comparing the patient's current position (e.g., from the most recent sensor measurement) to the patient's initial condition. The control condition detector 840 can compare the amount of patient movement to a patient movement condition (e.g., a predetermined threshold of patient movement over time that triggers cooperative control). The control condition detector 840 can continuously or periodically monitor measurements from the sensors 855 such that patient movement can be monitored in real time. The control condition detector 840 can compare patient movement to a threshold whenever patient movement is detected and generate a signal for the robot control system 805 to initiate coordinated control of the robot 820 when a movement condition is met.

[0142] In some implementations, one or more of the sensors 855 may be positioned at a junction between the robot 820 and the patient. For example, in some implementations, the robot 820 may be fixed to the patient (e.g., screwed into the patient's skull). If the control condition detector 840 detects measurements from a force sensor 855 on the robot indicating patient movement (e.g., exceeding a threshold value), the control condition detector 840 may generate a signal for the robot control system 805 to initiate coordinated control of the robot 820. In some implementations, one or more of the sensors 855 may be positioned on the instrument 830, such as a tip of a needle forming part of the instrument 830. The control condition detector 840 may detect that measurements from a sensor(s) 855 (e.g., an IMU sensor, an accelerometer, a torque sensor, etc.) positioned at the needle tip indicate that the needle tip has deviated from a predetermined path (e.g., the tip has bent or deformed due to an external force, etc.). In response, the control condition detector 840 can generate a signal for the robot control system 805 to initiate coordinated control of the robot 820.

[0143] In some implementations, the control condition detector 840 can detect a cooperative control condition based on signals received from sensors 855 positioned on the robot 820 or the instrument 830. For example, in some implementations, the sensors 855 can include forces applied to the robot 820 by a surgeon operating the robot 820. If the surgeon applies a force (e.g., a sudden movement, etc.) that exceeds a predetermined threshold within a predetermined time, this can indicate that the surgeon is attempting to take manual control of the robot 820. In response, the control condition detector 840 can generate a corresponding signal to initiate cooperative control of the robot 820. The control condition detector 840 can also detect collision events across the robot 820 from sensors coupled to the robot 820. For example, if the robot 820 is a robotic arm, such as the robot arm 310, one or more of the sensors 855 can be positioned on the robot arm and provide a measurement of an external force (e.g., a collision) experienced by the robot 820. The control condition detector 840 can detect any collision with the robot 820 based on measurements from the sensor 855 (e.g., measurements exceeding a threshold indicating unexpected movement) and generate a signal for the robot control system 805 to initiate coordinated control of the robot 820.

[0144] The control condition detector 840 can detect a cooperative control condition based on an error condition in an image-to-patient registration process, such as the image-to-patient process performed by the image registration component 445 described herein above in connection with FIG. 4. As described herein, the robotic control system 805 can perform image-to-patient registration techniques to align a 3D image of the patient, which may include indicators of potential targets, with a 3D point cloud captured from the patient. The 3D image, such as a CT scan of the patient's head, may include both a 3D representation of the patient's face and an indication of a biopsy site in the patient's brain. By registering the 3D image of the patient with the real-time 3D point cloud of the patient in the surgical environment, the robotic control system 805 can map the 3D image of the patient and any target indicators to the same reference coordinate system as both the 3D point cloud, the tracked position of the robot 820, and the tracked position of the instrument 430. Doing so allows the robotic control system 805 to track the position of the instrument 830 relative to both the patient and the target area indicated in the 3D image of the patient. To register the 3D image to the patient's 3D point cloud, the robot control system 805 can perform an iterative fitting process, such as a RANSAC algorithm and an iterative closest point algorithm. The robot control system 805 can continuously register the 3D image to the patient's 3D point cloud. If the registration fails (e.g., the fitting algorithm fails to fit the 3D image to the 3D point cloud within a predetermined error threshold), the robot control system 805 can generate a signal indicating the failure. In response to detecting the failure signal from the robot control system 805, the control condition detector 840 can generate a signal for the robot control system 805 to initiate cooperative control of the robot 820 if a condition is met.

[0145] In addition, the surgeon can manually initiate cooperative control of the robot 820. As described herein, the robot 820 may include one or more buttons or may be controlled through one or more user interfaces, such as a touch screen. In some implementations, a button positioned on the robot 820 (such as one of the buttons 335 described herein in connection with FIG. 3 ) or an available object on the user interface 120 (such as, for example, a graphical button, hyperlink, or other user interface element) can generate a signal to the control condition detector 840 to initiate cooperative control. The control condition detector 840 can receive a signal indicative of an interaction with a button corresponding to a cooperative control condition and generate a signal for the robot control system 805 to initiate cooperative control of the robot 820.

[0146] Upon detecting the signal generated by the control condition detector 840, the manual control initiator 845 can generate instructions for the robot 820 to provide manual control to the surgeon operating the robot 820. In some implementations, the manual control initiator 845 can store an indication of the collaborative control event, and any conditions that caused the collaborative control event, in one or more data structures in the memory of the robot control system 805. In some implementations, the manual control initiator 845 can store the collaborative control event in association with a timestamp indicating when the collaborative control event occurred. The manual control initiator 845 can use the communication interface to communicate instructions that can be generated using one or more APIs corresponding to the robot 820. In some implementations, the manual control initiator 845 can receive an instruction to resume automatic navigation of the robot 820 (e.g., via user input from the surgeon, pressing a button, interaction on the user interface 120, etc.), and the manual control initiator 845 can generate a signal for the robot navigator 835 to continue navigating the robot according to the target trajectory.

[0147] 9, an exemplary method 900 of controlling a surgical robot based on patient tracking technology is shown, according to one or more implementations. The method 900 may be performed by, for example, the robot control system 405, 605, or 805, or any other computing device described herein, including the computing system 1000 described herein in connection with FIGS. 12A and 12B. In overview of the method 900, in step 902, a robot control system (e.g., robot control system 605, etc.) may navigate a surgical robot (e.g., robot 820). In step 904, the robot control system may determine whether a cooperative control condition is detected. In step 906, the robot control system may generate instructions to initiate cooperative control of the robot.

[0148] In further detail of method 900, in step 902, a robotic control system (e.g., robotic control system 605, etc.) can navigate a surgical robot (e.g., robot 820). The robotic control system can control the position of the robot and an instrument coupled to the robot (e.g., instrument 830, etc.) within a surgical environment including a patient. For example, the robotic control system can identify one or more predefined trajectories along which the instrument should be guided to perform a surgical procedure. In some implementations, the surgeon can pre-plan one or more trajectories and map the trajectories to one or more points of interest within a 3D image (e.g., a CT scan or an MRI scan) of the patient as described herein. When performing a procedure, in implementations where there are multiple targets or paths (or segments of a path), the surgeon may select one or more paths along which the robot should navigate the instrument via user input (e.g., button selection, touch screen selection, etc.). The robotic control system can navigate the robot along the selected paths while compensating for patient movement in real time as described herein.

[0149] The robotic control system can navigate the robot in multiple scenarios. For example, the robotic control system can adjust the position of the robot while it is still in space (e.g., providing a rigid port to the patient). In some implementations, the robotic control system can adjust the position of the robot along one or more axes while the robot operator is moving the robot or instrument down through a predefined trajectory to a target location. The robotic control system can further navigate the robot using the techniques described above in this specification in connection with FIG. 4 and FIG. 6. For example, the robotic control system can generate instructions to adjust the position of the robot in response to a 3D point cloud representing the patient indicating that the patient has moved. In addition, the robotic control system can generate instructions to adjust the position of the robot in response to conditions of measured signals from one or more sensors (e.g., sensor 855). In some implementations, the robotic control system can navigate the robot continuously or in near real-time to closely match the conditions of the surgical environment (e.g., sudden patient movement, instantaneous forces experienced, etc.).

[0150] In step 904, the robot control system can determine whether a cooperative control condition is detected. The robot control system can detect a cooperative control condition of the robot based on one or more conditions of the surgical environment. As described above in this specification, the robot can operate autonomously, semi-autonomously, or manually, and the surgeon has full control over the position and orientation of the robot and the instruments. Under certain conditions, it is advantageous to initiate cooperative control of the robot such that the surgeon has full manual control over the position and orientation of the surgical instruments. The robot control system can monitor the conditions of the surgical environment, for example, using information from an image processing system (e.g., image processing system 100) or a sensor, to detect whether a cooperative control condition is met.

[0151] In some implementations, the robotic control system can access and monitor a 3D point cloud corresponding to the patient via the image processing system 100. As described in more detail herein above in connection with FIG. 4, the robotic control system (which may include any of the functions of the robotic control system 405) can receive 3D images corresponding to the patient and identify one or more points in the 3D point cloud over time. By monitoring the positions of the points in the 3D point cloud, the robotic control system can determine changes in the patient's position over time. In some cases, it is advantageous to allow the surgeon to have full manual control of the robot and instruments when patient movement beyond one or more thresholds is detected. For example, in some implementations, if the patient moves by a predetermined displacement over a predetermined time (e.g., a relatively long distance in a relatively short time), the robotic control system can determine that a control condition is met and generate a signal for the robotic control system to initiate coordinated control of the robot.

[0152] In some implementations, the robotic control system can identify measurements generated by the sensors. For example, to identify measurements from the sensors, the robotic control system can perform operations as described above in this specification in connection with the measurement identifier 635 of FIG. 6. For example, in some implementations, one or more of the sensors can be positioned on or coupled to the patient during a surgical procedure. The robotic control system can identify measurements from the sensors, for example, to determine the amount of patient movement over time. Using the recorded position values ​​of the patient determined from the sensor measurements, the robotic control system can determine the amount of movement over time by comparing the patient's current position (e.g., from the most recent sensor measurement) to an initial condition of the patient. The robotic control system can compare the amount of patient movement to a patient movement condition (e.g., a predetermined threshold of patient movement over time that triggers cooperative control). The robotic control system can continuously or periodically monitor measurements from the sensors such that patient movement can be monitored in real time. The robotic control system can compare the patient movement to a threshold each time patient movement is detected and generate a signal to the robotic control system to initiate cooperative control of the robot if the movement condition is met.

[0153] In some implementations, one or more of the sensors may be positioned at a junction between the robot and the patient. For example, in some implementations, the robot may be fixed to the patient (e.g., screwed into the patient's skull). If the robot control system detects measurements from a force sensor on the robot that indicate patient movement (e.g., exceeding a threshold value), the robot control system may generate a signal for the robot control system to initiate coordinated control of the robot. In some implementations, one or more of the sensors may be positioned on an instrument, such as a tip of a needle forming part of the instrument. The robot control system may detect that measurements from a sensor(s) (e.g., an IMU sensor, an accelerometer, a torque sensor, etc.) positioned at the needle tip indicate that the needle tip has deviated from a predetermined path (e.g., the tip has bent or deformed due to an external force, etc.). In response, the robot control system may generate a signal for the robot control system to initiate coordinated control of the robot.

[0154] In some implementations, the robotic control system can detect a cooperative control condition based on signals received from sensors positioned on the robot or the instrument. For example, in some implementations, the sensors can include forces applied to the robot by a surgeon operating the robot. If the surgeon applies a force (e.g., a sudden movement, etc.) that exceeds a predetermined threshold within a predetermined time, this can indicate that the surgeon is attempting to take manual control of the robot. In response, the robotic control system can generate a corresponding signal to initiate cooperative control of the robot. The robotic control system can also detect collision events across the robot from sensors coupled to the robot. For example, if the robot is a robotic arm, such as the robotic arm 310, one or more of the sensors can be positioned on the robot arm and provide measurements of external forces (e.g., collisions) experienced by the robot. The robotic control system can detect any collision with the robot based on measurements from the sensors (e.g., measurements that exceed a threshold that indicate an unexpected movement, etc.) and generate a signal for the robotic control system to initiate cooperative control of the robot.

[0155] The robotic control system can detect the cooperative control condition based on an error condition in an image-to-patient registration process, such as the image-to-patient process performed by the image registration component 445 described herein above in connection with FIG. 4. As described herein, the robotic control system can perform image-to-patient registration techniques to align a 3D image of the patient, which may include indicators of potential targets, with a 3D point cloud captured from the patient. The 3D image, such as a CT scan of the patient's head, may include both a 3D representation of the patient's face and an indication of a biopsy site in the patient's brain. By registering the 3D image of the patient with the real-time 3D point cloud of the patient in the surgical environment, the robotic control system can map the 3D image of the patient and any target indicators to the same reference coordinate system as both the 3D point cloud, the tracked position of the robot, and the tracked position of the instrument 430. Doing so allows the robotic control system to track the position of the instrument relative to both the patient and the target area shown in the 3D image of the patient. To register the 3D image to the 3D point cloud of the patient, the robotic control system can perform an iterative fitting process, such as a RANSAC algorithm and an iterative closest point algorithm. The robotic control system can continuously register the 3D image to the patient's 3D point cloud. If the registration fails (e.g., a fitting algorithm fails to match the 3D image to the 3D point cloud within a predetermined error threshold), the robotic control system can generate a signal indicating the failure. In response to detecting the failure signal from the robotic control system, the robotic control system can generate a signal for the robotic control system to initiate coordinated control of the robot if a condition is met.

[0156] In addition, the surgeon may manually initiate collaborative control of the robot. As described herein, the robot may include one or more buttons or may be controlled through one or more user interfaces, such as a touch screen. In some implementations, a button positioned on the robot (such as one of the buttons 335 described herein in connection with FIG. 3) or an available object on the user interface (such as a graphical button, hyperlink, or other user interface element) can generate a signal to the robot control system to initiate collaborative control. The robot control system can receive a signal indicating an interaction with the button corresponding to a collaborative control condition and generate a signal for the robot control system to initiate collaborative control of the robot.

[0157] In step 906, the robot control system can generate instructions to begin cooperative control of the robot. Upon detecting the signal generated by the control condition detector 840, the robot control system can generate instructions for the robot to provide manual control to the surgeon operating the robot. In some implementations, the robot control system can store an indication of the cooperative control event and any conditions that caused the cooperative control event in one or more data structures in the memory of the robot control system. In some implementations, the robot control system can store the cooperative control event in association with a timestamp indicating when the cooperative control event occurred. The robot control system can use the communication interface to communicate instructions that may be generated using one or more APIs corresponding to the robot. In some implementations, the robot control system can receive an instruction to resume automatic navigation of the robot (e.g., via user input from the surgeon, button press, interaction at the user interface 120, etc.), and the robot control system can generate a signal for the robot control system to continue navigating the robot according to the target trajectory.

[0158] (E. Real-time non-invasive surgical navigation technology) A real-time surface-based registration system, such as that described with respect to FIG. 1, among other figures, can track the position of pre-planned brain targets during a procedure. For example, 3D camera data can be aligned with medical image (e.g., CT or MRI) data and tracked in real time to track the pre-planned targets. The system can then control the position of a surgical device, such as a robotic device, to orient the instrument relative to the target. Various brain-related procedures typically require a skull clamp or other device to restrict movement of the subject's head, which can make performing the procedure more uncomfortable and time-consuming. However, performing non-invasive procedures without a skull clamp requires constant adjustments to account for patient movement. For example, in transcranial magnetic stimulation (TMS), the practitioner specifically targets an area of ​​the cortex to stimulate neurons. Current practice approximates the target area by marking the patient. Without precise targeting, sudden movements can result in stimulation in undesired skull areas with uncertain side effects. In addition, the skull can cause significant diffraction of signals, such as TMS or ultrasound signals, further complicating precise treatment delivery.

[0159] The present solution can map the patient's cortex before delivering the treatment via a CT scan. This allows for internal navigation of the patient's morphology to precisely target the location of interest. Once the treatment delivery begins, the present solution can automatically stop emitting energy from the treatment device (or otherwise adjust the amount of energy emitted) when the system detects incorrect alignment, the patient moves too fast, etc. It can also automatically stop delivering energy once a treatment condition is met, e.g., if a predefined treatment threshold is achieved. The present solution can use data such as the patient's morphology and the location of interest for treatment. In addition, the present solution can combine focal steering within the treatment device to achieve fine adjustment of the focal spot.

[0160] For applications that require the device to be in contact with the patient's skin, the solution can combine torque sensing with a surface-based alignment system. The solution can utilize instrument tracking data as well as data collected from 3D image processing to monitor surface contact. This creates a condition where the device can remain on the target while in contact with the patient's surface, such as the skin or scalp, and apply a predefined amount of force or range of forces to the surface. If the patient moves slightly, the solution can adjust and remain in contact with the target position with the same predefined amount of force. This can allow for precise treatment delivery as well as patient comfort, as treatment sessions can last for hours.

[0161] Figure 10 illustrates an example of a system 1000. System 1000 may incorporate features of various systems and devices described herein, including, but not limited to, system 200 described with reference to Figure 2. System 1000 may be used to perform non-invasive procedures, particularly real-time non-invasive procedures on or around a subject's head, including delivering therapy to the subject's brain.

[0162] The system 1000 may include features of the image processing system 1000 described with reference to Figures 1 and 2. The image processing system 1000 may include one or more image capture devices 1002 that may be similar to and include any of the structure and functionality of the image capture device 1002 described with respect to Figures 1 and 2. Each of the image capture devices 1002 may include one or more lenses 1003 that may be similar to and include any of the structure and functionality of the lens 1003 associated with Figures 1 and 2. The lenses 1003 may receive light indicative of an image. The image capture devices 1002 may include sensor circuitry that may detect light received through the one or more lenses 1003 and generate an image 1007 based on the received light. The image 1007 may be similar to the image 1007 described with respect to Figures 1 and 2.

[0163] The image processing system 1000 may include a communications circuit 216. The communications circuit 1016 may implement features of the computing device 1200 described with reference to Figures 12A and 12B. The communications circuit 1016 may be similar to the communications circuit 1016 described in connection with Figures 1 and 2 and may incorporate any of the structure or functionality of the communications circuit 1016 described in connection with Figures 1 and 2.

[0164] The image processing system 1000 may include one or more tracking sensors, such as an IR sensor 1018 and an image capture device 1002. The IR sensor 1018 may be similar to, and may include any of the structure or functionality of, the IR sensor 220 described in connection with Figures 1 and 2. The IR sensor 1018 may detect IR signals from various devices in the environment surrounding the image processing system 100. The IR sensor 1018 may be communicatively coupled to other components of the image processing system 100 such that the components of the image processing system 1000 can utilize the IR signals in the appropriate operations in the image processing pipeline.

[0165] The image processing system 1000 may include a surgical instrument 1004. The surgical instrument 1004 may deliver a treatment to a subject, and its relative position 1008 may be determined by an image 1007 captured by the image capture device 1002. The parameter 1012 represents an amount of energy delivered and processed by a processing circuit 1014. The processing circuit 1014 may be similar to the processing circuit 1014 described herein and may include any of the structure or functionality thereof. Similarly, the image processing system 1000 of FIG. 10 may be implemented in addition to or as an alternative to the image processing system 100 of FIGS. 1 and 2 to perform any of the functions described herein. The surgical instrument 1004 may be, for example, a focused ultrasound device, a transducer, a magnetic coil, or the like.

[0166] Two or more capture devices 1002 can capture 3D images of an object for accuracy and overall resolution. Processing circuitry 1014 can extract 3D data from each data point in the image 1007 received from the image capture devices 1002 and generate a point cloud corresponding to each capture device 1002. In some implementations, processing circuitry 1014 can downsample the data points to reduce the overall size of the image 1007 and improve image processing without significantly affecting the accuracy of further processing steps.

[0167] The processing circuit 1014 can select one of the point clouds to act as a reference coordinate system for alignment of any of the other point clouds. Selecting a reference coordinate system may include retrieving color data assigned to one or more of the first set of data points of the first point cloud, and the processing circuit 1014 can extract the color data. Selecting a reference coordinate system may also include determining the most illuminated point cloud, the least uniformly illuminated point cloud, or the processing circuit 1014 can arbitrarily select a reference coordinate system of a point cloud as the reference coordinate system.

[0168] The processing circuitry 1014 can determine a transformation data structure such that when each matrix is ​​applied to a respective point cloud, features of the transformed point cloud align with similar features in the reference coordinate system point cloud. The transformation matrices include transformation values ​​that indicate the change in position or rotation of points in the transformed point cloud.

[0169] Using information from the global scene, the processing circuitry 1014 can determine a location of interest in a first reference coordinate system associated with the first point cloud and the 3D medical image. If a location of interest is detected, the processing circuitry 1014 can generate a highlighted region in the display data rendered to the user interface 1020. The user interface 1020 can be similar to the user interface 120 described in connection with Figures 1 and 2 and can include any of its structure and functionality. This location of interest can be input by a medical professional for non-invasive applications of the image processing system 1000.

[0170] The processing circuit 1014 can be configured to determine distances of objects represented in the 3D medical images from the image capture device 1002 that is at least partially involved in generating the first point cloud. If a reference object or marker with a known distance or length is present in the global scene, the processing circuit 1014 can use the known distance or length to determine distances from the image capture device 1002 to other features in the global scene. The processing circuit 1014 can determine an average position of the objects using features of the objects in the global point cloud that correspond to features in the 3D medical images.

[0171] The processing circuit 1014 can use this same method to determine the average position of the surgical instrument 1004. A computer generated model of the surgical instrument 1004 can be aligned by the processing circuit 1014 and matched to the 3D image data collected by the image capture device 1002. The processing circuit 1014 can use the known distance or length to calculate different dimensions or parameters of the global scene point cloud to determine the distance of the surgical instrument 1004 from the image capture device 1002. Using the features and relative position 1008 of the object, the processing circuit can determine the distance of the surgical instrument 1004 to the object by processing the tracking data collected by the IR sensor 1018 and a reference coordinate system aligned with the 3D image data captured by the image capture device 1002. The relative position 1008 of the surgical instrument 1004 can be continuously tracked by the image capture device 1002 in parallel with the IR sensor 1018, and the tracking data is transmitted to the processing circuit 1014.

[0172] The surgical instrument 1004 can deliver a treatment to the location of interest. The processing circuit 1014 can communicate with the surgical instrument 1004 via the communication circuit 1016. The processing circuit 1014 can track the total amount of energy being delivered to the location of interest through parameters 1012. The processing circuit 1014 can also track the total amount of energy being delivered over time. The processing circuit 1014 can terminate, reduce the amount of energy being output, or otherwise change parameters of the energy delivery of the treatment to the location of interest if parameters 1012 are met or if the location of interest is no longer aligned with the surgical instrument.

[0173] In some implementations, the processing circuit 1014 can communicate an internal mapping of the object that the surgical instrument 1004 is targeting to the user interface 1020 via the display data through the communication circuit 1016. The processing circuit 1014 can use 3D medical imaging data (e.g., CT, MRI) and align the data to a global scene to generate the display data. For example, the surgical instrument 1004 can be a transducer that targets a specific internal location of the object.

[0174] The processing circuit 1014 can calculate and provide updated information regarding the relative position 1008 to the location of interest via the IR sensor 1018. The processing circuit 1014 aligns the initial alignment of the location of interest and the relative position 1008 through the tracking information received from the IR sensor 1018 and the 3D image data from the image capture device 1002. If the processing circuit 1014 detects movement of the location of interest at a speed below an acceptable speed threshold, the processing circuit 1014 will generate a movement command to the surgical instrument 1004 to realign with the location of interest. If the processing circuit 1014 detects movement with a distance below an acceptable distance threshold for the location of interest, the processing circuit 1014 will generate a movement command to the surgical instrument 1004 to realign with the location of interest. If the processing circuit 1014 detects movement with a speed above an acceptable speed threshold for the location of interest, the processing circuit 1014 will transmit a termination command through the communication circuit 1016. If the processing circuit 1014 detects movement of the location of interest a distance greater than the allowable distance threshold, the processing circuit 1014 will transmit a termination command over the communications circuit 1016 .

[0175] In some implementations, the surgical instrument 1004 can be in contact with the object. The processing circuit 1014 aligns the relative position 1008 of the global scene and the surgical instrument 1004 to the location of interest. The processing circuit 1014 can receive information from a sensor, such as an IR sensor 1018, and processes lateral and rotational movement of the location of interest. The processing circuit 1014 can generate movement instructions to keep the surgical instrument 1004 in contact with the object with a predetermined amount of force. The processing circuit 1014 can generate movement instructions to keep the surgical instrument 1004 in contact with the object with a predetermined amount of torque. The processing circuit 1014 transmits the movement instructions via the communication circuit 1016 so that the system 1000 includes torque sensing in its non-invasive surgical navigation.

[0176] In some implementations, the surgical instrument 1004 can output an ultrasound signal for treatment delivery purposes. For example, in focused ultrasound therapy, the surgical instrument 1004 can deliver ultrasound to a location of interest, particularly to open the blood-brain barrier and non-invasively deliver drug treatments. In some implementations, the surgical instrument 1004 can include multiple transmission elements, such as ultrasound transducers, arranged in an array. In some implementations, the surgical instrument 1004 can perform beamforming using multiple ultrasound transmission elements to overlap wavefronts to generate a plane wave. In some implementations, the surgical instrument 1004 can control various parameters, such as wavenumber, to control and steer the outputted ultrasound signal. The processing circuit 1014 can control the surgical instrument 1004 to perform focal steering of the ultrasound beam, control phased array operation or other operation of the surgical instrument 1004, and control at least one of the position and direction of the ultrasound beam based on at least one of the tracking data of the surgical instrument 1004 or the target parameters of the procedure being performed using the ultrasound beam.

[0177]

[0013] Figure 11 illustrates a method 1100 for real-time non-invasive surgical navigation to facilitate delivery of a procedure to a location of interest on a subject, such as the subject's head. Method 1100 may be performed using any of the computing systems described herein, including image processing system 100 of Figure 1, image processing system 1000 of Figure 10, robotic control system 405 of Figure 4, robotic control system 605 of Figure 6, robotic control system 805 of Figure 8, or computer system 1200 of Figures 12A and 12B. It will be understood that the steps of method 1100 may be completed in any order, including performing additional steps or omitting certain steps, to achieve desired results.

[0178] Method 1100 can be performed using magnetic coils for transcranial magnetic stimulation, high power ultrasound, or other surgical instruments used for non-invasive cranial procedures. For example, method 1100 can be performed to control how the treatment is performed, including maintaining alignment between real-time 3D image data of the subject and model data of the subject (e.g., model data from 3D medical image data such as CT or MRI examination data), controlling the surgical instrument to apply a treatment to the subject based on the alignment of the position of interest associated with the model data and the real-time 3D image data of the subject, monitoring the treatment and the relative position of the surgical instrument and the subject, and in some implementations, force (e.g., torque) data indicative of contact between the surgical instrument and the subject, and responsive to the monitoring, terminating the treatment, adjusting the energy or other output of the surgical instrument, and / or moving the surgical instrument. This can allow the treatment to be performed more precisely with less chance of off-target delivery of therapy, such as magnetic or ultrasound signals, to the subject.

[0179] In step 1105, the 3D image is registered with respect to the medical image of the object. The medical image may include CT or MRI image data, which may be used as a model of the object. The medical image may include 3D medical image data. The 3D image may be a point cloud captured using one or more image capture devices (e.g., such as image capture device 1002 of FIG. 10). The point cloud may show the patient in a surgical environment. A single capture of a 3D point cloud (sometimes referred to as a "3D image") may be referred to as a "frame" or a "point cloud frame" and may correspond to a single capture by an image capture device. The point cloud and / or reference coordinate system may be generated from the medical image or from the 3D image captured using the image capture device. The 3D image may be registered with respect to the medical image by registering or aligning the medical image with the 3D image using various methods described herein. The registration may be updated periodically as 3D image data is received, for example, from successive captures of 3D images using a 3D camera or other image capture device as described herein.

[0180] A 3D image registration process can be embedded using image-to-patient registration techniques to align a 3D image of a patient, which may include indicators of potential targets, with a 3D point cloud captured from the patient in the surgical environment. A 3D medical image, such as a CT scan of a patient's head, may include both a 3D representation of the patient's face and an indication of a target area within the patient's brain or another location of the patient's anatomy. By registering the 3D image of the patient with the real-time 3D point cloud of the patient in the surgical environment, the 3D image of the patient can be mapped to the medical image along with any target indicators in the same reference coordinate system as both the real-time 3D point cloud of the patient and any instruments in the environment surrounding the patient. Doing so can enable accurate tracking of the position of one or more instruments that may be utilized in a procedure involving the patient, relative to both the patient and the target area shown in the 3D image of the patient. To register the 3D image to the 3D point cloud of the patient, an iterative fitting process, such as a RANSAC algorithm or an iterative closest point algorithm, can be performed. In doing so, the 3D image can be continuously (or periodically, depending on the movement of the patient or instruments, etc.) registered to the 3D point cloud of the patient. If the registration fails (e.g., if the fitting algorithm fails to fit the 3D image to the 3D point cloud within a predetermined error threshold), a signal may be generated indicating the failure.

[0181] In step 1110, at least one of the surgical instrument and the object may be tracked, such as to track the position of the surgical instrument and / or object relative to a reference coordinate system or a particular position within the reference coordinate system. The surgical instrument and / or object may be tracked using various sensors, such as image capture devices (including 3D cameras used to detect 3D images), infrared sensors, torque or force sensors, or various combinations thereof. The surgical instrument and object may be tracked periodically, such as to periodically update the position of the surgical instrument and object relative to a model used to represent the object (and surgical instrument).

[0182] Tracking may include determining the position of the surgical robot, which may be mounted or coupled to a non-invasive instrument, in the same coordinate system of reference as the 3D point cloud of the patient in the surgical environment. As described herein, the image capture device may capture images of the patient in the surgical environment. The image capture device may have a predefined pose in the surgical environment relative to other sensors, such as an IR sensor (e.g., IR sensor 1018). As described above herein, the IR sensor may be an IR camera that captures IR wavelengths of light. One or more tracking techniques, such as IR tracking techniques, may be utilized to determine the position of the robot. The robot may include one or more markers or indicators, such as IR indicators, on its surface. The IR sensor may be used to determine the relative position of the robot. In some implementations, the orientation (e.g., pose) of the robot may be determined by performing a similar technique. Because the sensor used to track the position of the robot is a known distance from the image capture device, the position of the marker detected by the IR sensor may be mapped to the same coordinate system of reference as the 3D point cloud captured by the image capture device.

[0183] In some implementations, the image capture device may be used to determine the position of the robot. For example, the robot may include one or more graphical indicators, such as bright distinct colors or QR codes, among others. In addition to capturing the 3D point cloud of the patient, the image capture device may capture images of the surgical environment, and image analysis techniques may be performed to determine the position of the surgical robot in the images based on the detected positions of the indicators in the images. The position and orientation of the surgical robot may be calculated periodically, for example, in real-time or near real-time, allowing the robot control system to track the movement of the robot over time. In some implementations, a calibration procedure may be performed to establish a unified reference frame between the 3D point cloud and the indicators coupled to the surgical robot. The calibration procedure may include identifying the position of the robot relative to a global indicator positioned in the surgical environment. The position of the robot may be stored in one or more data structures in a computer-readable memory such that the real-time or near real-time position of the robot may be accessed by various components described herein.

[0184] Tracking the position of the robot may include tracking the position of an instrument coupled to the robot (e.g., surgical instrument 1004). As described herein above, the instrument may include its own indicator coupled to the instrument or a bracket / connector coupling the instrument to the robot. The position and orientation of the instrument as well as the robot may be tracked in real-time or near real-time using techniques similar to those described herein. The instrument positions and orientations may be stored in computer readable memory, for example, in association with respective timestamps.

[0185] Furthermore, the patient's position and movement can be precisely determined, and signals can be generated to adjust the robot's position accordingly. To do so, a previous position of the 3D point cloud representing the patient can be compared to the position of a current or new 3D point cloud captured by the image capture device. In some implementations, at least two sets of 3D point clouds can be maintained (e.g., stored, updated), one from a previously captured frame and another from a currently captured (e.g., most recently captured) frame. The comparison can be a distance (e.g., Euclidean distance, etc.) in a reference coordinate system of the 3D point cloud.

[0186] Iterative calculations can be performed to determine which 3D points in a first 3D point cloud (e.g., a previous frame) correspond to 3D points in a second 3D point cloud (e.g., a current frame). For example, an iterative ICP algorithm or a RANSAC fitting technique can be performed to approximate which points in the current frame correspond to points in the previous frame. The distance between the corresponding points can then be determined. In some implementations, to improve computational performance, the robotic control system can compare subsets of points (e.g., by performing a downsampling technique on the point clouds). For example, after determining point correspondence, a subset of matching point pairs can be selected between each point cloud for comparison (e.g., determining distance in space). In some implementations, an average movement of 3D points between frames can be calculated to determine the movement of the patient in the surgical environment. Similar techniques can be used to calculate the location or movement of a target location of a patient's anatomical structure in the surgical environment based on the image registration techniques described herein. Additionally, as described herein, sensor data from one or more torque sensors may be utilized to detect movement or position of a patient, a surgical robot, an instrument, or a target location of the patient's anatomy within a surgical environment.

[0187] At step 1115, alignment of the surgical instrument with the location of interest can be evaluated. The location of interest (sometimes referred to as a target location) can be a location on the subject, such as a location on the subject's skull related to the procedure. The location of interest can be identified in the medical image or 3D image data, for example, based on being marked by a user (e.g., a surgeon) in a treatment plan or other information related to the medical image. For example, the location of interest can be a site on the subject's head to which an ultrasound, magnetic, or other non-invasive signal is applied. The surgical instrument can be used to perform various procedures (e.g., including invasive and non-invasive procedures) on the location of interest of the subject. Alignment of the surgical instrument with the location of interest can be evaluated based on tracking data from tracking of the surgical instrument and the subject, and can be evaluated based on at least one of a detected distance between the surgical instrument and the location of interest compared to a target distance, and an orientation of the surgical instrument compared to a target orientation (e.g., an angle at which the surgical instrument should be positioned relative to the subject's head to facilitate effective treatment delivery). The output of the alignment assessment may include an indication as to whether the surgical instrument is aligned with the location of interest, such as by determining that the surgical instrument is (or is not) at a target distance or range of distances from the location of interest and oriented within an angle or range of angles relative to the surface of the object at the location of interest. Techniques similar to those described in Sections A-D may be performed to determine the position and orientation of the surgical instrument relative to the target location (as well as the target orientation of the procedure to be performed). If the instrument is aligned with the location of interest and the target orientation, step 1125 of method 1100 may be performed. Otherwise, if the instrument is not aligned with the location of interest or is not in the target orientation (or within any predetermined tolerance), step 1120 of method 1100 may be performed.

[0188] In step 1120, in response to detecting that the surgical instrument is not aligned with the location of interest, movement commands may be transmitted to at least one of the surgical instrument or a robotic device (e.g., a robotic arm) coupled to the surgical instrument to adjust the orientation of the surgical instrument to align it with the location of interest. The movement commands may be generated and transmitted periodically in response to periodic assessment of the alignment.

[0189] As described herein, the position or orientation of the robot may be adjusted when the robot (or a controller coupled to the robot) executes or interprets commands or signals that indicate a target position or target orientation for the instrument. For example, the robot may execute such commands that indicate a target position or target orientation for the instrument and actuate various movable components within the robot to move the instrument to the target position and target orientation. The robot control system may navigate the robot according to the movement of a target location on the patient or the patient's anatomy such that the robot or instrument is aligned with the target location or target path for the procedure even when the patient is not immobilized. In some implementations, the target location or target path may be defined as part of a 3D image (e.g., a CT scan or an MRI scan, etc.) that is registered to a real-time 3D point cloud that represents the patient in the reference coordinate system of the image capture device. The positions of the instrument and robot are also mapped into the same reference coordinate system so that an accurate measurement of the distance between the instrument and the robot can be determined. For example, an offset may be added to this distance based on known attributes of the instrument to approximate the location of a given portion of the instrument (e.g., tip or tool end) in the surgical environment.

[0190] When it is determined that the surgical instrument is misaligned with the target position or orientation, instructions may be generated to move or reorient the robot to align the instrument with the target position or orientation. For example, from a CT scan or from user input, the instrument may be aligned with a target path or location in the surgical environment to perform the procedure. If the patient moves during the surgical procedure, instructions may be generated to move the instrument in sync with the detected patient movement or the detected misalignment, for example, using one or more APIs of the robot. In one example, if a patient movement of 2 centimeters to the left of the target path is detected, the robot control system may adjust the position of the robot the same 2 centimeters to the left according to the target path. In some implementations, adjustments to the position or orientation of the instrument may be made while navigating the robot along a predetermined trajectory (e.g., a path into the patient's skull to reach a target location on the patient's anatomy). For example, the robot may be navigated left or right according to the patient's movement, or may be navigated to realign the instrument to the target position or orientation, while simultaneously navigating the instrument down along the predetermined trajectory to the target location on the patient's anatomy. This offers an improvement over other robotic implementations for non-invasive procedures that do not track and compensate for patient movement, as any patient movement during the procedure can result in patient harm. By moving the instrument according to the patient's movements, pre-defined target paths can be followed more accurately, reducing unintended collisions or interference with other parts of the patient.

[0191] Adjustments to the position or orientation of the robot or instrument can be performed repeatedly, or periodically and multiple times per second, to compensate for sudden patient movement or other misalignment of the surgical instrument. As described herein, the patient movement and instrument position within the surgical environment can be calculated periodically (e.g., according to the capture rate of the image capture device, etc.). Each time a new frame is captured by the image capture device, it can be determined whether the change in the patient position meets a threshold (e.g., a predetermined amount of movement relative to the previous frame, the patient position at the start of the procedure, or the instrument position, etc.). The position or orientation of the robot can be adjusted such that the instrument is aligned with a predetermined path (e.g., a predetermined trajectory) relative to the detected change in the patient position. The change in the trajectory or path position can be determined based on a change in the position or orientation of one or more target indicators in a 3D image (e.g., a CT scan or an MRI scan, etc.) that is registered to the patient in real time. In some implementations, the 3D image can be modified to show a path or trajectory to a selected target. Similarly, in implementations where there are multiple targets or paths (or path segments), the surgeon may select, via user input (e.g., button selection, touch screen selection, etc.), one or more paths along which the robot should navigate the instrument. The robot may be navigated along the selected path while compensating for patient movement or misalignment of the surgical instrument in real time, as described herein. After aligning the instrument to the target position and orientation, in step 1110, the position and orientation of the instrument may be tracked continuously. In some implementations, step 1120 may be performed in parallel with step 1125 (e.g., allowing for simultaneous realignment of the surgical instrument and application of the instrument output to the target position). In such implementations, method 1100 may include performing step 1130 during the performance of steps 1110-1125.

[0192] In step 1125, the surgical instrument can be controlled to apply a treatment, such as delivering a TMS or FUS therapy to the location of interest. The surgical instrument can be controlled to apply a treatment in response to detecting that the surgical instrument is aligned with the location of interest, and the treatment can be adjusted, paused, or terminated in response to detecting that the surgical instrument is not aligned with the location of interest. An API of the robot or instrument can be used to generate instructions to activate the instrument to apply the desired treatment. In some implementations, manual input can be provided (e.g., by a surgeon, etc.) to activate the instrument to apply the treatment. In implementations where the treatment is applied automatically (e.g., in response to detecting that the instrument is aligned with a target location and orientation), instructions for the treatment can be accessed or retrieved to determine the duration and strength of the signal to be applied. The treatment can be a non-invasive treatment. In some implementations, the duration, strength, and type of instrument output can be pre-selected by the surgeon or from a database of treatments. In some implementations, some target locations can be selected, each with a corresponding duration, strength, and type of instrument output. Various steps of the method 1100 may be applied to each of a number of target locations to achieve a desired output to complete the non-invasive procedure.

[0193] In step 1130, the performance of the treatment is evaluated. For example, various parameters of the treatment, such as duration, instantaneous, average, and / or total energy or power (e.g., of the delivered beam or signal), as well as subject responses, such as physiological or biological responses (e.g., heart rate, respiration rate, temperature, skin conductance, brainwave activity, or various other parameters detected by various sensors), may be evaluated by comparing them to individual thresholds. Such responses may be captured from sensors coupled to the patient during the treatment. In some implementations, the delivery of the treatment may be adjusted in response to the evaluation, such as to increase or decrease the power, energy, frequency, or other parameters of the magnetic field or ultrasound signal being used to perform the treatment (in addition to adjusting the orientation of the surgical instrument). The respective thresholds may be provided by the surgeon, for example, prior to the treatment. Instructions for adjusting the parameters of the treatment may be generated using techniques similar to those described herein. If the surgical treatment threshold(s) are met, the method 1100 may proceed to step 1135. If the surgical procedure threshold(s) are not met, method 1100 may return to step 1110, which may be performed in parallel with step 1125 to reposition or reorient the surgical instrument while the procedure is being applied to the target location on the patient.

[0194] In step 1135, in response to an evaluation of performance meeting termination conditions (e.g., sufficient duration and / or total energy delivery), the surgical instrument may abort the procedure. In some implementations, terminating the procedure may include generating instructions to modify the position or orientation of the robot to a default position or orientation that causes the instrument to separate from the patient's anatomy. These instructions may be generated using techniques similar to those described herein. Generating instructions may include generating instructions to guide the instrument along a predetermined trajectory to safely separate the instrument from the patient's anatomy. In the case of an invasive procedure, this may include following a trajectory (e.g., that is predetermined or may be determined based in part on the robot's current position) that safely removes the instrument from within the patient's anatomy. Similar techniques may be performed to remove the instrument from the surface of the patient's anatomy in a non-invasive procedure. In some implementations, an alert may be generated indicating that the procedure is complete, and the surgeon may be prompted to manually control the robot to remove the instrument from the patient's anatomy.

[0195] (F. Computing Environment) 12A and 12B show block diagrams of a computing device 1200. As shown in FIG. 12A and 12B, each computing device 1200 includes a central processing unit 1221 and a main memory unit 1222. As shown in FIG. 12A, the computing device 1200 may include a storage device 1228, an installation device 1216, a network interface 1218, an I / O controller 1223, display devices 1224a-1224n, a keyboard 1226, and a pointing device 1227, such as a mouse. The storage device 1228 may include, but is not limited to, an operating system, software, and software of the image processing system 100, the robotic system 300, the robotic control system 405, the robotic control system 605, the robotic control system 805, or the image processing system 1000. As shown in FIG. 12B, each computing device 1200 may also include additional optional elements, such as a memory port 1203, a bridge 1270, one or more input / output devices 1230a-1230n (generally referred to using the reference numeral 1230), and a cache memory 1240 in communication with the central processing unit 1221.

[0196] The central processing unit 1221 is any logic circuitry that responds to and processes instructions fetched from the main memory unit 1222 . In many implementations, the central processing unit 1221 is provided by a microprocessor unit, such as those manufactured by Intel Corporation (Mountain View, California), those manufactured by Motorola Corporation (Schaumburg, Illinois), ARM processors (e.g., manufactured by ARM Holdings and manufactured by ST, TI, ATMEL, etc.) and TEGRA system-on-chips (SoCs) manufactured by Nvidia (Santa Clara, California), POWER7 processors manufactured by International Business Machines (White Plains, New York), or those manufactured by Advanced Micro Devices (Sunnyvale, California), or field programmable gate arrays ("FPGAs") manufactured by Altera (San Jose, California), Intel Corporation (Xilinx in San Jose, California), or MicroSemi (Aliso Viejo, California), etc. Computing device 1200 may be based on any of these processors, or any other processor capable of operating as described herein. The central processing unit 1221 can utilize instruction level parallelism, thread level parallelism, different levels of caches, and multi-core processors. Multi-core processors can include two or more processing units on a single computing component. Examples of multi-core processors include AMD PHENOM IIX2, INTEL CORE i5, INTEL CORE i7, and INTEL CORE i9.

[0197] The main memory unit 1222 may include one or more memory chips capable of storing data and allowing any memory location to be directly accessed by the microprocessor 1221. The main memory unit 1222 may be volatile and faster than the storage 1228 memory. The main memory unit 1222 may be a dynamic random access memory (DRAM) or any variation, including static random access memory (SRAM), burst SRAM or SynchBurst SRAM (BSRAM), fast page mode DRAM (FPM DRAM), enhanced DRAM (EDRAM), extended data output RAM (EDO RAM), extended data output DRAM (EDO DRAM), burst extended data output DRAM (BEDO DRAM), single data rate synchronous DRAM (SDR SDRAM), double data rate SDRAM (DDR SDRAM), direct Rambus DRAM (DRDRAM), or extreme data rate DRAM (XDR DRAM). In some implementations, the main memory 1222 or storage 1228 can be non-volatile, e.g., non-volatile read access memory (NVRAM), flash memory non-volatile static RAM (nvSRAM), ferroelectric RAM (FeRAM), magnetoresistive RAM (MRAM), phase change memory (PRAM), conductive bridging RAM (CBRAM), silicon-oxide-nitride-oxide-silicon (SONOS), resistive RAM (RRAM), racetrack, nanoRAM (NRAM), or millipede memory. The main memory 1222 can be based on any of the memory chips mentioned above or any other available memory chip capable of operating as described herein. In the implementation shown in FIG. 12A, the processor 1221 communicates with the main memory 1222 via a system bus connection 1250 (described in more detail below). FIG. 12B illustrates one implementation of the computing device 1200 in which the processor communicates directly with the main memory 1222 via a memory port 1203. For example, in FIG. 12B, main memory 1222 may be DRDRAM.

[0198] FIG. 12B illustrates one implementation in which the main processor 1221 communicates directly with the cache memory 1240 through a secondary bus, sometimes referred to as a backside bus. In other implementations, the main processor 1221 communicates with the cache memory 1240 using a system bus 1250. The cache memory 1240 typically has a faster response time than the main memory 1222 and is typically provided by SRAM, BSRAM, or EDRAM. In the implementation illustrated in FIG. 12B, the processor 1221 communicates with various I / O devices 1230 through a local system bus 1250. Various buses can be used to connect the central processing unit 1221 to any of the I / O devices 1230, including a PCI bus, a PCI-X bus, or a PCI-Express bus, or a NuBus. In an implementation where the I / O device is a video display 1224, the processor 1221 can communicate with the display 1224 or an I / O controller 1223 for the display 1224 using an Advanced Graphics Port (AGP). Figure 12B shows an implementation of the computer 1200 where the main processor 1221 communicates directly with the I / O device 1230b or other processors 1221 via HYPERTRANSPORT, RAPIDIO, or INFINIBAND communication technologies. Figure 12B also shows an implementation where local bus and direct communication are mixed, with the processor 1221 communicating with the I / O device 1230a using a local interconnect bus while communicating directly with the I / O device 1230b.

[0199] A wide variety of I / O devices 1230a-1230n may be present in computing device 1200. Input devices may include keyboards, mice, track pads, track balls, touch pads, touch mice, multi-touch touch pads and touch mice, microphones (analog or MEMS), multi-array microphones, drawing tablets, cameras, single lens reflex cameras (SLR), digital SLR (DSLR), CMOS sensors, CCDs, accelerometers, inertial measurement units, infrared optical sensors, pressure sensors, magnetometer sensors, angular rate sensors, depth sensors, proximity sensors, ambient light sensors, gyroscope sensors, or other sensors. Output devices may include video displays, graphical displays, speakers, headphones, inkjet printers, laser printers, and 3D printers.

[0200] The devices 1230a-1230n may include a combination of multiple input or output devices, including, for example, Microsoft KINECT, Nintendo Wiimote for the WII, Nintendo WII U GAMEPAD, or Apple IPHONE. Some devices 1230a-1230n may enable gesture recognition input through a combination of some of the inputs and outputs. Some devices 1230a-1230n provide facial recognition that may be used as input for different purposes, including authentication and other commands. Some devices 1230a-1230n provide voice recognition and input, including, for example, Microsoft KINECT, SIRI for the IPHONE by Apple, Google Now, or Google Voice Search.

[0201] Additional devices 1230a-1230n have both input and output capabilities, including, for example, haptic feedback devices, touch screen displays, or multi-touch displays. Touch screens, multi-touch displays, touch pads, touch mice, or other touch sensing devices can use different technologies for sensing touch, including, for example, capacitive, surface capacitive, projected capacitive touch (PCT), in-cell capacitive, resistive, infrared, waveguide, distributed signal touch (DST), in-cell optical, surface acoustic wave (SAW), bending wave touch (BWT), or force-based sensing technologies. Some multi-touch devices can allow for two or more points of contact with a surface, allowing for advanced functionality including, for example, pinching, spreading, rotating, scrolling, or other gestures. Some touch screen devices, including, for example, Microsoft PIXELSENSE or Multi-Touch Collaboration Wall, can have larger surfaces, such as on a tabletop or on a wall, and can also interact with other electronic devices. Some of the I / O devices 1230a-1230n, display devices 1224a-1224n, or groups of devices may be augmented reality devices. The I / O devices may be controlled by an I / O controller 1223, as shown in FIG. 12A. The I / O controller 1223 may control one or more I / O devices, such as a keyboard 126 and a pointing device 1227 (e.g., a mouse or optical pen). Additionally, the I / O devices may also provide a storage medium and / or an installation medium 1216 for the computing device 1200. In other implementations, the computing device 1200 may provide a USB connection (not shown) for accepting a handheld USB storage device. In further implementations, the I / O device 1230 may be a bridge 1270 between the system bus 1250 and an external communication bus, such as a USB bus, a SCSI bus, a FireWire bus, an Ethernet bus, a Gigabit Ethernet bus, a Fibre Channel bus, or a Thunderbolt bus.

[0202] In some implementations, the display devices 1224a-1224n can be connected to the I / O controller 1223. The display devices 1224a-1224n can include, for example, a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), a blue-phase LCD, an electronic paper (e-ink) display, a flexible display, a light emitting diode display (LED), a digital light processing (DLP) display, a liquid crystal on silicon (LCOS) display, an organic light emitting diode (OLED) display, an active matrix organic light emitting diode (AMOLED) display, a liquid crystal laser display, a time multiplexed light shutter (TMOS) display, or a 3D display. Examples of 3D displays can use, for example, stereoscopic, polarizing filters, active shutters, or autostereoscopic. The display devices 1224a-1224n can be head mounted displays (HMDs). In some implementations, the display devices 1224a-1224n or corresponding I / O controllers 1223 may be controlled through or have hardware support for the OPENGL or DIRECTX API or other graphics libraries.

[0203] In some implementations, the computing device 1200 may include or be connected to multiple display devices 1224a-1224n, each of which may be of the same or different type and / or form. Thus, any of the I / O devices 1230a-1230n and / or the I / O controller 1223 may include any type and / or form of suitable hardware, software, or combination of hardware and software to support, enable, or provide for the connection and use of the multiple display devices 1224a-1224n by the computing device 1200. For example, the computing device 1200 may include any type and / or form of video adapter, video card, drivers, and / or libraries to interface, communicate, connect, or otherwise use the display devices 1224a-1224n. In one implementation, the video adapter may include multiple connectors for interfacing to the multiple display devices 1224a-1224n. In other implementations, computing device 1200 may include multiple video adapters, each connected to one or more of display devices 1224a-1224n. In some implementations, any portion of the operating system of computing device 1200 may be configured to use multiple displays 1224a-1224n. In other implementations, one or more of display devices 1224a-1224n may be provided by one or more other computing devices 1200a or 1200b connected to computing device 1200 via network 1240. In some implementations, software may be designed and constructed to use a display device of another computer as a second display device 1224a for computing device 1200. For example, in one implementation, an Apple iPad® may be connected to computing device 1200 and the display of device 1200 may be used as an additional display screen that may be used as an extended desktop.Those skilled in the art will recognize and understand the various ways and implementations in which a computing device 1200 may be configured to have multiple display devices 1224a-1224n.

[0204] Referring again to FIG. 12A, the computing device 1200 may include a storage device 1228 (e.g., one or more hard disk drives or a redundant array of independent disks) for storing an operating system or other related software, and for storing application software programs, such as any programs related to software for the image processing system 100, the robotic system 300, the robotic control system 405, the robotic control system 605, the robotic control system 805, or the image processing system 1000. Examples of the storage device 1228 include, for example, a hard disk drive (HDD), an optical drive, including a CD drive, a DVD drive, or a BLU-RAY drive, a solid state drive (SSD), a flash drive, or any other device suitable for storing data. Some storage devices 1228 may include multiple volatile and non-volatile memories, including, for example, a solid state hybrid drive that combines a hard disk with a solid state cache. Some storage devices 1228 may be non-volatile, alterable, or read-only. Some storage devices 1228 may be internal and connected to the computing device 1200 via a bus 1250. Some storage devices 1228 may be external and connected to the computing device 1200 through an I / O device 1230 providing an external bus. Some storage devices 1228 may be connected to the computing device 1200 through a network interface 1218 over a network, including, for example, Remote Disk for the MACBOOK AIR by Apple. Some client devices 1200 may not require a non-volatile storage device 1228 and may be thin clients or zero clients 202. Some storage devices 1228 may also be used as installation devices 1216 and may be suitable for installing software and programs.Further, the operating system and software may be executed from a bootable medium, such as a bootable CD, such as, for example, KNOPPIX, a bootable CD for GNU / Linux available as a GNU / Linux distribution from knoppix.net.

[0205] Computing device 1200 may also be capable of installing software or applications from an application distribution platform. Examples of application distribution platforms include the App Store for iOS offered by Apple, Inc., the Mac App Store offered by Apple, Inc., GOOGLE PLAY for Android OS offered by Google Inc., Chrome Webstore for CHROME OS offered by Google Inc., and the Amazon Appstore for Android OS and KINDLE FIRE offered by Amazon.com, Inc.

[0206] Additionally, computing device 1200 may include a LAN 1218 that interfaces to network 1240 via a variety of connections, including, but not limited to, a standard telephone-line network interface or WAN link (e.g., 802.11, T1, T3, Gigabit Ethernet, or Infiniband), a broadband connection (e.g., ISDN, Frame Relay, Gigabit Ethernet, Ethernet-over-SONET, ADSL, VDSL, BPON, GPON, or fiber optic including FiOS), a wireless connection, or any combination of any or all of the above. Connections may be established using a variety of communication protocols (e.g., TCP / IP, Ethernet, ARCNET, SONET, SDH, Fiber Distributed Data Interface (FDDI), IEEE 802.11a / b / g / n / ac CDMA, GSM, WiMax, and direct asynchronous connections). In one implementation, computing device 1200 communicates with other computing devices 1200 via any type and / or form of gateway or tunneling protocol, such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), or the Citrix Gateway Protocol manufactured by Citrix Systems, Inc. of Ft. Lauderdale, Fla. Network interface 1218 may comprise a built-in network adapter, a network interface card, a PCMCIA network card, an EXPRESSCARD network card, a card bus network adapter, a wireless network adapter, a USB network adapter, a modem, or any other device suitable for interfacing computing device 1200 to any type of network with which it can communicate and performing the operations described herein.

[0207] A computing device 1200 of the type shown in Figure 12A can operate under the control of an operating system that controls the scheduling of tasks and access to system resources. Computing device 1200 may be running any operating system, such as any version of the MICROSOFT WINDOWS operating system, different releases of Unix and Linux operating systems, any version of MAC OS for Macintosh computers, any embedded operating system, any real-time operating system, any open source operating system, any special-purpose operating system, any operating system for mobile computing devices, or any other operating system capable of running on computing device 1200 and performing the operations described herein. Exemplary operating systems include, but are not limited to, WINDOWS 2000', WINDOWS Server 2012, WINDOWS CE, WINDOWS Phone, WINDOWS XP, WINDOWS VISTA, WINDOWS 7, WINDOWS RT, WINDOWS 8 manufactured by Microsoft Corporation of Redmond, Washington, MAC OS and iOS manufactured by Apple Inc. of Cupertino, California, Linux, a freely available operating system such as the Linux Mint distribution ("distro") or Ubuntu distributed by Canonical Ltd. of London, England, or Unix or other Unix-derived operating systems, and Android designed by Google of Mountain View, California, etc. Some operating systems, including, for example, CHROME OS by Google, may be used on zero clients or thin clients, including, for example, CHROMEBOOKS.

[0208] Computer system 1200 can be any workstation, phone, desktop computer, laptop or notebook computer, netbook, ULTRABOOK, tablet, server, handheld computer, cell phone, smart phone or other portable telecommunication device, media playback device, gaming system, mobile computing device, or any other type and / or form of computing, telecommunication, or media device capable of communication. Computer system 1200 has sufficient processor power and memory capacity to perform the operations described herein. In some implementations, computing device 1200 may have different processors, operating systems, and input devices consistent with the device.

[0209] In some implementations, the status of one or more machines 1200 in the network can be monitored, for example, as part of network management. In one of these implementations, the status of the machines can include load information (e.g., the number of processes on the machine, the central processing unit, and memory utilization), port information (e.g., the number of available communication ports and port addresses), or identification of session state (e.g., the duration and type of the process, and whether the process is active or idle). In another of these implementations, this information can be identified by a number of metrics that can be applied at least in part towards determining load balancing, network traffic management, and network failure recovery, as well as any aspect of the operation of the present solution described herein. The above-mentioned aspects of the operating environment and components will become apparent in the context of the systems and methods disclosed herein.

[0210] Implementations of the subject matter and operations described herein may be implemented in digital electronic circuitry, including the structures disclosed herein and their structural equivalents, or in computer software executed on tangible media, firmware, or hardware, or in a combination of one or more of these. Implementations of the subject matter described herein may be implemented as one or more computer programs, e.g., one or more components of computer program instructions, encoded on a computer storage medium for execution by or to control the operation of a data processing device. The program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to an appropriate receiver device for execution by a data processing device. A computer storage medium may be or be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these. Furthermore, although a computer storage medium is not a propagating signal, a computer storage medium may include a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage media may also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices 1028).

[0211] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0212] The terms "data processing apparatus", "data processing system", "client device", "computing platform", "computing device", or "device" encompass all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or a plurality or combination of the foregoing. An apparatus may include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, an apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0213] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can, but need not, correspond to a file in a file system. A program can be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple cooperating files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0214] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may also be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0215] Processors suitable for executing a computer program include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory, or both. Elements of a computer include a processor for performing actions in accordance with the instructions, and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or be operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Furthermore, a computer can be incorporated in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus flash drive). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0216] To provide for interaction with a user, implementations of the subject matter described herein may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), plasma, or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and a pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user, e.g., feedback provided to the user may include any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0217] Implementations of the subject matter described herein may be implemented in a computing system that includes back-end components, e.g., a data server, or includes middleware components, e.g., an application server, or includes front-end components, e.g., a client computer having a graphical user interface (e.g., user interface 120 described with respect to FIGS. 1 and 2 or user interface 1020 described with respect to FIG. 10) or a web browser through which a user can interact with implementations of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0218] Although the present specification contains details of many specific implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular implementations of the systems and methods described herein. Certain features described herein in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a particular combination, and may even be initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0219] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations be performed, to achieve desirable results. In some cases, actions recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results.

[0220] In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, it should be understood that the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products.

[0221] Although some exemplary implementations and implementations have been described herein, it is clear that the above are presented by way of example, not limitation. In particular, many of the examples presented herein include specific combinations of method operations or system elements, but those operations and those elements may be combined in other ways to achieve the same purpose. Acts, elements, and features discussed only in connection with one implementation are not intended to be excluded from a similar role in other implementations or implementations.

[0222] The phraseology and terminology used herein are for purposes of description and should not be considered as limiting. The use herein of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized in that," and variations thereof, is meant to encompass the items recited thereafter, equivalents thereof, and additional items, as well as alternative implementations that consist of the items recited thereafter exclusively. In one implementation, the systems and methods described herein consist of one, any combination of two or more, or all of the described elements, acts, or components.

[0223] Any reference herein to system and method implementations or elements or acts in the singular can also encompass implementations that include a plurality of those elements, and any reference herein to any implementations or elements or acts in the plural can also encompass implementations that include only a single element. References in the singular or plural are not intended to limit the disclosed systems or methods, their components, acts, or elements to a singular or plural configuration. References to any act or element that is based on any information, act, or element can include implementations in which the act or element is based at least in part on any information, act, or element.

[0224] Any implementation disclosed herein may be combined with any other implementation, and references to "an implementation," "some implementations," "alternative implementations," "various implementations," "one implementation," etc. are not necessarily mutually exclusive and indicate that a particular feature, structure, or characteristic described with respect to an implementation may be included in at least one implementation. Such terms as used herein do not necessarily all refer to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0225] References to "or" may be construed as inclusive, such that any term described using "or" can refer to either one, more than one, and all of the described term.

[0226] Where reference signs follow a drawing, a description of the invention, or a technical feature in any claim, the reference signs are included for the sole purpose of enhancing comprehension of the drawing, the description of the invention, and the claims, and thus neither the reference signs nor their absence have any limiting effect on the scope of any claim element.

[0227] The systems and methods described herein may be embodied in other specific forms without departing from their characteristics. Although the examples provided may be useful for navigating a surgical robot according to patient movement and surgical environment conditions, the systems and methods described herein may be applied to other environments. The foregoing implementations are illustrative rather than limiting of the described systems and methods. Thus, the scope of the systems and methods described herein may be indicated by the appended claims rather than the above description, and changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein.

Claims

1. accessing, by one or more processors coupled to a memory, a 3D (three dimensional) point cloud corresponding to the surgical environment and the patient, the 3D point cloud having a reference coordinate system; determining, by the one or more processors, a position of a surgical robot within the reference coordinate system of the 3D point cloud, the position including calibrating, by the one or more processors, the surgical robot using a calibration technique; detecting, by the one or more processors, a change in position of the patient based on a corresponding change in position of one or more points in the 3D point cloud; generating, by the one or more processors, in response to detecting the change in the position of the patient, instructions to modify the position of the surgical robot based on the change in position of the one or more points; A method comprising:

2. 10. The method of claim 1, wherein the surgical robot further comprises a display positioned over a surgical site within the surgical environment, the method further comprising presenting, by the one or more processors, an image captured by a capture device mounted on the surgical robot.

3. 10. The method of claim 1, wherein the surgical robot comprises an attachment that receives a surgical tool, and determining the position of the surgical robot further comprises determining, by the one or more processors, the position of the surgical tool.

4. The method of claim 1 , further comprising navigating, by the one or more processors, the surgical robot along a predetermined path within the reference coordinate system.

5. The method of claim 1 , wherein determining the position of the surgical robot is based on infrared tracking technology.

6. 6. The method of claim 5, wherein the surgical robot comprises one or more markers, and wherein determining the position of the surgical robot based on the infrared tracking technology includes detecting a respective position of each of the one or more markers.

7. 2. The method of claim 1, wherein detecting the change in the position of the patient includes comparing, by the one or more processors, points of the 3D point cloud with second points of a second 3D point cloud captured after the 3D point cloud.

8. The method of claim 7 , wherein detecting the change in the position of the patient comprises determining that a distance between the point and the second point exceeds a predetermined threshold.

9. 1. A system comprising one or more processors coupled to a non-transitory memory, The one or more processors: accessing a 3D (three dimensional) point cloud corresponding to the surgical environment and the patient, the 3D point cloud having a reference coordinate system; determining a position of the surgical robot within the reference coordinate system of the 3D point cloud by performing operations including calibrating the surgical robot using a calibration technique; detecting a change in the position of the patient based on a corresponding change in the position of one or more points within the 3D point cloud; generating instructions to modify the position of the surgical robot based on the change in position of the one or more points in response to detecting the change in the position of the patient; A system configured to:

10. 10. The system of claim 9, wherein the surgical robot further comprises a display positioned over a surgical site within the surgical environment, and the one or more processors are further configured to present images captured by a capture device mounted on the surgical robot.

11. The system of claim 10 , wherein the surgical robot comprises an attachment that receives a surgical tool, and the one or more processors are further configured to determine a position of the surgical tool.

12. The system of claim 9 , wherein the one or more processors are further configured to navigate the surgical robot along a predetermined path within the reference coordinate system.

13. The system of claim 9 , wherein the one or more processors are further configured to determine the position of the surgical robot based on infrared tracking technology.

14. The system of claim 13 , wherein the surgical robot comprises one or more markers, and the one or more processors are further configured to detect a respective position of each of the one or more markers.

15. 10. The system of claim 9, wherein the one or more processors are further configured to detect the change in the position of the patient by performing operations including comparing points of the 3D point cloud with second points of a second 3D point cloud captured after the 3D point cloud.

16. 16. The system of claim 15, wherein the one or more processors are further configured to detect the change in the position of the patient by performing operations including determining that a distance between the point and the second point exceeds a predetermined threshold.

Images