Non-invasive patient tracker for surgical procedures

Abstract

A surgical system (100), a navigation system (118), and a method are provided. The method comprises obtaining, via a first imaging device (112a), a first image of an object in a surgical region; a second image of the first imaging device (112a) is obtained via a second imaging device (112b), wherein the second image comprises one or more second tracking markers (412) arranged in the first position. The method further comprises obtaining, via the first imaging device (112a), one or more additional first images of the object; obtaining one or more additional second images of the first imaging device (112a) via the second imaging device (112b); and determining, based on the one or more additional first images and the one or more additional second images, whether the object has moved and / or whether the first imaging device (112a) has moved relative to the object.

Description

Technical Field

[0001] This disclosure relates generally to surgical systems, and more specifically to robotic surgical devices. Background Technology

[0002] Surgical robots can assist surgeons or other healthcare providers in performing surgical procedures, or they can autonomously complete one or more surgical procedures. The layout of the operating room during surgical procedures is particularly important for supporting the successful use of surgical robots. Summary of the Invention

[0003] Examples of aspects of this disclosure include:

[0004] A method includes: acquiring a first image of an object in a surgical region via a first imaging device, wherein the first image includes one or more first tracking markers arranged in a first location relative to the object; acquiring a second image of the first imaging device via a second imaging device, wherein the second image includes one or more second tracking markers arranged in the first location; acquiring one or more additional first images of the object via the first imaging device, wherein the one or more additional first images include one or more first tracking markers; acquiring one or more additional second images of the first imaging device via the second imaging device, wherein the one or more additional second images include one or more second tracking markers; and determining, based on the one or more additional first images and the one or more additional second images, whether the object has been moved and / or whether the first imaging device has been moved relative to the object.

[0005] In some aspects, the object includes soft tissue, and the method further includes: determining, based on one or more additional first images, that the object has moved beyond a predetermined object motion threshold; and, in response to determining that the object has moved beyond the predetermined object motion threshold, reporting the soft tissue movement to at least one of a surgeon and a robot.

[0006] In some respects, soft tissue motion is reported to the robot, and the method also includes adjusting the robotic surgical plan based on soft tissue motion.

[0007] In some respects, the first imaging device is connected to the robot.

[0008] In some respects, the first imaging device is connected to at least one of the robot's robotic arm and end effector.

[0009] In some aspects, the method further includes: determining, based on one or more additional second images, that the first imaging device has moved more than a predetermined distance threshold relative to the object; correlating the motion of the first imaging device with the motion of the robot; and, in response to determining that the first imaging device has moved more than the predetermined distance threshold relative to the object, reporting the movement of the robot or the patient to the surgeon.

[0010] In some respects, one or more first tracking markers are set on a flexible material, and the flexible material is attached to the patient at or near the surgical area.

[0011] In some cases, at least one of nails, sutures, and glue is used to attach flexible materials to the patient.

[0012] In some respects, one or more of the first tracking markers are sized according to the resolution of the first imaging device and are invisible to the second imaging device.

[0013] In some aspects, one or more first tracking marks are printed on a flexible material.

[0014] In some respects, the flexible material includes a central region that is substantially devoid of one or more first tracking markers, and one or more first tracking markers are arranged radially around the central region.

[0015] In some respects, the central area includes the area through which an incision is made.

[0016] In some respects, flexible materials include transparent or translucent materials.

[0017] In some respects, one or more second tracking markers are larger than one or more first tracking markers, and one or more of the second tracking markers are attached to the first imaging device.

[0018] In some aspects, the method further includes: determining, based on one or more additional first images, that one or more first tracking markers have moved to a second position relative to the object; determining, based on one or more additional second images, that one or more second tracking markers have moved to a second position relative to the object, the second position indicating the motion of the first imaging device; and reporting the motion of the object and the motion of the first imaging device.

[0019] In some respects, at least one of the first image and the second image includes one or more of ultrasound images, magnetic resonance images, fluorescence images, infrared images, visible light images, radiation images, computed tomography images, nuclear medicine images and / or positron emission tomography images.

[0020] On the other hand, a system is included, comprising: a first imaging device including a first field of view capturing one or more first tracking markers arranged on or around a surgical region; a second imaging device including a second field of view capturing the first imaging device; and a computing device including: a processor; and a computer memory coupled to the processor and having data stored thereon, the data, when executed by the processor, enabling the processor to: receive first image data from the first imaging device, wherein the first image data indicates movement of one or more first tracking markers relative to the surgical region; receive second image data from the second imaging device, wherein the second image data indicates movement of the first imaging device relative to the surgical region; and determine, based on a combination of the first image data and the second image data, whether an object in the surgical region has been moved and / or whether the first imaging device has been moved relative to the object.

[0021] In some respects, one or more second tracking markers are attached to the first imaging device, wherein one or more first tracking markers are smaller than one or more second tracking markers, wherein the second field of view includes the surgical region, and wherein one or more first tracking markers are too small to be identified in the second image data.

[0022] In some respects, the first imaging device is connected to a robot that supports surgical procedures at the surgical site, and the first imaging device includes a 3D camera.

[0023] On the other hand, a surgical system is included, comprising: a robot configured to perform a surgical plan; a first imaging device attached to the robot and configured to capture images of one or more first tracking markers arranged on or around a surgical region, wherein the first imaging device includes one or more second tracking markers attached thereto; a second imaging device configured to capture images of the first imaging device and the one or more second tracking markers attached thereto; and a computing device comprising: a processor; and a computer memory coupled to the processor and having data stored thereon, which, when executed by the processor, enables the processor to: receive first image data from the first imaging device, wherein the first image data indicates movement of one or more first tracking markers relative to the surgical region; receive second image data from the second imaging device, wherein the second image data indicates movement of the first imaging device relative to the surgical region; determine, based on a combination of the first image data and the second image data, that an object in the surgical region has moved and / or the first imaging device has moved relative to the object; and update the surgical plan in response to determining that an object in the surgical region has moved and / or the first imaging device has moved relative to the object.

[0024] Any one aspect can be combined with any one or more other aspects.

[0025] Any one or more of the features disclosed in this article.

[0026] This article generally discloses one or more of the features.

[0027] Any one or more of the features generally disclosed in this article are combined with any one or more other features generally disclosed in this article.

[0028] Any aspect / feature / implementation combined with one or more other aspects / features / implementations.

[0029] Use any one or more of the aspects or features disclosed herein.

[0030] It should be understood that any feature described herein may be combined with any other feature described herein to claim protection, regardless of whether the feature comes from a specific implementation of the same description.

[0031] Details of one or more aspects of this disclosure are set forth in the following drawings and description. Other features, objects, and advantages of the technology described in this disclosure will be apparent from the description, drawings, and claims.

[0032] The foregoing is a simplified overview of this disclosure to provide an understanding of some aspects thereof. This summary is neither a broad nor an exhaustive overview of this disclosure and its various aspects, embodiments, and configurations. It is not intended to identify key or essential elements of this disclosure, nor to define its scope, but rather to present the concepts of this disclosure in a simplified form as an introduction to the more detailed description presented below. It should be understood that other aspects, embodiments, and configurations of this disclosure may utilize one or more of the features set forth above or described in detail below, individually or in combination.

[0033] Many additional features and advantages of this disclosure will become apparent to those skilled in the art upon consideration of the specific embodiments described below. Attached Figure Description

[0034] The accompanying drawings are incorporated in and form a part of this specification to illustrate several examples of this disclosure. These drawings, together with the description, explain the principles of this disclosure. The drawings illustrate only preferred and alternative examples of how to implement and use this disclosure, and these examples should not be construed as limiting this disclosure solely to the illustrated and described examples. Further features and advantages will become apparent from the following more detailed description of various aspects, specific implementations, and configurations of this disclosure, as illustrated by the accompanying drawings referenced below.

[0035] Figure 1 It is a block diagram of a system according to at least one specific embodiment of this disclosure;

[0036] Figure 2 A non-invasive patient tracker according to at least one specific embodiment of the present disclosure is shown;

[0037] Figure 3A A first configuration of a flexible material relative to a patient is shown according to at least one embodiment of the present disclosure;

[0038] Figure 3B A second configuration of a patient according to at least one embodiment of the present disclosure is shown, the patient having more than one flexible material patch applied thereto;

[0039] Figure 4 It is a plan view of the environment in which the surgical robot can operate according to at least one specific embodiment of the present disclosure;

[0040] Figure 5 This is an example of a first processing flow according to at least one specific implementation of this disclosure; and

[0041] Figure 6 This is an example of a second processing flow according to at least one specific implementation of this disclosure. Detailed Implementation

[0042] It should be understood that the various aspects disclosed herein can be combined with combinations different from those specifically presented in the specification and drawings. It should also be understood that, depending on the example or specific implementation, certain actions or events of any process or method described herein may be performed in a different order, and / or may be added, combined, or completely omitted (e.g., depending on different specific implementations of this disclosure, implementing the disclosed technology may not require all the described actions or events). Furthermore, although for clarity some aspects of this disclosure are described as being performed by a single module or unit, it should be understood that the technology of this disclosure can be performed by a combination of units or modules associated with, for example, computing devices and / or medical devices.

[0043] In one or more examples, the described methods, processes, and techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the function may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Alternatively or additionally, the function may be implemented using machine learning models, neural networks, artificial neural networks, or combinations thereof (instructions alone or in combination). The computer-readable medium may include a non-transitory computer-readable medium that corresponds to a tangible medium, such as a data storage medium (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer).

[0044] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple A11, A12, A12X, A12Z, or A13 Bionic processors; or any other general-purpose microprocessors), graphics processing units (e.g., Nvidia GeForce RTX 2000 series processors, Nvidia GeForce RTX 3000 series processors, AMD Radeon RX 5000 series processors, AMD Radeon RX 6000 series processors, or any other graphics processing units), application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the foregoing structures or any other physical structures suitable for implementing the described techniques. Furthermore, these techniques may be fully implemented in one or more circuit or logic elements.

[0045] Before explaining any specific implementation of this disclosure in detail, it should be understood that this disclosure is not limited in its application to the construction details and component arrangements set forth in the following description or illustrated in the accompanying drawings. This disclosure can have other specific implementations and can be practiced or implemented in various ways. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of “comprising,” “including,” or “having,” and variations thereof, is intended to cover items listed thereafter and their equivalents, as well as additional items. In addition, this disclosure may use examples to illustrate one or more aspects thereof. Unless otherwise expressly stated, the use or listing of one or more examples (which may be indicated by “for example,” “by means of an example,” “such as,” or similar language) is not intended to, and does not limit, the scope of this disclosure.

[0046] The terms proximal and distal are used in this disclosure in their usual medical sense, with the proximal being closer to the operator or user of the system and further away from the patient's body or surgical area of ​​concern, and the distal being closer to the patient's body or surgical area of ​​concern and further away from the operator or user of the system.

[0047] Some implementations of robotic surgical systems involve operating the robotic system near the operating table (e.g., operating table, operating table, etc.) and the patient. The term "robotic system" may also be referred to as "robotic surgical system" in this document.

[0048] In some implementations of robotic systems (e.g., where the robotic system or robotic arm is not mounted on an operating table or on a patient), movement of the robotic system relative to the operating table and / or patient may become necessary or desirable during surgical procedures. Improved robotic performance can be achieved when the robotic system and its components are tracked more accurately, particularly relative to the surgical area. Aspects of this disclosure support improvements in the localization and tracking of robotic systems relative to the operating table, patient, or surgical area.

[0049] In some implementations, navigation components (e.g., optical trackers or tracking markers) may be mounted at or near the surgical area, on a robot end effector, on a robot arm, on a camera, or a combination thereof. The navigation system may be configured to determine the position of the robot, the camera mounted on the robot, the robot end effector, and the robot arm relative to the surgical area by tracking the position of the navigation components. The camera mounted on the robot may also track the movement of tracking markers placed at or near the surgical area, and the camera may provide images of the surgical area to the navigation system. The navigation system may use its view of the tracking markers and objects in the operating room, as well as information received from the camera mounted on the robot, to control robotic surgical procedures. Specifically, based on the determined relative positions of the robot, camera, robot end effector, robot arm, and surgical area, the navigation system may also determine whether the surgical plan should be updated, whether the robot should be manipulated in a particular manner, etc.

[0050] The embodiments disclosed herein provide technical solutions to one or more of the following problems: (1) tracking soft tissue changes during surgery, (2) tracking soft tissue changes after surgery, (3) reporting soft tissue movement in real time or over a clinically meaningful amount of time from soft tissue movement, and / or (4) controlling robot navigation in response to soft tissue movement.

[0051] First go to Figure 1 This diagram illustrates a block diagram of a system 100 according to at least one embodiment of the present disclosure. System 100 can be used to track soft tissue movement, assist robot navigation in response to soft tissue movement, report soft tissue movement, and / or implement one or more other aspects of one or more methods disclosed herein. System 100 illustratively includes a computing device 102, one or more imaging devices 112, a robot 114, a navigation system 118, a database 130, and / or a cloud or other network 128. Systems according to other embodiments of the present disclosure may include more or fewer components than system 100. For example, system 100 may not include imaging device 112, robot 114, navigation system 118, one or more components of computing device 102, database 130, and / or cloud 128.

[0052] In some embodiments, computing device 102 includes a processor 104, a memory 106, a communication interface 108, and a user interface 110. Other embodiments of the computing device according to this disclosure may include more or fewer components than computing device 102.

[0053] The processor 104 of computing device 102 can be any processor described herein, any collection of processors, or any similar processor. Processor 104 can be configured to execute instructions or neural networks (e.g., data) stored in memory 106, which can cause processor 104 to perform one or more computational steps using or based on data received from imaging device 112, robot 114, navigation system 118, database 130, and / or cloud 128.

[0054] Memory 106 may be or include RAM, DRAM, SDRAM, other solid-state memory, any memory described herein, or any other tangible non-transitory memory used to store computer-readable data and / or instructions. Memory 106 may store information or data useful for performing any steps of, for example, the methods 400, 500, 600, 700, and 800 described herein, or any other method. Memory 106 may store, for example, one or more image processing algorithms or neural networks 120, object recognition process 122, tissue motion tracking process 124, and reporting and feedback process 126. In some embodiments, such instructions, data, or algorithms may be organized into one or more applications, modules, packages, layers, or engines. Algorithms, data, and / or instructions may cause processor 104 to manipulate data stored in memory 106 and / or received from or via imaging device 112, robot 114, database 130, and / or cloud 128.

[0055] Illustratively, memory 106 is shown to include image processing instructions 120, object recognition instructions 122, tissue motion tracking instructions 124, and reporting / feedback instructions 126. Again, without departing from the scope of this disclosure, one or more of the instructions may be implemented as a neural network, an artificial neural network, or a machine learning model.

[0056] When executed by processor 104, image processing 120 enables computing device 102 to collaborate with imaging device 112, robot 114, and / or navigation system 118 to acquire and use patient images. In some embodiments, image processing 120 may be configured to receive patient images (e.g., preoperative patient images, intraoperative patient images, and / or postoperative patient images), receive object images, receive environmental images (e.g., images of the operating room), and prepare images for processing by other components of computing device 102. Illustratively, image processing 120 may be configured to receive images and format such images for use by object recognition 122 and / or tissue motion tracking 124. In some embodiments, image processing 120 may be configured to transform images or image data into different available formats by: converting images from one digital format to another, determining object locations by performing pixel analysis, identifying reference locations in the image, compressing the image, decompressing the image, overlaying object models on the image, and / or any other tasks associated with preparing images for use by object recognition 122 and / or tissue motion tracking 124.

[0057] When executed by processor 104, object recognition 122 can enable computing device 102 to recognize one or more objects in an image, recognize the pose (e.g., position and / or orientation) of objects in an image and in space, recognize the pose of an object relative to another object, recognize the pose of an object by analyzing the position of one or more references, recognize patient anatomy from pixel and / or gradient analysis of an image, recognize the pose of a non-patient object (e.g., robotic arm 116) relative to a patient object (e.g., organ, nerve, muscle, bone, etc.), label objects within an image, and any other combination of tasks associated with recognizing objects, locating objects, and / or distinguishing one object from another.

[0058] When executed by processor 104, tissue motion tracking 124 enables computing device 102 to track the motion of one or more objects within the field of view of imaging device 112. In some embodiments, tissue motion tracking 124 may be configured to identify one or more objects of interest and track their motion during and / or after surgery. Tissue motion tracking 124 may be configured to track the absolute motion of one or more objects (e.g., by identifying the movement of the object in space or relative to a fixed coordinate system) and / or track the motion of one object relative to another object. Tissue motion tracking 124 may be configured to locate and track objects by identifying objects in an image via pixel analysis and then monitoring the movement of the objects based on analysis of subsequent images. Tissue motion tracking 124 may be configured to track the position and / or motion of tissue (e.g., soft tissue) by identifying and tracking the position of references in one or more images. As will be described herein, if a positional relationship is established between an object and one or more references, tissue motion tracking 124 may be configured to correlate the motion of one or more references with the motion of one or more objects, wherein the objects may include soft tissue objects such as organs, nerves, muscles, tumor growths, skin, etc.

[0059] When executed by processor 104, reporting and feedback 126 enables computing device 102 to provide information to the user of computing device 102 relating to object identification and / or object movement. Alternatively or additionally, reporting and feedback 126 may also be configured to provide feedback to navigation system 118 to assist navigation of robot 114. As an example, reporting and feedback 126 may provide information to navigation system 118 and / or surgeon relating to object movement, soft tissue movement, baseline movement, or combinations thereof. If an object or multiple objects are determined to have moved (absolutely or relatively) beyond a predetermined movement threshold, reporting and feedback 126 may alternatively or additionally provide one or more alerts. In the event that the movement of an object or multiple objects exceeds a predetermined movement threshold (e.g., a predetermined distance) or that two objects move too close to each other, reporting and feedback 126 may issue an alert or cause navigation system 118 to update the navigation plan of robot 114. An updated navigation plan prepared in response to the detection of soft tissue movement may allow robot 114 to avoid cutting, puncturing, or otherwise damaging objects, including soft tissue, during surgical procedures.

[0060] The computing device 102 may also include a communication interface 108. The communication interface 108 can be used to receive image data or other information from external sources (such as imaging device 112, robot 114, navigation system 118, database 130, cloud 128, and / or any other system or component not part of system 100), and / or to send instructions, images, or other information to external systems or devices (e.g., another computing device 102, imaging device 112, robot 114, navigation system 118, database 130, cloud 128, and / or any other system or component not part of system 100). The communication interface 108 may include one or more wired interfaces (e.g., USB port, Ethernet port, FireWire port) and / or one or more wireless transceivers or interfaces (configured to transmit and / or receive information, for example, via one or more wireless communication protocols such as 802.11a / b / g / n, Bluetooth, NFC, ZigBee, etc.). In some implementations, the communication interface 108 can be used to enable the device 102 to communicate with one or more other processors 104 or computing devices 102, whether to reduce the time required to complete computationally intensive tasks or for any other reason.

[0061] The computing device 102 may also include one or more user interfaces 110. User interface 110 may be or include a keyboard, mouse, trackball, monitor, television, screen, touchscreen, and / or any other device for receiving information from a user and / or providing information to a user. User interface 110 may be used, for example, to receive user selections or other user input regarding any step of any method described herein. Nevertheless, any required input for any step of any method described herein may be automatically generated by system 100 (e.g., by processor 104 or another component of system 100) or received by system 100 from a source external to system 100. In some embodiments, user interface 110 may be used to allow a surgeon or other user to modify instructions to be executed by processor 104, and / or modify or adjust settings displayed on user interface 110 or corresponding to other information on user interface 110, according to one or more embodiments of this disclosure.

[0062] Although user interface 110 is shown as part of computing device 102, in some embodiments, computing device 102 may utilize user interface 110, which may be housed separately from one or more other components of computing device 102. In some embodiments, user interface 110 may be located proximal to one or more other components of computing device 102, while in other embodiments, user interface 110 may be located remotely from one or more other components of computing device 102. Illustratively, user interface 110 may include user input devices, user output devices, and combined user input / user output devices (e.g., touch-sensitive display devices).

[0063] Imaging device 112 can be used to image anatomical features (e.g., bones, veins, soft tissues, etc.) and / or other aspects of a patient's anatomy to produce image data (e.g., image data depicting or corresponding to bones, veins, soft tissues, etc.). As used herein, "image data" refers to data generated or captured by imaging device 112, including data in machine-readable, graphical / visual, and any other form. In various examples, image data may include data corresponding to a patient's anatomical features or a portion thereof. The image data may be or include preoperative images, intraoperative images, postoperative images, or images taken independently of any surgical procedure. In some embodiments, first imaging device 112 may be used to acquire first image data (e.g., a first image) at a first time, and second imaging device 112 may be used to acquire second image data (e.g., a second image) at a second time after the first time. Imaging device 112 may be capable of capturing 2D or 3D images to produce image data. Imaging device 112 may be or include, for example, an ultrasound scanner (which may include, for example, physically separate transducers and receivers, or a single ultrasound transceiver), an O-arm, C-arm, G-arm, or any other device using X-ray-based imaging (e.g., a fluorescence microscope, CT scanner, or other X-ray machine), a magnetic resonance imaging (MRI) scanner, an ultrasound imaging device, an optical coherence tomography (OCT) scanner, an endoscope, a microscope, an optical camera, a thermal imaging camera (e.g., an infrared camera), a radar or lidar system (which may include, for example, a transmitter, a receiver, a processor, and one or more antennas), or any other imaging device 112 adapted to obtain images of a patient's anatomical features. Imaging device 112 may be entirely contained within a single housing, or may include transmitters / transmitters and receivers / detectors located in separate housings or otherwise physically separated.

[0064] In some embodiments, imaging device 112 may include more than one imaging device 112. For example, a first imaging device may provide first image data and / or a first image, and a second imaging device may provide second image data and / or a second image. In yet other embodiments, the same imaging device may be used to provide both the first image data and the second image data and / or any other image data described herein. Imaging device 112 may be used to generate an image data stream. For example, imaging device 112 may be configured to use an open shutter operation or to use shutter operations that alternate continuously between open and closed to capture a series of images. For the purposes of this disclosure, unless otherwise specified, if the image data represents two or more frames per second, the image data may be considered continuous and / or provided as an image data stream. As will be discussed in further detail herein, imaging device 112 may be mounted to robot 114 or robot arm 116 and may be configured to acquire images of tracking markers, which are also imaged by other imaging devices 112 not mounted to robot 114. In some implementations, images obtained from imaging equipment 112 mounted to robot 114 can be used to supplement image data obtained from other imaging equipment 112 at navigation system 118. The movement of imaging equipment 112 mounted to robot 114 can also be tracked to determine the movement and pose of robot 114 and imaging equipment 112 relative to the surgical area.

[0065] Robot 114 can be any surgical robot or surgical robot system. Robot 114 can be, or includes, for example, Mazor X. ™ Stealth robot guidance system. Robot 114 can be configured to position imaging device 112 in one or more precise locations and orientations, and / or return imaging device 112 to the same location and orientation at a later time point. Robot 114 may additionally or alternatively be configured to manipulate surgical tools (whether or not based on guidance from navigation system 118) to perform or assist surgical tasks. In some embodiments, robot 114 may be configured to hold and / or manipulate anatomical elements during or in conjunction with surgical procedures. Robot 114 may include one or more robotic arms 116. In some embodiments, robotic arms 116 may include a first robotic arm and a second robotic arm, but robot 114 may include more than two robotic arms. In some embodiments, one or more robotic arms 116 may be used to hold and / or manipulate imaging device 112. In embodiments where imaging device 112 includes two or more physically separate components (e.g., transmitters and receivers), one robotic arm 116 may hold one such component, and another robotic arm 116 may hold another such component. Each robotic arm 116 can be positioned independently of other robotic arms. The robotic arms can be controlled in a single shared coordinate space or in a separate coordinate space.

[0066] The robot 114, together with the robotic arm 116, may have, for example, one, two, three, four, five, six, seven, or more degrees of freedom. Furthermore, the robotic arm 116 can be or is capable of being positioned in any pose, plane, and / or focal position. This pose includes position and orientation. Therefore, an imaging device 112, surgical instrument, or other object held by the robot 114 (or more specifically, by the robotic arm 116) can be precisely positioned in one or more desired and specific locations and orientations.

[0067] The robotic arm 116 may include one or more sensors that enable the processor 104 (or the processor of the robot 114) to determine the precise pose of the robotic arm (and any object or element held or fixed to the robotic arm) in space.

[0068] In some embodiments, reference markers (i.e., navigation markers) may be placed on robot 114 (including, for example, on robotic arm 116), imaging device 112, or any other object in the surgical space. The reference markers may be tracked by navigation system 118, and the results of the tracking may be used by the operator of robot 114 and / or system 100 or any component thereof. In some embodiments, navigation system 118 may be used to track other components of the system (e.g., imaging device 112), and the system may operate without the use of robot 114 (e.g., a surgeon manually manipulating imaging device 112 and / or one or more surgical instruments, for example, based on information and / or instructions generated by navigation system 118).

[0069] During operation, navigation system 118 can provide navigation for the surgeon and / or surgical robot. Navigation system 118 can be any navigation system currently known or developed in the future, including, for example, Medtronic StealthStation. ™The S8 surgical navigation system or any successor system thereof. Navigation system 118 may include one or more cameras or other sensors for tracking one or more reference markers, navigation trackers, or other objects within the operating room or other room where part or all of system 100 is located. The one or more cameras may be optical cameras, infrared cameras, or other cameras. In some embodiments, the navigation system may include one or more electromagnetic sensors. In various embodiments, navigation system 118 may be used to track the position and orientation (e.g., pose) of imaging device 112, robot 114 and / or robotic arm 116 and / or one or more surgical tools (or more specifically, for tracking the pose of navigation trackers directly or indirectly attached in a fixed relationship to one or more of the foregoing). Navigation system 118 may include a display for displaying one or more images from an external source (e.g., computing device 102, imaging device 112, or other sources), or for displaying images and / or video streams from one or more cameras or other sensors of navigation system 118. In some embodiments, system 100 may operate without using navigation system 118. The navigation system 118 may be configured to provide guidance to the surgeon or other users of the system 100 or its components, to the robot 114, or to any other element of the system 100 regarding, for example, the pose of one or more anatomical elements, whether the tool is in the appropriate trajectory, and / or how to move the tool into the appropriate trajectory to perform the surgical task in accordance with the preoperative or other surgical plan.

[0070] System 100 or a similar system may be used, for example, to perform one or more aspects of any of the methods 500 and 600 described herein. System 100 or a similar system may also be used for other purposes. Any of the methods depicted and described herein may be implemented, for example, by at least one processor or otherwise performed. The at least one processor may be the same as or similar to processor 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as robot 114) or a navigation system (such as navigation system 118). Methods 500 and / or 600 may also be performed using processors other than any of the processors described herein. At least one processor may perform a method (e.g., method 500 or 600) by executing instructions stored in a memory (such as memory 106).

[0071] Now refer to Figure 2 , Figure 3A and Figure 3BAdditional details regarding the reference and other navigation markers used for tracking soft tissue movement will be described according to at least some embodiments of this disclosure. One or more tracking markers 202 may be provided on a flexible material 204, and the flexible material 204 may be configured to attach to an object or patient 300 (e.g., on the skin, organs, or other soft tissue of the patient 300) to assist in tracking soft tissue movement during and / or after surgery. In some embodiments, the flexible material 204 may include a marker array 200 disposed thereon, wherein each tracking marker 202 in the marker array 200 is distributed on the flexible material 204 in a known pattern.

[0072] like Figure 2 As shown, the tracking marker 202 may be radially disposed relative to the central region 208 of the array 200. Specifically, the central region 208 may correspond to the portion of the array 200 in which an incision is formed. The central region 208 may or may not have the tracking marker 202 disposed thereon. In some embodiments, the tracking marker 202 is positioned around the central region 208 to define a pattern that achieves movement relative to the central region 208. Because the central region 208 may correspond to the location where an incision is made during surgical procedures, the imaging device 112 can track the movement of soft tissue around the tracking marker 202 to determine the movement of the surgical area (e.g., the area within the central region 208). In some embodiments, the tracking marker 202 is disposed on a strip surrounding the central region 208.

[0073] In some implementations, the tracking marker 202 can be integrated into or attached to the flexible material 204, such that the movement of the flexible material 204 is translated into the movement of the tracking marker 202. More specifically, in one example, the tracking marker 202 can be printed in the flexible material 204 such that the outline of the array 200 is substantially constant across the entire array 200. In other words, the tracking marker 202 can be integrated into the flexible material as printed dots, squares, circles, etc., meaning that the movement of the flexible material 204 is immediately translated into the movement of the tracking marker 202.

[0074] Illustratively but not limitingly, the flexible material 204 can be configured for attachment to the patient 300 or an object of interest (e.g., soft tissue elements belonging to the patient 300, such as organs, nerves, muscles, tumor growths, skin, bones, etc.). Figure 3BAs shown, multiple flexible material patches 204 can be attached to patient 300 or one or more anatomical elements of patient 300. When the flexible material 204 is attached to patient 300 or an object of patient 300, the object recognition 122 and / or tissue motion tracking 124 of computing device 102 can be configured to identify and track the movement of soft tissue elements belonging to patient 300 during and / or after surgery. Specifically, but not limitingly, because the flexible material 204 may include one or more tracking markers 202, the position and movement of the tracking markers 202 can be correlated with the movement of the flexible material 204, which is correlated with the movement of the object to which the flexible material 204 is attached.

[0075] Flexible material 204 can be attached or connected to patient 300 or objects of patient 300 in a variety of ways. In some embodiments, attachment mechanisms (e.g., connectors, staples, sutures, and adhesives) can be used to attach flexible material 204 to patient 300 or anatomical elements of patient 300. In some embodiments, flexible material 204 may be provided with an integrated attachment mechanism that allows flexible material 204 to be releasably connected or attached to patient 300 or anatomical elements of patient 300. Flexible material 204 can correspond to 3D-printed elastic materials, rubber, tissue paper, woven materials, non-woven materials, etc. Tracking marker 202 can be produced using the same techniques used to produce flexible material 204, but the color of tracking marker 202 may differ from other parts of flexible material 204 not intended to correspond to tracking marker 202.

[0076] In some examples, the flexible material 204 may correspond to a mesh or film configured to wrap around an object and accept the shape of the object once attached thereto. The flexible material 204 may correspond to a membrane attached to a soft tissue element of the patient 300. The flexible material 204 may be able to attach to nerves, spinal cord, scaffolds, or blood vessels that should not be cut during surgical procedures. In this way, the flexible material 204 can be used to track objects that should not be cut or moved in response to contact with the robot 114. Alternatively or additionally, the flexible material 204 may be able to attach to objects that should be cut during surgical procedures. Examples of such objects include, but are not limited to, tumors, growths, organs, foreign objects (e.g., non-human objects), etc. While this disclosure primarily relates to the tracking of soft tissues, embodiments of this disclosure may also be additionally or alternatively used to track hard tissues, including by attaching the flexible material 204, including the marker array 200, to the patient's bones or other hard tissues.

[0077] The flexible material 204 can be configured to be removed from the patient 300 upon completion of the surgical procedure, or it can be configured to remain in the patient 300 post-operatively. In some embodiments, the flexible material 204 may include a bioabsorbable mesh configured to dissolve after remaining in the patient 300 for a predetermined period of time. In some embodiments, the flexible material 204 may include an optically transparent or translucent material that does not obstruct observation of the object to which the flexible material 204 is attached. In some embodiments, the flexible material 204 can be configured to be cut during surgery, but when the flexible material 204 and the object are cut simultaneously, the cut flexible material 204 still maintains some degree of attachment to the object.

[0078] Tracking markers 202 may correspond to active and / or passive references. The tracking markers 202 may vary depending on whether they are active or inactive, and whether they respond to certain imaging techniques (e.g., visible light, ultrasound, magnetic resonance, radiation, lidar, etc.). As an example, some tracking markers 202 may correspond to fluorescent ink, visible ink, thermally activated ink, etc. In some embodiments, the tracking markers 202 may correspond to relatively small references (e.g., between 2 mm and 4 mm) such that they are visible to the human eye but do not obstruct the visualization of the object to which the flexible material 204 is attached. The spacing between adjacent tracking markers 202 may be larger than the size of the tracking markers 202, but such a configuration is not required. The size of the tracking markers 202 and / or the size of the space between the tracking markers 202 may depend on the resolution of the imaging device 112 used to track the tracking markers 202. For example, a 3D camera may be attached to the robot 114 and may be used to obtain close-up images of the tracking markers 202 because the 3D camera is closer to the surgical area than other imaging devices 112. In some examples, the camera may have a resolution of 0.1mm, which means that the space between tracking markers 202 of size 1mm or 2mm and / or tracking markers 202 of size 1mm or 2mm will be feasible.

[0079] Any of the tracking markers 202 can be identified and registered by the object identification 122 and / or tissue motion tracking 124 of the computing device 102 to aid in object identification and tracking of object movement during and / or after the surgical procedure. In some embodiments, each tracking marker in the tracking markers 202 can be specifically identified once attached to the patient 300 or an anatomical element of the patient 300. After the tracking markers 202 are identified and registered, the tissue motion tracking 124 can be configured to specifically monitor the position and movement of each individual tracking marker 202 during the surgical procedure. Monitoring can occur continuously or periodically during the surgical procedure to track the movement of soft tissue elements and help ensure adherence to the navigation and / or surgical plan and / or determine whether the navigation and / or surgical plan should be adjusted.

[0080] Figure 4 Additional details of an environment 400 in which a system may be implemented according to at least some embodiments of the present disclosure are shown. Additional details of the environment 400 in which a robotic system may operate will be described according to at least some embodiments of the present disclosure. Environment 300 may include a surgical environment, such as an operating room. Environment 300 may include a sterile area 404 and a non-sterile area. Objects contained within a sterile area 404 may be considered sterile or “safe” compared to objects located outside the sterile area 404.

[0081] In some implementations, patient 300 may be located on operating table 416 within sterile area 316, along with healthcare personnel 408 (e.g., physician, surgeon, nurse, support staff, etc.). Some or all of robot 114 may also be located within sterile area 404. First imaging device 112a may be located on, held by, or connected to a portion of robot 114. For example, first imaging device 112a may be connected to the robotic arm 116 or end effector of robot 114. First imaging device 112a may correspond to a visible light camera. As an example, first imaging device 112a may correspond to a 3D camera with a resolution of 1 mm per pixel. First imaging device 112a may be relatively close to the surgical area on which an array of markers 200 is disposed. First imaging device 112a may be configured to focus its field of view on patient 300, thereby excluding other objects in environment 400. As a more specific example, first imaging device 112a may focus its field of view on the surgical area, around which an array of markers 200 is provided.

[0082] The first imaging device 112a is also shown having a plurality of tracking marks 412 attached thereto. The tracking marks 412 may correspond to a reference larger than the tracking marks 202 on the array 200. For example, the tracking marks may have a size of 10 mm to 100 mm in diameter and may correspond to a larger sphere or spherical object attached to the first imaging device 112a at one or more predetermined locations.

[0083] The navigation system 118 can access one or more additional imaging devices 112, such as a second imaging device 112b. The second imaging device 112b may be of the same type as the first imaging device 112a or may be of a different type. For example, the second imaging device 112b may correspond to an ultrasound imaging device, an infrared imaging device, a magnetic resonance imaging diagnostic device, a 2D camera, or a 3D camera.

[0084] The second imaging device 112b may not be as close to the patient 300 as the first imaging device 112b, but may have a field of view that captures both the patient 300 and the first imaging device 112b. In this way, images obtained from the second imaging device 112b can be used to determine the position of the first imaging device 112a relative to the patient 300 and the surgical area. As will be described in further detail herein, the navigation system 118 may be configured to receive image data from the first imaging device 112a and the second imaging device 112b to determine the movement of the patient 300 (e.g., breathing, undulation, etc.) and to determine the position of the robot 114 relative to the patient 300. Such information can be used to support the position and pose of the robot 114 during surgical procedures.

[0085] Now for reference Figure 5 and Figure 6 Various methods will be described according to at least some embodiments of this disclosure. It should be understood that the methods and / or method steps can be performed in any order and can be performed by any or a combination of the components depicted and described in reference system 100. Furthermore, without departing from the scope of this disclosure, steps of one method can be combined with steps of any other method in any order.

[0086] First refer to Figure 5Surgical method 500 will be described according to at least one embodiment of the present disclosure. Method 500 (and / or one or more steps thereof) may be implemented by, for example, at least one processor or otherwise performed. The at least one processor may be the same as or similar to processor 104 of computing device 102 described above. The at least one processor may be part of a robot (such as robot 114) or a navigation system (such as navigation system 118). Method 500 may also be performed using processors other than any of the processors described herein. At least one processor may perform method 500 by executing data stored in memory (such as memory 106). The data (e.g., instructions or neural networks) may cause the processor to perform one or more of image processing 120, object recognition 122, tissue motion tracking 124, and / or reporting and feedback 126.

[0087] Method 500 begins by selecting one or more flexible materials 204 for use during surgical procedures (step 504). The selected flexible materials 204 may be of the same or different types (e.g., mesh, film, membrane, etc.). The selected flexible materials 204 may include one or more tracking markers 202 of the same or different types. In some embodiments, the flexible material 204 may be selected based on the object to which it will be attached. For example, the size, shape, or type of the object to which the flexible material 204 will be attached may determine the type of flexible material 204 selected for that object. The flexible material 204 may also be selected based on the type of tracking marker 202 disposed thereon and / or the capability of the imaging device 112 to track the position of the tracking marker 202.

[0088] Method 500 can continue by attaching the flexible material 204 to the patient 300 and / or anatomical elements of the patient 300 (step 508). This step may also include attaching the imaging device 112 to the robot 114 located near the surgical area. As will be understood, the flexible material 204 can be attached to one or more objects and can be configured to conform to the shape of the object once attached. The attachment mechanism used to attach the flexible material 204 to the object can depend on the expected amount of movement the object will experience during the procedure. For example, if the object is not expected to move significantly during the procedure (e.g., less than a few centimeters or millimeters), a first type of attachment mechanism (e.g., nanohooks) can be used, while if the object is expected to move significantly during the procedure (e.g., more than a few centimeters or millimeters), a second type of attachment mechanism (e.g., staples, sutures, glue, etc.) can be used. The flexible material 204 can be attached during minimally invasive procedures or during open procedures. The flexible material 204 can be attached before the commencement of a surgical plan that includes monitoring the pose of soft tissues while performing one or more other surgical steps. In some implementations, the flexible material 204 can be attached intraoperatively but before soft tissue movement monitoring is required. Imaging device 112 can be attached to robot 114 in a manner that enables imaging device 112 to capture images of tracking marker 202. In other words, imaging device 112 can be attached to robot 114 such that the field of view of imaging device 112 captures the surgical region and the tracking marker 202 surrounding the surgical region. Imaging device 112 attached to robot 114 may have one or more tracking markers 412 attached thereto, which enable tracking of imaging device 112 via another imaging device 112.

[0089] Method 500 may then continue (step 512) by capturing one or more images of the patient 300, objects of the patient 300, imaging device 112 connected to robot 114, and / or tracking markers 202, 412. Images may include any type of image or image data generated using machine vision techniques. Non-limiting examples of imaging techniques that may be used to capture one or more images include ultrasound imaging, magnetic resonance imaging, fluorescence imaging, infrared imaging, visible light imaging, radiation imaging, computed tomography imaging, nuclear medicine imaging, positron emission tomography, combinations thereof, etc. Images may include preoperative images, in which case the images may not necessarily include tracking markers 202, 412 (if, for example, flexible material 204 has not yet been attached to the patient). Images may include intraoperative images, which may or may not include tracking markers 202, 412. Images may include postoperative images, which may or may not include tracking markers 202, 412. In some embodiments, images may be received directly from imaging device 112 or from a computer device storing an electronic copy of the images. The images received in step 512 may include pixel-based images and / or model-based images. In some embodiments, step 512 may include generating a two-dimensional or three-dimensional model from one or more images captured during step 406 using known model generation techniques.

[0090] If the image captured in step 512 does indeed include tracking markers 202, 412, method 500 can then continue with processor 104 performing object recognition 122 and / or tissue motion tracking 124 to determine the initial relative positions of tracking markers 202, 412 (step 516). Specifically, image data from imaging device 112 connected to robot 114 can identify the position of tracking marker 202, while image data from another imaging device 112 not connected to robot 114 can identify the position of tracking marker 412. The combination of image data from these multiple imaging devices 112 can help determine the relative positions of robot 114 and robotic arm 116 relative to the surgical area and patient 300. The initial relative positions of tracking markers 202, 412 can be used to determine the (absolute or relative) initial positions of one or more objects in the image. This position can be determined relative to other objects, relative to any coordinate origin, or a combination thereof.

[0091] Then, method 500 may include capturing one or more additional images of patient 300, one or more additional images of anatomical elements of patient 300, and / or one or more additional images of tracking markers 202, 412 during the surgical procedure (step 520). The images captured intraoperatively may utilize the same or different types of imaging techniques used to capture the images in step 516. The images may correspond to still images and / or video images. Images may be captured continuously or periodically (e.g., at predetermined intervals) during the surgical procedure. Where one or more additional images captured in step 520 include tracking markers 202, 412, step 520 may include registering the tracking markers 202, 412 as depicted in one or more additional images to the tracking markers 202, 412 as depicted in one or more images captured during step 512.

[0092] Method 500 may then continue with processor 104 performing object recognition 122 and / or tissue motion tracking 124 to determine a second position of tracking markers 202, 412 (step 524). This second position may correspond to an absolute position in space or a position relative to at least one other object or adjacent tracking marker 202, 412. In some embodiments, movement of a soft tissue element having flexible material 204 attached thereto may cause one or more tracking markers 202, 412 to move from a first position to a second position between images. The second position of tracking markers 202, 412 may be compared with a previous position of the same tracking markers 202, 412 and / or the current position of other tracking markers 202, 412 (step 528) to determine whether the soft tissue element has moved during surgical procedures (step 532). Processor 104 may perform object recognition 122 and / or tissue motion tracking 124 to perform steps 528 and / or 532.

[0093] If tissue motion tracking 124 determines that a soft tissue element has moved beyond a predetermined threshold, tissue motion tracking 124 may invoke reporting and feedback 126 to report the soft tissue movement (step 536). In some embodiments, the soft tissue movement may be reported to the motion controller of robot 114 and / or the surgeon operating robot 114. Therefore, the feedback generated in step 536 may include electronic feedback for the motion controller and / or auditory / visual feedback for the human user. In some embodiments, the feedback generated in step 536 may be used to optionally adjust the manual or robotic navigation and / or surgical plan taking into account the soft tissue movement (step 540). Adjustments to the navigation plan may include adjustments to avoid cutting the soft tissue after movement, ensure cutting the soft tissue after movement, or a combination thereof. The feedback generated in step 536 may additionally or alternatively include recommended modifications to the navigation and / or surgical plan, which may be automatically generated.

[0094] This disclosure covers embodiments of method 500 that include more or fewer steps than those described above and / or one or more steps that are different from those described above.

[0095] Now refer to Figure 6 Additional details of the second method 600 will be described in accordance with at least some embodiments of this disclosure. Without departing from the scope of this disclosure, the operation of method 600 may be combined with the operation of method 500.

[0096] Method 600 begins by obtaining an image of the flexible material 204 using a first imaging device 112a attached to the robot 114 (step 604). The first imaging device 112a may correspond to a visible light camera. For example, but not limited to, the first imaging device 112a may include a 3D camera with a resolution of 1 mm per pixel. The first imaging device 112a may be attached to the robot arm 116, an end effector, or some other component of the robot 114 that supports the surgical procedure. By attaching the first imaging device 112a to the robot 114, the first imaging device 112a can be positioned to capture a better image of the flexible material 204 compared to other imaging devices 112 in the operating room, because the first imaging device 112a is the imaging device 112 closest to the surgical area.

[0097] Method 600 may further include providing one or more tracking markers 412 to the first imaging device 112a (step 608). The tracking markers 412 may be attached to the first imaging device 112a such that motion of the first imaging device 112a is converted into motion of the tracking markers 412, the motion of which may be tracked by other imaging devices in the operating room. Specifically, images obtained from the second imaging device 112b may be used to track the motion and / or pose of the first imaging device 112a (step 612). In some embodiments, the motion and pose of the first imaging device 112a may be tracked relative to the surgical region, the flexible material 204, and the tracking markers 202 disposed on / in the flexible material 204.

[0098] Method 600 can continue by capturing additional images of the flexible material 204 using the first imaging device 112a (step 616). The additional images captured by the first imaging device 112a may be able to track the movement of the tracking marker 202 attached to the flexible material 204, while other images captured by other imaging devices 112 may not have the capability or resolution required to track the movement of the tracking marker 202. Specifically, the images captured by the first imaging device 112a can be used to determine the body position of the patient 300, the movement of the patient 300, and / or the chest movement of the patient 300 caused by breathing (step 620).

[0099] Images from the first imaging device 112a and the second imaging device 112b can be provided to the navigation system 118 for processing (step 624). Specifically, the navigation system 118 can be configured to process image data from the first imaging device 112a and other imaging devices 112 (e.g., the second imaging device 112b) to determine whether the movement of the patient 300 and / or other objects is appropriate to support surgical navigation. For example, the determination at step 620 can be used to initiate actions as described in operations 536 and / or 540.

[0100] The foregoing is not intended to limit this disclosure to the one or more forms disclosed herein. In the foregoing specific embodiments, for example, for the purpose of simplifying this disclosure, various features of this disclosure are grouped together in one or more aspects, embodiments, and / or configurations. Features of aspects, embodiments, and / or configurations of this disclosure may be combined in alternative aspects, embodiments, and / or configurations other than those discussed above. The approach of this disclosure should not be construed as reflecting an intention that the claims require more features than expressly recited in each claim. Rather, as reflected in the following claims, aspects of the invention lie in fewer than all the features of a single foregoing aspect, embodiment, and / or configuration. Therefore, the following claims are hereby incorporated into this specific embodiment, wherein each claim exists independently as a separate preferred embodiment of this disclosure.

[0101] Furthermore, while the foregoing has already included descriptions of one or more aspects, embodiments, and / or configurations, as well as certain variations and modifications, other variations, combinations, and modifications may be made within the scope of this disclosure, for example, within the skill and knowledge of those skilled in the art, upon understanding of this disclosure. It is intended to obtain, to the permissible extent, rights including alternative aspects, embodiments, and / or configurations, including alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps of those claimed, regardless of whether such alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps are disclosed herein, and not to disclose subject matter specific to any patentable entity.

[0102] The phrases “at least one,” “one or more,” and “and / or” are open-ended expressions that have both connecting and separating characteristics in operation. For example, each of the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” “A, B, and / or C,” and “A, B, or C” means only A, only B, only C, A and B together, A and C together, B and C together, or A, B, and C together.

[0103] The term "a" refers to one or more of the same entity. Therefore, the terms "a," "one or more," and "at least one" are used interchangeably herein. It should also be noted that the terms "comprising," "including," and "having" are used interchangeably.

[0104] As used herein, the term "automatic" and its variations refer to any process or operation, typically continuous or semi-continuous, that can be performed without substantial human input. However, a process or operation can be automatic even if its execution uses substantial or material human input, if input is received prior to its execution. Human input is considered material input if it influences how the process or operation is performed. Human input agreeing to execute a process or operation is not considered "material."

[0105] Various aspects of this disclosure may take the form of a wholly hardware implementation, a wholly software implementation (including firmware, resident software, microcode, etc.), or a combination of software and hardware implementations, all of which may be generally referred to herein as a "circuit," "module," or "system." Any combination of one or more computer-readable media may be used. The computer-readable media may be a computer-readable signal medium or a computer-readable storage medium.

[0106] Computer-readable storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, computer-readable storage media can be any tangible medium that can contain or store programs for use by or in connection with an instruction execution system, apparatus, or device.

[0107] Computer-readable signal media may include, for example, propagated data signals in baseband or as part of a carrier wave, having computer-readable program code embodied therein. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. Computer-readable signal media may be any computer-readable medium that is not a computer-readable storage medium and may convey, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination thereof.

[0108] As used herein, the terms “determine,” “calculate,” “estimate,” and their variations are used interchangeably and include any type of method, process, mathematical operation, or technique.

Claims

1. A method comprising: A first image of an object in a surgical region is obtained via a first imaging device, wherein the first image includes one or more first tracking markers arranged in a first location relative to the object; A second image of the first imaging device is obtained via a second imaging device, wherein the second image includes one or more second tracking markers arranged in a first position; One or more additional first images of the object are obtained via the first imaging device, wherein the one or more additional first images include the one or more first tracking markers; One or more additional second images of the first imaging device are obtained via the second imaging device, wherein the one or more additional second images include the one or more second tracking markers; as well as Based on the one or more additional first images and the one or more additional second images, determine whether the object has moved and / or whether the first imaging device has moved relative to the object.

2. The method of claim 1, wherein the object comprises soft tissue, and the method further comprises: Based on the one or more additional first images, it is determined that the object has moved beyond a predetermined object motion threshold; as well as In response to determining that an object has moved beyond a predetermined object movement threshold, the soft tissue movement is reported to at least one of the surgeon and the robot.

3. The method of claim 2, wherein the soft tissue movement is reported to the robot, the method further comprising: The robotic surgical plan is adjusted based on the soft tissue movement.

4. The method of claim 2, wherein the first imaging device is connected to the robot.

5. The method of claim 4, wherein the first imaging device is connected to at least one of the robot arm and the end effector of the robot.

6. The method according to claim 4, further comprising: Based on the one or more additional second images, it is determined that the first image device has moved more than a predetermined distance threshold relative to the object; The motion of the first imaging device is correlated with the motion of the robot; as well as In response to determining that the first imaging device has moved more than a predetermined distance threshold relative to the object, the surgeon is informed of the movement of the robot or the patient.

7. The method of claim 1, wherein the one or more first tracking markers are disposed on a flexible material, and wherein the flexible material is attached to the patient at or near the surgical area.

8. The method of claim 7, wherein the flexible material is attached to the patient using at least one of nails, sutures, and adhesives.

9. The method of claim 7, wherein the one or more first tracking markers are sized according to the resolution of the first imaging device and are invisible to the second imaging device.

10. The method of claim 7, wherein one or more first tracking marks are printed on the flexible material.

11. The method of claim 10, wherein the flexible material includes a central region substantially devoid of the one or more first tracking markers, and wherein the one or more first tracking markers are radially disposed around the central region.

12. The method of claim 11, wherein the central region includes a region through which a cut is formed.

13. The method of claim 7, wherein the flexible material comprises a transparent or translucent material.

14. The method of claim 7, wherein the one or more second tracking markers are larger than the one or more first tracking markers, and wherein the one or more second tracking markers are attached to the first imaging device.

15. The method according to claim 1, further comprising: Based on the one or more additional first images, it is determined that the one or more first tracking markers have moved to a second position relative to the object; Based on the one or more additional second images, it is determined that the one or more second tracking markers have moved to a second position relative to the object, the second position indicating the motion of the first imaging device; as well as Report the motion of the object and the motion of the first imaging device.

16. The method of claim 1, wherein at least one of the first image and the second image comprises one or more of an ultrasound image, a magnetic resonance image, a fluorescence imaging image, an infrared image, a visible light image, a radiation image, a computed tomography image, a nuclear medicine image, and a positron emission tomography image.

17. A system comprising: A first imaging device, the first imaging device including a first field of view, the first field of view capturing one or more first tracking markers arranged on or around the surgical area; A second imaging device, the second imaging device including a second field of view, the second field of view capturing the first imaging device; and Computing device, wherein the computing device includes: processor; and Computer memory, coupled to the processor and having data stored therein, which, when executed by the processor, enables the processor to: Receive first image data from the first imaging device, wherein the first image data indicates the movement of the one or more first tracking markers relative to the surgical area; Receive second image data from the second imaging device, wherein the second image data indicates the movement of the first imaging device relative to the surgical region; and Based on the combination of the first image data and the second image data, it is determined whether the object in the surgical area has been moved and / or whether the first imaging device has been moved relative to the object.

18. The system of claim 17, wherein one or more second tracking markers are attached to the first imaging device, wherein the one or more first tracking markers are smaller than the one or more second tracking markers, wherein the second field of view includes the surgical region, and wherein the one or more first tracking markers are too small to be identified in the second image data.

19. The system of claim 17, wherein the first imaging device is connected to a robot supporting surgical procedures at the surgical region, and wherein the first imaging device includes a 3D camera.

20. A surgical system comprising: A robot configured to perform surgical plans; A first imaging device is attached to the robot and configured to capture images of one or more first tracking markers arranged on or around a surgical area, wherein the first imaging device includes one or more second tracking markers attached thereto. A second imaging device is configured to capture images of the first imaging device and the one or more second tracking markers attached thereto; and Computing device, the computing device comprising: processor; and Computer memory, coupled to the processor and having data stored therein, which, when executed by the processor, enables the processor to: Receive first image data from the first imaging device, wherein the first image data indicates the movement of the one or more first tracking markers relative to the surgical area; Receive second image data from the second imaging device, wherein the second image data indicates the movement of the first imaging device relative to the surgical region; Based on the combination of the first image data and the second image data, it is determined that the object in the surgical area has moved and / or the first imaging device has moved relative to the object; and In response to determining that the object in the surgical area has moved and / or that the first imaging device has moved relative to the object, the surgical plan is updated.

21. A method comprising: A first image of an object in a surgical region is obtained via a first imaging device (112a), wherein the first image includes one or more first tracking markers (202) arranged in a first position relative to the object. A second image of the first imaging device is obtained via a second imaging device (112b), wherein the second image includes one or more second tracking markers (412) arranged in a first position. One or more additional first images of the object are obtained via the first imaging device, wherein the one or more additional first images include the one or more first tracking markers; One or more additional second images of the first imaging device are obtained via the second imaging device, wherein the one or more additional second images include the one or more second tracking markers; as well as Based on the one or more additional first images and the one or more additional second images, determine whether the object has moved and / or whether the first imaging device has moved relative to the object.

22. The method of claim 21, wherein the object comprises soft tissue, the method further comprising: Based on the one or more additional first images, it is determined that the object has moved beyond a predetermined object motion threshold; as well as In response to determining that the object has moved beyond the predetermined object motion threshold, the soft tissue motion is reported to at least one of the surgeon and the robot (114).

23. The method of claim 21 or 22, wherein the soft tissue movement is reported to the robot, the method further comprising: The robotic surgical plan is adjusted based on the soft tissue movement.

24. The method according to any one of claims 21 to 23, wherein the first imaging device is connected to the robot.

25. The method of claim 24, wherein the first imaging device is connected to at least one of the robot arm (116) and the end effector of the robot.

26. The method according to claim 24 or 25, further comprising: Based on the one or more additional second images, it is determined that the first image device has moved more than a predetermined distance threshold relative to the object; The motion of the first imaging device is correlated with the motion of the robot; as well as In response to determining that the first imaging device has moved more than a predetermined distance threshold relative to the object, the surgeon is informed of the movement of the robot or the patient.

27. The method of any one of claims 21 to 26, wherein the one or more first tracking markers are disposed on a flexible material, and wherein the flexible material is attached to the patient at or near the surgical area.

28. The method of claim 27, wherein the flexible material is attached to the patient using at least one of a nail, suture, and adhesive.

29. The method of claim 27 or 28, wherein the one or more first tracking markers are sized according to the resolution of the first imaging device and are invisible to the second imaging device.

30. The method according to any one of claims 27 to 29, wherein one or more first tracking marks are printed on the flexible material.

31. The method of claim 30, wherein the flexible material includes a central region substantially devoid of the one or more first tracking markers, and wherein the one or more first tracking markers are radially disposed around the central region.

32. The method of claim 31, wherein the central region includes a region through which a cut is formed.

33. The method according to any one of claims 27 to 32, wherein the flexible material comprises a transparent or translucent material.

34. The method according to any one of claims 27 to 32, wherein the one or more second tracking markers are larger than the one or more first tracking markers, and wherein the one or more second tracking markers are attached to the first imaging device.

35. The method according to any one of claims 21 to 34, further comprising: Based on the one or more additional first images, it is determined that the one or more first tracking markers have moved to a second position relative to the object; Based on the one or more additional second images, it is determined that the one or more second tracking markers have moved to a second position relative to the object, the second position indicating the motion of the first imaging device; as well as Report the motion of the object and the motion of the first imaging device.

36. The method according to any one of claims 21 to 35, wherein at least one of the first image and the second image comprises one or more of ultrasound image, magnetic resonance image, fluorescence imaging, infrared image, visible light image, radiation image, computed tomography image, nuclear medicine image and positron emission tomography image.

37. A surgical system comprising: A robot configured to perform surgical plans; A first imaging device is attached to the robot and configured to capture images of one or more first tracking markers arranged on or around a surgical area, wherein the first imaging device includes one or more second tracking markers attached thereto. A second imaging device is configured to capture images of the first imaging device and the one or more second tracking markers attached thereto; and Computing device, the computing device comprising: processor; and A computer memory coupled to the processor and having data stored therein, the data enabling the processor to perform any of the steps according to claim 21 when executed by the processor.

Images