Surgical robotic camera and imaging systems

Abstract

A surgical robotic system includes a surgical robot and a surgical robotic controller. The surgical robot includes at least a first robotic arm and a second robotic arm, and a first camera is mounted on the first robotic arm and a second camera is mounted on the second robotic arm. The surgical robotic controller positions the first and second robotic arms in a surgical coordinate space where the position and line-of-sight of each camera is kinematically tracked by the controller. The controller further receives images of a surgical workspace from the first and second cameras and merge images from the first and second cameras to produce a composite image of the surgical workspace based upon the kinematically tracked position and line-of-sight of each camera.

Description

Atorney Docket No. 67551-722601; P403076WOSURGICAL ROBOTIC CAMERA AND IMAGING SYSTEMSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This PCT application claims the benefit of U.S. Provisional Application 63 / 697,244 [Attorney Docket No. 67551-722.101], filed on September 20, 2024, the full disclosure of which is incorporated herein by reference.BACKGROUND

[0002] Field. The disclosed technology relates generally to medical apparatus and methods and, more particularly, to surgical robotic tools, systems, and methods using multiple cameras and sensors for enhanced imaging.

[0003] Robotic surgery and surgical robots are now in common use. A typical robotic surgical procedure uses multiple surgical tools deployed on two, three, or more surgical robotic arms under the control of a surgical robotic controller. The surgical robotic arms and the tools they carry are typically navigated using robotically controlled cameras and / or other sensors, often in combination with kinematic tracking and placement of the surgical robotic arms and surgical tools.

[0004] The disclosed technology relates particularly to the placement and use of multiple cameras and other sensors for viewing, navigation, and control of the robotic surgical tools performing a procedure in a surgical space surrounding a patient positioned on a surgical table. While many specific combinations and placements of individual cameras and sensors may be used in surgical robotic procedures, many common arrangements will include at least one primary “navigation” camera placed to view most or all of the surgical space and one or more secondary cameras to provide “close-up” views of the tools or anatomy.

[0005] The navigation cameras are usually “stereoscopic” and provide a three-dimension view of the surgical space to allow the surgical robotic controller to track and control the positions of the surgical tools and the robotic arms that carry the tools. Often, the tools, robotic arms, and the target patient anatomy will carry visual tags or fiducials, such as two- or three-dimensional QR codes, that help the robotic controller identify and locate (with six degrees-of-freedom) the viewed anatomy or object in the three-dimensional surgical space. The navigation cameras are often free-standing, e.g. floor or ceiling mounted, but in other cases may be mounted on a surgical robotic arm as part of the surgical robot.

[0006] The secondary cameras may also be stereoscopic, but will often be monoscope (two-dimensional) cameras, such as less expensive RGB cameras using a CMOS sensor. TheAtorney Docket No. 67551-722601; P403076WO secondary cameras will usually be placed on one or more of the surgical robotic arms so that the controller can move the arm to position each secondary camera for optimum viewing.See, commonly owned PCT Publication WO2024 / 160875 (Application No. EP2024 / 052338), filed on January 31, 2024, entitled METHODS AND SYSTEMS FOR TRACKING MULTIPLE OPTICAL MARKERS IN A ROBOTIC SURGICAL PROCEDURE, incorporated herein by reference, for a specific example of a multi-camera surgical robotic system.

[0007] While offering many advantages, presently available multi-camera surgical robotic systems do have some limitations. A first significant limitation is that the line-of-sight of navigation camera is often blocked by the tools and robotic arms being used in the procedure. A second disadvantage is that the secondary cameras are often not stereoscopic and unable to provide a three-dimension image. While these disadvantages can be addressed by reconfiguring the robotic arms and cameras, such reconfiguring often requires manual repositioning of the cameras and tools by the surgical team since the optical controllers are compromised by the blocked field-of- view.

[0008] For these reasons, there is a need for surgical robotic apparatus, systems, and methods that can address the shortcomings described above. In particular, it would be desirable to provide surgical robotic apparatus, systems, and methods which can at least in part automatically reconfigure an arrangement of multiple cameras or other sensors to lessen or eliminate blockages of the field-of-view of the navigation and secondary cameras. It would be still further desirable to provide surgical robotic apparatus, systems, and methods which can reconfigured to provide three-dimension imaging without the necessity of using stereoscopic cameras. These surgical robotic apparatus, systems, and methods should be useful in a variety of robotic surgical settings and should not be limited to use in any particular surgical procedure. Such apparatus, systems and methods should preferably require minimum or no modification to existing surgical robotic systems and should instead use existing arrangements of stereoscopic and two-dimensional camera arrangements to achieve the stated objectives. At least some of these objectives will be met by the technologies disclosed herein.

[0009] 2. Background Art. Relevant publications include WO2024 / 160875 and US2022 / 0079687. Commonly owned publications and applications describing surgical robots having cameras and tools include PCT Publications: WO2022 / 195460; WO2023 / 067415; WO2023 / 118984; WO2023 / 118985; WO2023 / 144602; WO2023 / 152561; WO2023 / 223215; WO2023 / 237922; PCT Applications: PCT / IB2023 / 055439; PCT / IB2023 / 055662;Atorney Docket No. 67551-722601; P403076WOPCT / EP2024 / 052338; PCT / IB2023 / 055663; PCT / EP2024 / 052338; PCT / IB2023 / 056911;PCT / EP2024 / 052353; PCT / EP2024 / 068766; and PCT / EP2024 / 072673; and US Provisional Applications: 63 / 568,102, 63 / 578,395; 63 / 606,001; 63 / 609,490; 63 / 615,076; 63 / 634161, the full disclosures of each of which are incorporated herein by reference.SUMMARY

[0010] In a first aspect, the technologies disclosed herein provide surgical robotic systems comprising a surgical robot and a surgical robotic controller. The surgical robot includes a plurality of robotically positionable components including at least a first robotic arm and a second robotic arm as well as at least two cameras including a first camera on the first robotic arm and a second camera on the second robotic arm. The surgical robotic controller is configured to (a) position the first and second robotic arms in a surgical coordinate space, wherein the position and line-of-sight of each camera is kinematically tracked by the controller, (b) receive images of a surgical workspace from the first and second cameras, and (c) merge images from the first and second cameras to produce a composite image of the surgical workspace, wherein the controller merges the images based upon the kinematically tracked position and line-of-sight of each camera.

[0011] In some instances, the surgical robotic controller may be further configured to selectively position the at least two cameras and to process the images from said at least two cameras to merge the images from said at least two cameras to generate a composite image free from blind spots.

[0012] In some instances, the surgical robotic controller may be further configured to selectively position the at least two cameras and to process the images from said at least two cameras to produce a three-dimensional image, depth map, and / or point cloud. Multiple three-dimensional images, acquired from multiple cameras and / or from single cameras from different viewing angles, can also be merged into single, high detail three-dimensional image.

[0013] In many instances, the surgical robotic system will further comprise at least a third robotic arm and a third camera mounted thereon. In such instances, the surgical robotic controller will be further configured to merge images from the third camera with images from at least one of the first and second cameras based upon the kinematically tracked position and line-of-sight of each camera.

[0014] In some instances, the surgical robotic controller is further configured to register pre-operative patient image data in the surgical coordinate space.Atorney Docket No. 67551-722601; P403076WO

[0015] In some instances, the surgical robotic controller is further configured to allow a user to designate a viewpoint origin for the composite image.

[0016] In some instances, at least some of the cameras comprise three-dimensional (stereoscopic) cameras.

[0017] In some instances, at least some of the cameras comprise two-dimensional cameras.

[0018] In some instances, at least one robotic arm carries a three-dimensional navigation camera and at least two robotic arms carry two-dimensional imaging cameras.

[0019] In some instances, at least one robotic arm carries two cameras. For example, the robotic arm caries one two-dimensional camera and one three-dimensional camera.

[0020] In some instances, at least some of the robotic arms are configured to carry both cameras and surgical tools.

[0021] In other instances, at least some of the robotic arms are configured to carry only cameras.

[0022] In still other instances, at least some of the robotic arms are configured to carry only surgical tools.

[0023] In some instances, two of the robotic arms may each be configured to carry a monoscopic camera and to be arranged in a specific relationship to each other to enable generating a three-dimensional image on a device enabling depth perception, such as a three- dimensional screen, head mounted display, or similar virtual / augmented reality devices.

[0024] The robotic controller is configured to kinematically coordinate movement of at least some of the robotic arms, including most or all arms that carry cameras or other sensors and most or all arms that carry operative surgical tools. In contrast to optical (camera-based) control of the surgical robotic arms, which relies on visually tracking the arm and tool positions using navigation or other camaras, kinematic control allows the robotic controller to move the arms and tools through the surgical space based upon the known relationship between the dimensions and connectivity of each link or section of each arm of the robot. In this way, the robotic controller can control the position, velocity, and acceleration of each of the links in the robotic arm, in order to plan and control movement and actuator forces and torques of the tools with relying on optical tracking of the arms and tools. While the surgical robotic controller can rely solely on kinematic control for the robotic arms and tools, the robot controller can also rely wholly or partially on real-time optical control of the arms and tools when needed or convenient.

[0025] The robotic controller kinematically controls the operation of all of the robotic arms within a common robotic surgical coordinate system which is defined with reference to aAtorney Docket No. 67551-722601; P403076WO common origin or reference point. Such a common robotic surgical coordinate system allows the controller to perform highly accurate kinematic control of the robot arms without the need to rely on optical (camera-based) real time tracking, although such optical tracking may be performed in addition to kinematic tracking and / or may be used to initially register the patient within the common robotic surgical coordinate system.

[0026] In a second aspect, the technologies disclosed herein provide methods for performing a robotic surgical procedure on a patient in a surgical workspace. The methods comprise providing a surgical robot including (a) robotically positionable components including at least a first robotic arm and a second robotic arm and (b) at least two cameras including a first camera on the first robotic arm and a second camera on the second robotic arm. The first and second robotic arms are positioned in a surgical coordinate space and the positions and lines-of-sight for each camera are kinematically tracked, typically by a surgical robotic controller, as the first and second robotic arms are being positioned in the surgical coordinate space. Images of the surgical workspace are received from the first and second cameras, and the images are merged to produce a composite image of the surgical workspace. Typically, the surgical robotic controller merges the images based upon the kinematically tracked position and line-of-sight of each camera.

[0027] In some instances, the robot-mounted cameras can also be tracked optically or using sensors by attaching markers to the cameras, robotic arms, tool holders, flanges, or the like, in addition to kinematic tracking. Typically, the markers could be tracked by a navigation camera when the marker is in line-of-sight.

[0028] In some instances, the surgical robot comprises a surgical robotic controller which performs the positioning, tracking, receiving and merging steps.

[0029] In some instances, the surgical robot comprises a display which presents the composite image.

[0030] In some instances, the method further comprises selectively positioning the at least two cameras and to process the images from said at least two cameras to merge the images from said at least two cameras to generate a composite image free from blind spots.

[0031] In some instances, the methods further comprise selectively positioning the at least two cameras and processing the images from said at least two cameras to produce a three- dimensional image.

[0032] In some instances, the surgical robot includes at least a third robotic arm and a third camera mounted thereon, and the methods further comprise merging images from the thirdAtorney Docket No. 67551-722601; P403076WO camera with images from at least one of the first and second cameras based upon the kinematically tracked position and line-of-sight of each camera.

[0033] In some instances, the methods further comprise registering pre-operative patient image data in the surgical coordinate space. In some instances, the methods further comprise designating a viewpoint origin for the composite image.BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The novel features of the disclosure are set forth with particularity in the appended claims. Abetter understanding of the features and advantages of the present disclosure will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the disclosure are utilized, and the accompanying drawings of which:

[0035] FIG. l is a perspective view of a mobile surgical robotic cart having multiple surgical robotic arms carrying multiple cameras in accordance with the disclosed technology shown positioned beneath a surgical table with the surgical robotic arms deployed over a draped patient, in accordance with some embodiments.

[0036] FIGS. 2 A and 2B are more detailed views of the mobile surgical robotic cart of FIG.1 showing specific tool and camera arrangements, in accordance with some embodiments.

[0037] FIG. 3 is a schematic illustration of how the cameras of FIGS. 2A and 2B may be arranged to obtain multiple overlapping views that can be arranged into a panoramic view free from blind spots, in accordance with some embodiments.

[0038] FIG. 4 illustrates how two two-dimensional cameras may be arranged in order to generate a three-dimensional image, in accordance with some embodiments.

[0039] FIG. 5 is a logic flow diagram providing an exemplary protocol for converting multiple two-dimensional and three-dimensional digital images taken by the cameras described herein into a single fused image, in accordance with some embodiments.DETAILED DESCRIPTION

[0040] With reference now to the figures and several representative embodiments of the described technology, the following detailed description is provided.

[0041] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.Atorney Docket No. 67551-722601; P403076WO

[0042] As used herein, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Any reference to “or” herein is intended to encompass “and / or” unless otherwise stated.

[0043] As used herein, the term “about” in some cases refers to an amount that is approximately the stated amount.

[0044] As used herein, the term “about” refers to an amount that is near the stated amount by 10%, 5%, or 1%, including increments therein.

[0045] As used herein, the term “about” in reference to a percentage refers to an amount that is greater or less the stated percentage by 10%, 5%, or 1%, including increments therein.

[0046] As used herein, the phrases “at least one”, “one or more”, and “and / or” are open- ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B and C”, “at least one of A, B, or C”, “one or more of A, B, and C”, “one or more of A, B, or C” and “A, B, and / or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together.

[0047] An exemplary robotic surgical system 10 which incorporates the technologies disclosed herein is shown in FIG. 1. The robotic surgical system 10 comprises a chassis 12, typically a single, rigid frame which provides a base or platform for three robotic arms 20, 22 and 24 that are placed relatively far apart on opposite longitudinal ends (only one end 14 is visible) of an upper surface 18 of the chassis 12, typically approximately one meter apart, thus allowing for desirable attributes such as reachability, maneuverability, and an ability to apply significant force. In the illustrated embodiment, robotic surgical arms 22 and 24 are on the first end 14 of the chassis 12 and robotic surgical arm 20 is on the other longitudinal end of the chassis. The chassis is preferably but not necessarily mobile, e.g. being in the form of a mobile cart as described in commonly owned WO2022 / 195460, incorporated herein by reference. In other embodiments and implementations, the surgical arms 20, 22 and 24 could be mounted on a base or other structure of a surgical table. For performing the camera arrangements in accordance with the disclosed technology, it is necessary only that the robotic surgical arms be located on a stable platform that allows the arms to be moved kinematically or otherwise within a common robotic coordinate system under the control of a surgical robotic controller, typically an on-board controller having a user interface, such as display screen 32.

[0048] While usually comprising a single, rigid chassis, in some instances the chassis 12 may comprise separate modules, platforms, or components, that are assembled at or near the surgical table, as described for example in commonly owned PCT PublicationAtorney Docket No. 67551-722601; P403076WOWO2024 / 165397 (Application PCT / EP2024 / 052353, entitled Integrated Multi-Arm Mobile Surgical Robotic System, filed on January 29, 2024, the full disclosure of which is incorporated herein by reference. The only requirement of the chassis 12 is that it provides a stable base for all the surgical arms so that they may be accurately and precisely kinematically positioned and tracked by the surgical robotic controller in a single surgical robotic coordinate space.

[0049] The chassis 12 of the robotic surgical system 10 is typically configured to be temporarily placed under a surgical table 40 when performing the robotic surgical procedure, allowing the robotic surgical system 10 to be stored remotely before and after the procedure. The robotic arms 20, 22, and 24 may optionally be configured to be retracted into the chassis 12 of the robotic surgical system, allowing the system to be moved into or out of the surgical field in a compact configuration.

[0050] The first and second robotic arms 20 and 22 are typically configured to carry tool holder assemblies, such as those described in commonly owned PCT Publication W02025 / 008416 (Application PCT / EP2024 / 068766), filed on July 3, 2024, entitled MECHANICAL GEARBOX FOR ROBOTIC APPLICATIONS, the full disclosure of which is incorporated herein by reference. The tool holder assemblies 26 and 28 may include a flange portion which houses the electronics and will typically be maintained out of the sterile field and a mechanical holder / gripper portion that holds and manipulates the interventional tools, e.g. probes, grinders, drills, screwdrivers, and the like. Tool holder 26 is shown to hold a screwdriver in FIG. 2B and tool holder 28 is shown to carry a drill 46 in both FIGS. 2A and 2B.

[0051] The tool holder assemblies 26 and 28 will typically each carry a 2D camera, such as a RGB camera 44, that provides a close-up view of the operating filed of the tool. In addition, the tool holder assemblies 26 and 28 can be configured to carry a mountable 3D (stereoscopic) camera 60 as shown in FIG. 2A (unmounted) and FIG. 2B mounted).

[0052] Use of the multiple cameras 32, 44a, 44b, and 60 for viewing a target location on a patient’s anatomy, such as surgically exposed vertebra V, is shown in FIG. 3. The field of view of each of the cameras is shown in broken line. As can be seen, no one camera has a complete view of all facets of the vertebral surface. The view of the navigational cameras is partially blocked by the tool holders 26 and 28 as well as the 3D camera 30 while the views of the secondary cameras are partially obscured by the anatomy. The robotic controller 32, however, can be programmed and configured to generate a composite image spanning the entire surgical field by combined the information from each of the individual camera view.Atorney Docket No. 67551-722601; P403076WOSuch composite views are produced by kinematically tracking the position and line-of sight for each of the cameras and then aligning the images to generate the full of view of the region of interest. Such composite views are useful both to allow the user to have an unobstructed view of the surgical filed in real time and to improve optical positioning and control of the surgical tools.

[0053] In addition to allowing the construction of composite panoramic and other images, the camera systems of the present technologies may also allow the generation of three- dimensional images without using a stereoscopic camera. As shown in FIG. 4, tool holders 26 and 28 can be aligned by the robotic controller 32 so that the 2D camera 44a and 44B are laterally adjacent and spaced a short distance apart. By thus providing the necessary parallax, the two individual 2D cameras can be used to generate three-dimensional image by proper focus a target site, such as a vertebral region V.

[0054] FIG. 5 is a logic flow diagram providing an exemplary protocol for converting multiple two-dimensional and three-dimensional digital images taken by the cameras described herein into a single fused image.

[0055] Reference Nos.Atorney Docket No. 67551-722601; P403076WO

[0056] While preferred embodiments of the present disclosure have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the disclosure. It should be understood that various alternatives to the embodiments of the disclosure described herein may be employed in practicing the disclosure. One of skill in the art will realize that several variations on the disclosed embodiments are possible while staying within the bounds of the current disclosure. Solely by way of example, different variations in the number of navigation cameras, robotic arms, markers, and end effectors can be used without departing from the disclosure. As another example, markers of varying sizes can be used. The embodiments provided are representative in nature.

Claims

Atorney Docket No. 67551-722601; P403076WOCLAIMSWHAT IS CLAIMED IS:

1. A robotic surgical system comprising: a surgical robot comprising:(a) robotically positionable components including at least a first robotic arm and a second robotic arm, and(b) at least two cameras including a first camera on the first robotic arm and a second camera on the second robotic arm; and a surgical robotic controller configured to:(a) position the first and second robotic arms in a surgical coordinate space, wherein a position and line-of-sight of each camera is kinematically tracked by the controller;(b) receive images of a surgical workspace from the first and second cameras;(c) merge images from the first and second cameras to produce a composite image of the surgical workspace, wherein the controller merges the images based upon the kinematically tracked position and line-of-sight of each camera.

2. The robotic surgical system of claim 1, wherein the surgical robotic controller is further configured to selectively position the at least two cameras and to process the images from the at least two cameras to merge the images from the at least two cameras and to generate a composite image free from blind spots.

3. The robotic surgical system of claim 1 or 2, wherein the surgical robotic controller is further configured to selectively position the at least two cameras and to process the images from the at least two cameras to produce a three-dimensional image.

4. The robotic surgical system of any one of claims 1 to 3, further comprising a third robotic arm and a third camera mounted thereon, wherein the surgical robotic controller is further configured to merge images from the third camera with images from at least one of the first and second cameras based upon the kinematically tracked position and line-of-sight of each camera.

5. The robotic surgical system of any one of claims 1 to 4, wherein the surgical robotic controller is further configured to register pre-operative patient image data in the surgical coordinate space.

6. The robotic surgical system of any one of claims 1 to 5, wherein the surgical robotic controller is further configured to allow a user to designate a viewpoint origin for the composite image.Atorney Docket No. 67551-722601; P403076WO7. The robotic surgical system of any one of claims 1 to 6, wherein at least some of the cameras comprise three-dimensional (stereoscopic) cameras.

8. The robotic surgical system of any one of claims 1 to 7, wherein at least some of the cameras comprise two-dimensional cameras.

9. The robotic surgical system of any one of claims 1 to 8, wherein at least one robotic arm carries a three-dimensional navigation camera and at least two robotic arms carry two-dimensional imaging cameras.

10. The robotic surgical system of any one of claims 1 to 9, wherein one robotic arm carries two cameras.

11. The robotic surgical system of claim 10, wherein the one robotic arm carries one two-dimensional camera and one three-dimensional camera.

12. The robotic surgical system of any one of claims 1 to 11, wherein at least some of the robotic arms are configured to carry both cameras and surgical tools.

13. The robotic surgical system of any one of claims 1 to 11, wherein at least some of the robotic arms are configured to carry only cameras.

14. The robotic surgical system of any one of claims 1 to 11, wherein at least some of the robotic arms are configured to carry only surgical tools.

15. A method for performing a robotic surgical procedure on a patient in a surgical workspace, the method comprising: providing a surgical robot comprising:(a) robotically positionable components including at least a first robotic arm and a second robotic arm, and(b) at least two cameras including a first camera on the first robotic arm and a second camera on the second robotic arm; positioning the first and second robotic arms in a surgical coordinate space; kinematically tracking position and line-of-sight for each camera as the first and second robotic arms are being positioned in the surgical coordinate space; receiving images of the surgical workspace from the first and second cameras; and merging images from the first and second cameras to produce a composite image of the surgical workspace, wherein a controller merges the images based upon the kinematically tracked position and line-of-sight of each camera.

16. The method of claim 15, wherein the surgical robot comprises a surgical robotic controller which performs the positioning, tracking, receiving and merging steps.Atorney Docket No. 67551-722601; P403076WO17. The method of claim 15 or 16, wherein the surgical robot comprises a display which presents the composite image.

18. The method of any one of claims 15 to 17, further comprising selectively positioning the at least two cameras and processing the images from the at least two cameras to merge the images from the at least two cameras to generate a composite image free from blind spots.

19. The method of any one of claims 15 to 18, further comprising selectively positioning the at least two cameras and processing the images from the at least two cameras to produce a three-dimensional image.

20. The method of any one of claims 15 to 19, wherein the surgical robot includes at least a third robotic arm and a third camera mounted thereon, further comprising merging images from the third camera with images from at least one of the first and second cameras based upon the kinematically tracked position and line-of-sight of each camera.

21. The method of any one of claims 15 to 20, further comprising registering preoperative patient image data in the surgical coordinate space.

22. The method of any one of claims 15 to 21, further comprising designating a viewpoint origin for the composite image.

Images