System and method for autofocus of a camera assembly of a surgical robotic system - Patents.com

Abstract

The surgical robotic system includes a sensor unit, a controller, and a robot subsystem. The robot subsystem is in communication with the sensor unit and the controller. The robot subsystem further includes a plurality of robot arms, each having an end effector at a distal end thereof. The robot subsystem also includes a camera assembly having at least two cameras and an autofocus unit for autofocusing a lens of each of the cameras.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This disclosure claims priority to U.S. Provisional Patent Application No. 63 / 153,128, filed February 24, 2021, and U.S. Provisional Patent Application No. 63 / 176,634, filed April 19, 2021, the entire contents of which are incorporated by reference herein.

[0002] The present disclosure is directed to minimally invasive surgical apparatus and associated methods, and more particularly to robotic surgical systems insertable into a patient for performing selected surgical procedures therein. [Background technology]

[0003] The field of minimally invasive surgery has expanded rapidly since its introduction in the early 1990s. Although minimally invasive surgery significantly improves patient outcomes, this improvement comes at the expense of the surgeon's ability to operate with precision and ease. During laparoscopic surgery, surgeons must insert laparoscopic instruments through small incisions in the patient's abdominal wall. The nature of inserting instruments through the abdominal wall limits laparoscopic instrument movement as they cannot be moved side to side without causing damage to the abdominal wall. Standard laparoscopic instruments are limited to four axes of motion. These four axes of motion are the movement of the instrument in and out of the trocar (axis 1), the rotation of the instrument within the trocar (axis 2), and the angular movement of the trocar in two planes while maintaining the pivot point at which the trocar enters the abdominal cavity (axes 3 and 4). For over two decades, the majority of minimally invasive surgical procedures have been performed using only these four degrees of freedom of motion. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 10,285,765 [Patent Document 2] US Patent Application Publication No. 2019 / 0076199 Summary of the Invention [Problem to be solved by the invention]

[0005] Existing robotic surgical devices have attempted to solve many of these problems. Some existing robotic surgical devices replicate non-robotic laparoscopic surgery with additional degrees of freedom at the end of the instrument. However, even with many costly modifications to the surgical procedure, existing robotic surgical devices have failed to provide improved patient outcomes in the majority of procedures in which they are used. Additionally, existing robotic devices create increased separation between the surgeon and the surgical end effector. This increased separation leads to injury due to misinterpretation of the surgeon's actions and the forces applied by the robotic device. Because the degrees of freedom of many existing robotic devices are unfamiliar to human operators, surgeons must undergo extensive training on robotic simulators prior to operating on a patient to minimize the possibility of inadvertently causing injury.

[0006] A typical robotic system includes one or more robotic arms and one or more associated cameras. To control existing robotic systems, the surgeon sits at a console and controls manipulators with both hands and feet, thus controlling the camera and the robotic arms. Additionally, the camera may remain in a semi-fixed position and be moved by combined foot and hand movements from the surgeon. These semi-fixed cameras have a limited field of view, which results in difficult visualization of the surgical field.

[0007] In conventional surgical robotic systems, there are multiple ways to focus the camera at the intended surgical site. Typically and conventionally, the surgeon must adjust the focus either using a manual dial or based solely on image data received from the camera's image sensor, based on the system's autofocus features. Typical autofocus mechanisms in the medical field may also employ phase detection autofocus, time of flight (light reflection), or some other light-based estimation method.

[0008] A drawback of these conventional systems is that they require the surgeon to interrupt the surgical procedure and manually change the focus of the camera, which distracts the surgeon during the surgical procedure. In systems that employ autofocus technology, the focus or field of view of the camera often does not align with or cover the actual portion of the surgical site that the surgeon needs to view and requires a larger depth of field. Conventional cameras that employ a larger depth of field require more light, which reduces the overall resolution. [Means for solving the problem]

[0009] In the presently disclosed robotic surgical system, the system utilizes the position of the camera of the camera assembly and the position of the grasper or end effector of the robotic arm to determine the field of view and focal length or focus of the camera. By using the position information of the end effector relative to the camera, the presently disclosed system is able to determine the focal length and focus of the camera, while allowing the surgeon to concomitantly view the parts of the surgical site that are important or that the surgeon wishes to see, without having to rely solely on image data from the camera. Thus, the system has a more accurate focus with fewer false positives, and further has the ability to always focus at the desired field of view.

[0010] According to one embodiment, the present disclosure provides a surgical robotic system including a sensor unit, a controller, and a robot subsystem. The robot subsystem is in communication with the sensor unit and the controller. The robot subsystem further includes a camera assembly having a plurality of robot arms, each having an end effector at a distal end thereof, at least two cameras, and an autofocus unit configured to autofocus a lens of each of the at least two cameras. The controller can be configured to calculate a desired focal length based on camera and robot arm statement information received from the sensor unit. In response, the autofocus unit can be configured to autofocus a lens of each of the at least two cameras based on the desired focal length.

[0011] The status information may include a distance from each camera to each end effector of the robot arm that is within the surgeon's field of view. Additionally, the status information may include position and orientation information for each camera and each end effector. Based on the calculated desired focal length, the controller may be configured to determine a focus command according to the particular focal depth and send the focus command to the autofocus unit. In response, the autofocus unit may be configured to adjust the physical focal length of each camera to focus the lens of each of the cameras.

[0012] Additionally, the controller may be configured to filter the desired focal length to reduce rapid changes in focus data. The strength of the filter used for filtering may vary based on the degree of head movement of the surgeon. A different desired focal length may also be calculated for each of the cameras. The desired focal length may be further calculated using a weighting algorithm. Each of the robotic arms may be weighted differently in the weighting algorithm. The weight of each robotic arm is a function based on system parameters.

[0013] In particular, each robot arm may include multiple joints. These joints may include a shoulder joint, an elbow joint, and a wrist joint. Thus, the system parameters may include the distance from the center of each end effector within the field of view of each camera, the state of each end effector, and the position of the elbow joint. In one embodiment, the focusing speed may be increased as each end effector moves outward from the target position and decreased as each end effector moves toward the target position.

[0014] According to another embodiment, the present disclosure provides a robotic subsystem including a plurality of robotic arms, each having an end effector at a distal end thereof, and a camera assembly. The camera assembly may include at least two cameras, a controller, and an autofocus unit configured to autofocus a lens of each of the cameras. The controller may be configured to calculate a desired focal length based on state information of the at least two cameras and the plurality of robotic arms received from the sensor unit. In response, the autofocus unit may be configured to autofocus a lens of each of the at least two cameras based on the desired focal length.

[0015] The focusing speed is increased as the robotic arm moves outward from the target location and is decreased as the robotic arm moves inward toward the target location. Additionally, the status information includes at least one of a distance from each camera to each end effector of the multiple robotic arms within the surgeon's field of view, and position and orientation information of the at least two cameras and each end effector of the multiple robotic arms.

[0016] These and other features and advantages of the present disclosure will be more fully understood by reference to the following detailed description in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout the various views, the drawings illustrate the principles of the present disclosure and, although not to scale, show relative dimensions. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a schematic diagram of a surgical robotic system of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram of a camera assembly of the surgical robotic system of FIG. 1 in accordance with the teachings of the present disclosure. [Diagram 3] FIG. 3 is a schematic diagram of the robotic subsystem of FIG. 1 in accordance with the teachings of the present disclosure. [Figure 4] FIG. 4 is a schematic diagram illustrating the processing of system state data by a weighting algorithm unit used by the system of the present disclosure. [Figure 5A] 5A and 5B are schematic diagrams illustrating processing of system state data by a weighting algorithm unit used by a surgical robotic system according to a second embodiment of the present disclosure. [Figure 5B] Same as above. [Figure 6A] FIG. 6A is a schematic diagram illustrating processing of system state data by a focal length calculator unit used by a surgical robotic system in accordance with the teachings of the present disclosure. [Figure 6B] FIG. 6B is a schematic diagram illustrating processing of system state data by a second embodiment of a focal length calculation unit used by a surgical robotic system in accordance with the teachings of the present disclosure. [Figure 6C] FIG. 6C is a schematic diagram illustrating processing of system state data by a third embodiment of a focal length calculation unit used by a surgical robotic system in accordance with the teachings of the present disclosure. [Figure 7] FIG. 7 is a schematic diagram of the field of view of a camera assembly unit used by a surgical robotic system in accordance with the teachings of the present disclosure. [Figure 8] FIG. 8 is a graphical representation of a weight graph used by the surgical robotic system of the present disclosure. [Figure 9] FIG. 9 is a graphical representation of a second embodiment of a weight graph used by the surgical robotic system of the present disclosure. [Figure 10A]10A-10C are schematic diagrams of the field of view of a camera assembly unit for limited focusing in accordance with the teachings of the present disclosure. [Figure 10B] Same as above. [Figure 10C] Same as above. [Figure 11] FIG. 11 is a schematic diagram illustrating processing of system state data by a fourth embodiment of a focal length calculation unit used by a surgical robotic system in accordance with the teachings of the present disclosure. [Figure 12] FIG. 12 is a schematic flow chart diagram illustrating a method for autofocusing a camera of a camera assembly in accordance with the teachings of the present disclosure. [Figure 13] 13 and 14 are schematic diagrams illustrating the processing of system state data, such as motor and joint data, processed by a forward kinematics unit used by the surgical robotic system of the present disclosure. [Figure 14] Same as above. [Figure 15] FIG. 15 is an example focus curve that may be pre-stored and utilized by the controller of the surgical robotic system of the present disclosure. [Figure 16] FIG. 16 is an example focus curve with multiple targets at different distances superimposed on top of each other. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] In the following description, numerous specific details are described with respect to the disclosed system and method, and the environment in which the system and method may operate, to provide a thorough understanding of the disclosed subject matter. However, it will be apparent to those skilled in the art that the disclosed subject matter may be practiced without such specific details, and that certain features well known in the art have not been described in detail to avoid complexity and enhance clarity of the disclosed subject matter. Furthermore, it will be understood that any examples provided below are merely illustrative and should not be construed in a limiting manner, and that other systems, apparatuses, and / or methods may be used to implement or accomplish the teachings of the present invention and are considered to be within the scope of the present invention.

[0019] Although the exemplary embodiment is described as using multiple units to perform the exemplary process, it is understood that the exemplary process may also be performed by one or more modules. Furthermore, it is understood that the term controller / control unit refers to a hardware device that includes a memory and a processor and is specifically programmed to perform the processes described herein. The memory is configured to store modules, and the processor is specifically configured to execute the modules to perform one or more processes further described below.

[0020] Furthermore, the control logic of the present disclosure may be embodied as a non-transitory computer-readable medium on a computer-readable medium that includes executable program instructions executed by a processor, a controller / control unit, or the like. Examples of computer-readable media include, but are not limited to, ROM, RAM, compact disk (CD)-ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. The computer-readable recording medium may also be distributed in a network-coupled computer system such that the computer-readable medium is stored and executed in a distributed manner, for example, by a telematics server or a controller area network (CAN).

[0021] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0022] Unless specifically stated or clear from the context, the term "about" as used herein is understood to be within the normal tolerance in the art, for example, within 2 standard deviations of the mean. "About" can be understood to be within 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, 0.05%, or 0.01% of the stated value. Unless otherwise clear from the context, all numerical values ​​provided herein are modified by the term "about".

[0023] While the systems and methods of the present disclosure may be designed for use with one or more surgical robotic systems used as part of a virtual reality surgical system, the robotic systems of the present disclosure may be used in connection with any type of surgical system, including, for example, robotic surgical systems, linear stick surgical systems, and laparoscopic systems. Additionally, the systems of the present disclosure may be used in other non-surgical systems where a user requires access to a myriad of information while controlling an apparatus or instrument.

[0024] The systems and methods disclosed herein may be incorporated into and utilized with, for example, the robotic surgical apparatus and associated systems disclosed in U.S. Patent Application Publication No. PCT / US2020 / 39203 and PCT / US2020 / 39203, and / or the camera system disclosed in U.S. Patent Application Publication No. PCT / US2020 / 39203, the entire contents and teachings of which are incorporated herein by reference. Surgical robotic systems forming part of the present disclosure include surgical systems having a user workstation, a robotic assistance system (RSS), a motor unit, and an implantable surgical robot subsystem including one or more robotic arms and one or more camera assemblies. The implantable robotic arms and camera assemblies may form part of a single support axis robotic system or may form part of a split arm architecture robotic system.

[0025] The robotic arm may have joint portions or regions associated with motions associated with the user's shoulder, elbow, wrist and fingers, for example as shown in FIG. 3, to provide human-like motions. For example, a robotic elbow joint may track the position and orientation of a human elbow, and a robotic wrist joint may track the position and orientation of a human wrist. The robotic arm may also have a terminal region associated with it, which may terminate in an end effector or gripper that tracks the motion of one or more of the user's fingers, such as the index finger, for example, when the user pinches the index finger and thumb together. The robotic shoulder may be fixed in place while the robotic arm tracks the motion of the user's arm. In one embodiment, the position and orientation of the user's torso is subtracted from the position and orientation of the user's arm. This subtraction allows the user to move the torso without the robotic arm moving.

[0026] FIG. 1 is a schematic block diagram of a surgical robotic system 10 according to the teachings of the present invention. The system 10 includes a display device or unit 12, a virtual reality (VR) computing unit 14, a sensing and tracking unit 16, a computing unit 18, and a robotic subsystem 20. The display unit 12 can be any selected type of display for displaying information, images, or videos generated by the VR computing unit 14, the computing unit 18, and / or the robotic subsystem 20. The display unit 12 can include or form part of, for example, a head mounted display (HMD), a screen or display, a three-dimensional (3D) screen, or the like. The display unit can also include an optional sensor and tracking unit 16A, such as may be found in commercially available head mounted displays. The sensing and tracking units 16 and 16A can include one or more sensors or detectors that are coupled to a user of the system, such as, for example, a nurse or surgeon. The sensors can be coupled to the user's arms, and if a head mounted display is not used, additional sensors can also be coupled to the user's head and / or neck area. The sensors in this arrangement are represented by the sensors and tracking unit 16. If the user employs a head-mounted display, eye, head and / or neck sensors and associated tracking technology may be incorporated or employed within that device and therefore form part of the optional sensors and tracking unit 16A.

[0027] The sensors of the sensor and tracking unit 16 coupled to the surgeon's arm may preferably be coupled to selected regions of the arm, such as, for example, the shoulder region, elbow region, wrist or hand region, and, if desired, the fingers. The sensors generate position data indicative of the position of the selected portion of the user. The sensing and tracking unit 16 and / or 16A may be utilized to control the operation of the camera assembly 44 and the robotic arm 42 of the robotic subsystem 20. The position data 34 generated by the sensors of the sensor and tracking unit 16 may be communicated to the computation unit 18 for processing by the processor 22. The computation unit 20 may be configured to determine or calculate from the position data the position and / or orientation of each portion of the surgeon's arm and communicate this data to the robotic subsystem 20.

[0028] According to an alternative embodiment, the sensing and tracking unit 16 may use sensors coupled to the surgeon's torso or any other body part. Furthermore, the sensing and tracking unit 16 may use, in addition to the sensors, an inertial momentum unit (IMU) having, for example, an accelerometer, a gyroscope, a magnetometer, and a motion processor. The addition of a magnetometer is standard practice in the field, as the magnetic orientation can reduce sensor drift around the vertical axis. Alternative embodiments include sensors placed within surgical materials, such as gloves, surgical scrubs, or surgical gowns. The sensors may be reusable or disposable. Additionally, the sensors may be disposed external to the user, such as in a fixed location in a room, such as an operating room. The external sensors may be configured to generate external data 36 that may be processed by the computation unit and thus used by the system 10. According to another embodiment, when the display unit 12 is a head-mounted display using an associated sensor and tracking unit 16A, the device generates tracking and position data 34A that is received and processed by the VR computation unit 14. Additionally, the sensor and tracking unit 16 may include a hand control device, if desired.

[0029] In embodiments in which the display is an HMD, the display unit 12 may be, for example, a virtual reality head-mounted display such as Oculus Rift, Varjo VR-1, or HTC Vive Pro Eye. The HMD may provide the user with a display coupled to or attached to the user's head, lenses that allow for focused viewing of the display, and a sensor and / or tracking system 16A that provides tracking of the position and orientation of the display. The position and orientation sensor system may include, for example, accelerometers, gyroscopes, magnetometers, motion processors, infrared tracking, eye tracking, computer vision, emitting and sensing alternating magnetic fields, and any other method of tracking at least one of position and orientation, or any combination thereof. As is known, the HMD may provide image data from a camera assembly 44 to both eyes of the surgeon. To maintain a virtual reality experience for the surgeon, the sensor system may track the position and orientation of the surgeon's head and then relay the data to the VR computation unit 14 and, if desired, to the computation unit 18. The computing unit 18 may further adjust the pan and tilt of the robot's camera assembly 44 to follow the user's head movements.

[0030] When associated with an HMD, such as associated with the display unit 12 and / or tracking unit 16A, sensor or position data 34A generated by the sensors may be communicated to the computing unit 18, either directly or via the VR computing unit 14. Similarly, tracking and position data 34 generated by other sensors in the system, such as the sensing and tracking unit 16 that may be associated with the user's arms and hands, may be communicated to the computing unit 18. The tracking and position data 34, 34A may be processed by the processor 22 and stored, for example, in the storage unit 24. The tracking and position data 34, 34A may also be used by the control unit 26, which may responsively generate control signals for controlling the operation of one or more parts of the robotic subsystem 20. The robotic subsystem 20 may include a user workstation, a robotic assistance system (RSS), a motor unit 40, and an implantable surgical robot including one or more robotic arms 42 and one or more camera assemblies 44. The implantable robotic arm and camera assembly may form part of a single support axis robotic system as disclosed and described in U.S. Patent No. 5,393,633, or may form part of a split-arm (SA) architecture robotic system as disclosed and described in PCT Patent Application No. PCT / US20 / 39203.

[0031] The control signals generated by the control unit 26 may be received by a motor unit 40 of the robotic subsystem 20. The motor unit 40 may include a series of servo motors configured to separately drive a robotic arm 42 and a camera assembly 44. The robotic arm 42 may be controlled to follow the scaled movements or motions of the surgeon's arm as sensed by associated sensors. The robotic arm 42 may have portions or regions associated with motions that may be associated with a user's shoulder, elbow, and wrist joints and fingers. For example, a robotic elbow joint may follow the position and orientation of a human elbow, and a robotic wrist joint may follow the position and orientation of a human wrist. The robotic arm 42 may also have a terminal region associated with it that may terminate in an end effector that follows the motion of one or more of the user's fingers, such as the index finger, for example, when the user pinches the index finger and thumb together. The robotic shoulder is fixed in position while the robotic arm follows the motion of the user's arm. In one embodiment, the position and orientation of the user's torso is subtracted from the position and orientation of the user's arms, allowing the user to move the torso without the robotic arms moving.

[0032] The robotic camera assembly 44 is configured to provide the surgeon with image data 48, such as, for example, a live video feed of the procedure or surgical site, as well as to allow the surgeon to operate and control the cameras forming part of the camera assembly 44. The camera assembly 44 preferably includes a pair of cameras 60A, 60B whose optical axes are axially spaced a selected distance, known as the inter-camera distance, to provide a stereoscopic view or image of the surgical site. The surgeon can control the operation of the cameras 60A, 60B either by operation of a head-mounted display, or through sensors coupled to the surgeon's head, or using hand controls or sensors that track the user's head or arm movements, thereby allowing the surgeon to intuitively and naturally obtain a desired view of the surgical site. The cameras are movable in multiple directions, including, for example, yaw, pitch, and roll, as is well known. The components of the stereoscopic camera can be configured to provide a user experience that feels natural and comfortable. In some embodiments, the inter-axial distance between the cameras can be modified to adjust the user's perceived depth of the surgical site.

[0033] According to one embodiment, the camera assembly 44 may be actuated by the surgeon's head motion. For example, during surgery, if the surgeon wishes to view an object located above the current field of view (FOV), the surgeon looks upward, which causes the stereoscopic camera to rotate about a pitch axis upward from the user's perspective. Image or video data 48 generated by the camera assembly 44 may be displayed on the display unit 12. If the display unit 12 is a head-mounted display, the display may include an embedded tracking and sensor system 16A that obtains raw orientation data in the yaw, pitch and roll directions of the HMD, as well as position data in Cartesian space (x,y,z) of the HMD. However, alternative tracking systems may be used to provide supplemental position and orientation tracking data of the display instead of or in addition to the HMD's embedded tracking system. Examples of camera assemblies suitable for use in the present disclosure include the camera assemblies disclosed in U.S. Patent Nos. 5,993,333 and 5,993,352, the contents of which are incorporated herein by reference.

[0034] Image data 48 generated by the camera assembly 44 may be communicated to the virtual reality (VR) computation unit 14 and processed by the VR or image rendering unit 30. The image data 48 may include still photographs or image data as well as video data. The VR rendering unit 30 may include suitable hardware and software for processing the image data and then rendering the image data for display by the display unit 12, as is known in the art. Additionally, the VR rendering unit 30 may combine the image data received from the camera assembly 44 with information associated with the position and orientation of the cameras in the camera assembly and with the position and orientation of the surgeon's head. With this information, the VR rendering unit 30 may generate an output video or image rendering signal and transmit this signal to the display unit 12. That is, the VR rendering unit 30 renders the hand control device position and orientation readings and the surgeon's head position for display on the display unit, such as, for example, on an HMD worn by the surgeon.

[0035] The VR computation unit 14 may also include a virtual reality (VR) camera unit 38 for generating one or more virtual reality (VR) cameras for use or installation in the VR world displayed in the display unit 12. The VR camera unit 38 may generate one or more virtual cameras in the virtual world, which may be used by the system 10 to render images for the head mounted display. In this way, the VR camera always renders the same view as the user wearing the head mounted display sees in the cube map. In one embodiment, a single VR camera may be used, and in another embodiment, separate binocular VR cameras may be used, rendering onto separate binocular cube maps in the display to provide stereoscopic vision. The FOV settings of the VR camera may self-configure with respect to the FOV exposed by the camera assembly 44. In addition to providing a contextual background for the live camera view or image data, the cube map may be used to generate dynamic reflections on virtual objects. With this effect, reflective surfaces on virtual objects capture reflections from the cube map, and these objects appear to the user as if they are actually reflecting their real-world environment.

[0036] The robotic arm 42 may be comprised of multiple mechanically linked actuating sections or portions that form joints that may be constructed and combined for rotational and / or hinged motion to mimic different portions of a human arm, such as, for example, the shoulder, elbow, and wrist regions of the arm. The actuator sections of the robotic arm are constructed to provide, for example, cable-driven rotational motion, but within reasonable rotational limits. The actuator sections are configured to provide maximum torque and speed at a minimum size.

[0037] 2 is a further detailed view of the camera assembly 44 of the robotic subsystem 20 of the present disclosure. The illustrated camera assembly 44 may include cameras 60A and 60B for providing a stereoscopic view of the surgical site. The cameras may include known elements, including lenses and associated optical lenses, image sensors, controllers, and the like. Thus, the camera assembly may include, for example, an autofocus unit 62 for autofocusing the lenses of the cameras 60A, 60B. Although the autofocus unit 62 is shown as a separate unit, it may be included with each camera. The controller 64 may provide control signals for controlling the autofocus unit 62, as well as the cameras 62A, 62B, in response to control signals received from the computing unit 18 and / or the motor unit 40.

[0038] The illustrated camera assembly 44 exhibits two additional characteristics that make the autofocus unit 62 more important than other devices on the market. First, the camera assembly 44 has much more motion during normal operation, resulting in the need to focus on different positions more quickly. Second, the autofocus unit 62 may employ a lens system that utilizes a narrower depth of field than would otherwise be required. Because of this, the autofocus unit 62 may use less expensive lens elements while providing better clarity in the focus region.

[0039] 3 is a general schematic diagram of the robot arm 42 and camera assembly 44 of the robot subsystem 20. The robot arm 42 may include, for example, separate robot arms 42A and 42B. Each of the robot arms 42A, 42B may include an end effector or gripper 46A, 46B, respectively.

[0040] The controller 64 may use or employ position information received from the motor unit 40 and from any sensors associated with the system, such as, for example, from the sensor and tracking units 16, 16A, to calculate or determine the desired focal distance. In one embodiment, the information utilized is the distance from the camera to each end effector portion 46A, 46B of the robotic arm 42A, 42B that is currently in the surgeon's field of view. The controller 64 may also store the focus curve of each camera 60A, 60B that is calibrated in distance space at the factory, and the focus of each camera may be adjusted by the autofocus unit 62 to match where the surgeon is looking using the robotic arm 42 as the minimum position of the intended depth of field. As the system 10 moves the robotic arm 42 or the camera assembly 44, the focus of the cameras 60A, 60B may be adjusted accordingly.

[0041] As shown in FIG. 4, the desired focal length may be determined by determining the system state 110 (e.g., arm position and camera position) and then using a selected weighting algorithm 112 to process the system state data 110. The weighting algorithm technique 112 may be configured to match the surgeon's area of ​​interest with high fidelity and then focus the cameras 40A, 40B at the area by the autofocus unit 62 without direct knowledge of the scene or direct input from the surgeon. The weighting algorithm may determine a desired focal length 114. The desired focal length 114 may be transmitted to the controller 64, which may then utilize a variety of methods, including using a calibrated focus curve 100, to determine a selected focus command 116 for a given desired focal depth. The focus command 116 is transmitted to the autofocus unit 62 to change the physical focal length of the camera. The weighting algorithm and associated processing may occur within the computation unit 18. Alternatively, the computation may occur within the camera controller 64.

[0042] According to an alternative embodiment, the surgical robot system 10 may employ a filter to process the desired focal length 114. For example, as shown in FIG. 5A, the system via the computation unit 18 may employ a weighting algorithm technique 112 to generate the desired focal length 114. The desired focal length 114 may then be passed through an optional filter, such as a low pass filter 118, to reduce large or rapid changes in the focus data. The strength of the low pass filter 118 may be adjusted or varied with the degree of head motion (i.e., the greater the head motion, the weaker the filter strength). The output of the low pass filter 118 may be communicated to the controller 64. Alternatively, the filter may be positioned at the output of the autofocus unit 62, as shown in FIG. 5B. Notably, the filter is not limited to a low pass filter, and the present disclosure contemplates other known filters.

[0043] According to another implementation of the present disclosure, the desired focal length 114 may be varied between the cameras 60A, 60B. By focusing the cameras 60A, 60B at a position away from the desired focal length 114 (e.g., one camera closer and the other farther away), a composite image may be created with a greater depth of field when the images from the cameras are overlaid.

[0044] The system may determine the desired focal length by taking the distance between the end effector portion of the robot arm and the camera and by employing selected weighting values ​​associated with each robot arm. By mathematically combining the weighting values ​​in a selected manner, the controller 64 may then determine the desired focal length 114. The position and orientation of the camera assembly 44 relative to each end effector 46A, 46B may be determined by the camera assembly controller 64 as well as by the control unit 26. The controller 64 then generates control signals that are received by the autofocus unit 62. In response, the autofocus unit 62 generates signals to change, adjust or control the focus or length of the camera 60A in accordance with known techniques. As the distance between the end effector and the camera assembly changes, the autofocus unit 62 may respond by automatically adjusting the focus or length of the camera.

[0045] According to one embodiment, the system state data is processed by a weighting algorithm unit to adjust the relative influence of the inputs on the output of the system 10. For example, as shown in Figures 6A and 6B, system state data 110 is generated by the control unit 26, defining the state of the position (e.g., pose) of the robot arm and camera. The system state data 110 is then fed to a focal length calculator 140 for determining a desired focal length 114. The focal length calculator 140 may form part of the computation unit 18 or may form part of the robot subsystem 20. The focal length calculator 140 may include a data extractor unit 142 for extracting selected types of data and generating processed system state data 144. The processed system state data 144 is communicated to a weighting algorithm unit 146 for applying a weighting algorithm technique to the system state data 144. The weighting algorithm unit 146 generates the desired focal length 114 from the system state data. More specifically, the data extractor unit 142 may employ multiple different distance calculation units for calculating the camera normal distance to each robot arm, including, for example, a left normal distance calculation unit 142A and a right normal distance calculation unit 142B. For example, the system state information 110 may include left end effector position and orientation information, as well as left camera position and orientation information, which are introduced into the left normal distance calculation unit 142A.

[0046] Additionally, the system state information 110 may include information of the position and orientation of the right end effector, as well as information of the position and orientation of the right camera, which are fed into a right normal distance calculation unit 142A. The left normal distance calculation unit 142A calculates left distance data 144A indicative of the distance between the left camera and the left end effector from the input data. Similarly, the right normal distance calculation unit 142B calculates right distance data 144B indicative of the distance between the right camera and the right end effector from the corresponding input data. The distance data 144A, 144B are then fed into a weighting algorithm unit 146 for further processing. The weighting algorithm unit 146 may include a focal length calculation unit that determines the desired focal length 114 using the following formula: Focal length=(W1×Z L +W2×Z R ) / (W1+W2) where W1 and W2 represent selected weighting values ​​associated with the left and right robot arms, respectively, Z1 represents the distance value 144A from the left normal distance calculation unit 142A, and Z2 represents the distance value 144B from the right normal distance calculation unit 142B.

[0047] Thus, according to the weighting algorithm technique, each robot arm is weighted separately, and then the weighted values ​​are normalized by dividing by the sum of those weights so that the calculated desired focal length 114 is within the appropriate range. According to one embodiment of the weighting algorithm technique, both weights W1 and W2 are fixed and equal to 1, so that the weighting algorithm effectively calculates the average value of the two distances. According to another embodiment of the weighting algorithm technique, the weights W1 and W2 of the two robot arms can be varied so that one arm has more influence relative to the other in determining the desired focal length 114. In yet another embodiment of the weighting algorithm technique, the weights of the two robot arms can be a function (i.e., not fixed) based on other system parameters. The system parameters can include, for example, how centered the end effector is within the camera's field of view (FOV), the state of the end effector (i.e., whether the gripper is open, closed, or somewhere in between), the position of the elbow joint, or any other parameter that the system can measure and relay to the focal length calculation device unit 140.

[0048] System parameters may be extracted from the system state data 110 by the data extractor unit 142. An example of this technique is shown, for example, in FIG. 6C. In the illustrated embodiment, the data extractor unit 142 may include an additional computation unit for determining additional system state data 144. The additional system state data 144 is then processed by the weighting algorithm unit 146A. In utilizing how centered the end effector is in the camera's field of view, the data extractor unit 142 uses geometric knowledge of the X and Y components of the end effector position and how it relates to the camera's FOV to calculate the distance of a given end effector to the center of the FOV. In this embodiment, the weight of a given end effector in the weighting algorithm technique is a function of its distance to the center of the FOV. For example, the more centered the end effector is, the stronger the weight for that end effector. If the object is out of view, the weight is reduced. Such a dependency may be desirable because the user is likely to be focusing on the center of the FOV, and therefore an end effector closer to the center may correlate with the user's desire to focus there.

[0049] An example of a field of view of the camera assembly is shown, for example, in Figure 7. The illustrated field of view 150 has a center 152. The robot arm 42 includes a robot arm 42A including an end effector or gripper 46A, and a robot arm 42B with a right end effector or gripper 46B. The system derives from the image data right and left robot arm center distance data R 左 and R 右 can be determined.

[0050] 8 shows a graph 160 that may be used by a control unit of the present disclosure. The graph 160 shows weight values ​​along a Y-axis 162 and distance from the center of the field of view 152 along an X-axis 164. As can be seen, the weighting by the end effector decreases as the distance from the center of the FOV increases. In other embodiments, the relationship between the distance from the center of the FOV and any associated weighting may take other non-linear forms, such as polynomic, logarithmic, inverse exponential, or any other relationship known in the art.

[0051] In yet another embodiment, the weighting of each robot arm 42A, 42B may depend on the state of the gripper at the end effector (e.g., open or closed, or in between). For example, if the user has the gripper closed, this may indicate that the user is performing an action that the user wants to observe with that end effector, and therefore it is desirable for that end effector to be in focus and therefore weighted more. FIG. 9 shows a graph 160 that may be used by the control unit of the present disclosure. The graph 170 shows weight values ​​along a Y-axis 172 and the state of the gripper along an X-axis 174. The state of the gripper may vary between a closed state 176 and an open state 178. As can be seen, the weighting by the end effector decreases as the gripper transitions from a closed state to an open state. In other embodiments, the relationship between the state of the gripper and any associated weighting may take other non-linear forms, such as polynomic, logarithmic, inverse exponential, or any other relationship known in the art.

[0052] In other embodiments, larger weights may be applied to end effectors that are further away, biasing the output towards objects that are further away and making the background more visible than the foreground, effectively creating a second order (or higher) dependency on the depth of the end effector.

[0053] As shown in Figures 10A-10C, in certain situations it is desirable to limit the autofocus adjustment. For example, during suturing or other manipulations, it is desirable to decrease the speed of the focus adjustment as the end effector moves toward the patient. The focus adjustment should be rapid as the end effector of the robotic arm moves away from the patient or target location (e.g., outward), while the adjustment should be slower as the end effector moves toward the patient or target location. This difference in the speed of the autofocus provides stronger autofocus performance and a clearer background image. Small movements, such as discarding a suture, cause minimal changes in the actual set focus position, while larger movements result in more abrupt changes. This behavior can be produced using weights that influence the distant positive movements more than (stronger than) the negative movements. Proportional-integral-derivative (PID) control with a highly weighted I / D term can influence the negative movements in a way that provides less change at lower motion changes. Figure 10A shows an example of such a movement where the focus is slightly anterior to the tissue.

[0054] As shown in FIG. 10B, the tissue is manipulated and it is desirable for the focus to remain close to the target. That is, FIG. 10B shows uncoordinated hand movements or suturing. For example, when the surgeon begins to discard the suture, it is not desirable to immediately change the focus and the focus should remain closer to the tissue than the center position of the arm. Thus, a large filter may be applied to the autofocus calculation during rapid movements. FIG. 10C shows an example of a longer movement resulting in a focus position shift towards the task. In this example, the focus may advance faster the closer the end effectors (graspers) are together. The proximity of the end effectors indicates coordinated movements.

[0055] Still further, the weights W1 and W2 are functions of multiple system parameters and can be convenient multiplicative functions of the individual weights of the multiple parameters. An example of a multiplicative weight is as follows: W1 = wLC × wLG × wLZ where W1 is the total weight of the left end effector, wLC is the weight of the left arm from the distance to the center of the FOV, wLG is the weight from the left gripper open / closed state, and wLZ is the weight from the depth to the left end effector.

[0056] In other embodiments, additional distances may be incorporated into the weighting algorithm technique, such as, for example, elbow distance, background distance, and distance from an autofocus algorithm. An example of a suitable weighting algorithm technique is shown, for example, in FIG. 11. The illustrated focal length calculator unit 140 may include the illustrated data extractor unit 142 and the illustrated weighting algorithm unit 146. In the current illustrated example, the elbow joint of the robot arm is an additional interest point that is weighted. According to one implementation, the elbow joint may have a weight value associated with it that is less than the weight value associated with the end effector. This is desirable because, when the end effector is out of the FOV and only the elbow joint is viewed, the output of the weighting algorithm technique may be dominated by the distance the elbow is away from the camera.

[0057] Additionally, if the user views an end effector portion having an elbow joint in the field of view, the weight value associated with the elbow joint may be significantly less than the weight value associated with the end effector, and therefore the output may be dominated by the distance of the end effector. In other embodiments, the focal length calculator 140 may receive image data from a camera assembly as input and utilize PDAF or other autofocus algorithms to calculate an autofocus distance to bring the image into focus. This may be particularly useful when other weight values ​​drop to zero (or one in some embodiments). In yet other embodiments, the input to the weighting algorithm may include three-dimensional depth information calculated from stereo image data of the camera, calculate the distance to the background of the image, and then use that information as another input in the calculation of the desired focal length 114. This may be useful to bias the desired focal length 114 closer to the background, helping to extend the depth of field by focusing further away.

[0058] The field of view may be determined by the optical elements within the camera. Each camera may also be tested at the factory to ensure that the FOV is within a normal range (e.g., within 60601±15% of the diagonal of the stated value). Furthermore, as the focus changes, the camera's FOV also changes. A correction curve for FOV changes with focus may be calculated based on the lens elements.

[0059] Thus, the focal length calculator unit 140 takes the weight of each position of interest (e.g., elbow joint, gripper, background, etc.) and then multiplies it with the dot product of the position of the object of interest and the pointing vector of the camera. In other embodiments, the system may calculate a spherical range of influence from the camera that provides a weight value based on the position within the spherical range. Regions of the field of view may be assigned to have a larger weight value associated with them, and the weight value may peak at the center of the spherical range. Because the camera does not generate a spherical window, but rather a rectangle, the positions need to be weighted based on the axis position, not a pure dot product. Thus, movement along the X axis of the camera may have a weaker gradient than movement along the Y axis, since the X axis has a larger field of view.

[0060] It should be noted that in some embodiments, the weighting algorithm may take a form other than the normalized linear combination described above. The weighting algorithm may also use any selected type of filter to combine state data with the desired focal length, such as, for example, a Kalman filter or even machine learning techniques, to help determine the desired focal length based on the system state data.

[0061] 12 is a schematic flow chart diagram illustrating steps or methods used by the surgical robotic system 10 to autofocus the camera of the camera assembly 44 based on one or more system parameters. As shown, raw position data of selected components of the surgical robotic system, such as sensors (e.g., magnetometers and IMUs) and other control inputs, are aggregated and stored in the storage unit 24 (Step 70). The position data may be used by the control unit 26 and / or controller 64 to determine the pose or position of each of the robotic arms 42A, 42B and the camera assembly 44 in three-dimensional space (Step 72).

[0062] That is, the controller may calculate the position of the robot arm using either direct or indirect measurement techniques. According to direct measurement techniques, the robot arms 42A, 42B and the cameras 60A, 60B each have an absolute sensor associated with them located at the most distal joint. The absolute sensor may measure the position and orientation of the most distal end of the robot arm in six degrees of freedom (e.g., X, Y, Z, yaw, pitch, and roll) relative to a common origin in space. This sensor allows the controller to easily determine the relative position and orientation of the robot arm and the camera. Examples of absolute sensors suitable for use in the present disclosure may include alternating magnetic field tracking (such as those incorporated in Polhemus' proprietary technology), optical tracking methods, and IMUs, as well as others known in the art.

[0063] According to indirect measurement techniques, the pose and position of the robot arm and camera are estimated from other parameters known in the system, such as, for example, by the individual angles of each joint in the arm. The angles of each joint are then input into a forward kinematic model of the robot arm to calculate the position and orientation of the distal most end of the robot or the camera in six degrees of freedom. The position of the robot arm or camera can be measured in a number of ways, all dependent on knowing the state of the system. According to one implementation, for example, as shown in FIG. 13, in a cable-driven surgical robotic system, the controller determines the state of the cables and then uses that information to estimate the position of each joint in the robot arm. As is known, each cable used in a robot arm changes a particular joint position and therefore has a calculable effect on the position of the robot arm and camera assembly.

[0064] For example, the system may employ suitable position sensors, such as Hall effect sensors or potentiometers, positioned on the output of the motor unit 40, which controls the drive cables for a given joint of the robot arm or camera. In this example, the ratio of the motor's motion to the robot joint's motion is known and fixed based on the geometry (e.g., radius) of any motor and drive pulleys used in the robot arm's joints. Thus, the system may determine that if the motor rotates X degrees, the robot joint moves Y degrees, where Y=X×R, where R is the ratio mentioned above. Thus, the control unit 26 may determine the motor positions 120 for each joint of the robot arm. The control unit 26 also determines the joint positions 124 for all joints of the robot arm by using a selected joint ratio 122, which may be defined as Rj / Rm, where Rj is the radius of the joint and Rm is the radius of the motor. The calculated joint positions 124 and the associated motions of the joints can then be input into a forward kinematics model 126 to calculate the position and orientation 128 of the distal-most part of the robot arm. This method works best when the cables are assumed to be stiff and have low friction acting on them.

[0065] According to one embodiment, as shown in FIG. 14, the angular position of the robot joints may be measured via sensors attached directly to the joints. This may include Hall effect sensor arrays, potentiometers, or other methods known in the art. This type of joint measurement may be useful for cable-driven systems where the cable is not very stiff or where friction acting on the cable is high. In one example, a Hall effect sensor array embedded in a joint of the robot arm measures the rotational position of a magnet embedded on the opposite side of the joint, thereby allowing a direct measurement of the angular position of that joint. Thus, the system may determine the joint position 130 of the robot arm via the sensors. The joint position 130 may be introduced into a forward kinematic model 126 that processes the joint position data to calculate or determine the robot position and orientation 128 of the most distal portion of the robot arm.

[0066] In all indirect measurement methods and techniques, the forward kinematic model 126 of the arm robot and camera assembly refers to a common origin and coordinate frame. This can be done by utilizing knowledge of the geometric relationships or quantities between the robot arm and the camera assembly, which can be adopted as additional parameters in the forward kinematic model 126. This is not required for direct measurement techniques, as they already inherently measure back to a common origin.

[0067] Regardless of which technique is employed to determine the position and orientation of the robot, once the position and orientation of the end effector of the robot arm and the position and orientation of the camera are calculated, the distance between the camera and each individual end effector can be calculated. Additionally, indirect and direct measurement techniques may be combined. For example, the camera assembly may be directly measured and the robot arm may be indirectly measured, or vice versa. In other embodiments, measurement techniques are combined within one system. For example, a robot may utilize Hall effect sensors for one joint and motor cable measurement techniques for another joint on the same robot.

[0068] Once the pose or position of each of the robotic arms 42A, 42B and the camera assembly 44 has been determined by the system 10, the distance of the camera assembly 44 from both of the robotic arm's end effectors 46A, 46B may be determined or calculated by the control unit 26. (Step 74) Once the positions and orientations of the robotic arm's end effectors and the positions and orientations of the cameras have been calculated, the distance between the camera and each individual end effector may be calculated. This may be calculated in different ways, including employing known mathematical relationships such as Pythagoras' Theorem (3D), quaternion rotations and translations, vector and dot products, and many other methods known in the art. When using Pythagoras' Theorem, the theorem determines the X, Y, Z coordinates and then calculates the distance between the points in the following manner: Darm_to_Camera=sqrt[(XEE-XC)2+(YEE-YC)2+(ZEE-ZC)2], where XEE, YEE, and ZEE are the coordinates of the distal end effector of the robot arm, and XC, YC, and ZC are the coordinates of the camera. The vector and dot product method utilizes the dot product of the line of sight of the camera in vector form, which can be determined directly from the camera's orientation and geometry at a given state, and a vector starting at the camera and ending at the end effector. This yields the distance the end effector is from the camera perpendicular to the camera's line of sight. This same distance can be calculated using quaternion rotations and translations to place the end effector's position and orientation (X, Y, Z, yaw, pitch, roll) in the same coordinate frame as the camera. Once this operation is performed, the Z term (normal to the camera) of the end effector in this new coordinate frame is the distance the end effector is from the camera perpendicular to the camera's line of sight.

[0069] The field of view (FOV) is also determined by the optical elements within the camera assembly. Each camera 60A, 60B in the camera assembly 44 may be factory tested to ensure that the FOV is within an acceptable range. Furthermore, since the FOV changes in direct correlation with changes in focus, a correction curve for changes in FOV with respect to focus may be calculated based on the optical lens stack used by the camera.

[0070] The controller 64 may then generate a control signal that is received by the autofocus unit 62, which may automatically adjust the focus of the cameras 42A, 42B. (Step 84) Additionally, the system 10 may allow the surgeon or user to manually adjust the focus of the cameras. (Step 80) Additionally, the focus curves of the cameras of the camera assembly may be pre-stored in the storage element 24. (Step 82) The focus curves are an example of how the focus controller converts the desired focal length 114 into commands that the system can use to achieve that desired focal length. Thus, the desired focal length 114 may be implemented via a look-up table or suitable function of the position of the focus element relative to the focal length. It is known that the lenses for each camera need to be calibrated to allow the focus to be adjusted as a function of the object distance.

[0071] 15 is an example of one type of focus curve 90 that may be associated with the cameras 60A, 60B of the surgical robotic system 10. As illustrated in the focus curve 90, as the distance amplitude correction (DAC) value changes, the optimal distance to focus also changes. As noted, larger changes are required for targets closer to the camera. This is due to the design of the voice coil module (VCM), where the current commands have an inverse square relationship to the lens movement and the changes required by the VCM become smaller the further away the distance.

[0072] 16 is a curve illustrating the defined focal region of the camera of the camera assembly 44. The illustrated focal curve 100 includes multiple targets at different distances superimposed on one another. The peak of the illustrated curve 100 over all distances is shown above. Note that just because the peak is larger the further away the distance is, it does not mean that the image is sharper. Rather, it is the width of the curve for that number of distances that is important.

[0073] The focal curves of the cameras of the camera assembly 44 can be generated and pre-stored in distance space at the factory and applied, and the focus of the cameras can be adjusted to where the surgeon is looking using the robotic arm as the minimum position on the depth of field of the intended scene. The term distance space as used herein is intended to describe physical distances in space.

[0074] The cameras 60A, 60B of the camera assembly 44 may be calibrated and configured at the factory prior to being employed in the surgical robotic system 10 of the present disclosure. The camera assembly 44 of the present disclosure is configured to focus on the surgeon's surgical target (e.g., surgical site) while maintaining the maximum range of motion possible for the robotic arm. Calibration of the camera assembly 44 may occur at the factory and may include selected testing to determine the resolution of the cameras 60A, 60B. Calibration of the cameras may occur after final assembly due to the placement of the cameras and the need to register the camera data in selected memory.

[0075] The output data of each camera may be accumulated during calibration and stored in a look-up table. The look-up table may be associated, for example, with each camera and associated lens assembly. The process of calibrating the cameras is as follows: The cameras may be attached to a motor unit and locked to a selected support fixture that controls the orientation and position of the camera and associated sensor. A single axis stage may be used to vary the distance of the target from the camera from about 5 cm to about 20 cm, and the camera is calibrated at all distances within this range. A 10 mm spacing distance between the cameras in the camera assembly may be used and adjusted. The calibration target is preferably a calibrated resolution target with sharp edges. Each position is swept with various VCM currents to obtain the highest resolution score as determined by the variance of the Laplacian of the central region of the image. The region of interest from the central 50% to the central 10% may be performed in a linear fashion from a distance of about 50 mm to about 200 mm. The output data of the calibration may be used to form a focus curve 90.

Claims

1. 1. A surgical robotic system, comprising: a plurality of robotic arms, each robotic arm having an end effector at a distal end thereof; a camera assembly including at least two cameras and an autofocus unit configured to automatically focus a lens of each of the at least two cameras; a robot subsystem including: a sensor unit in communication with the robotic subsystem; a controller in communication with the robot subsystem and configured to calculate a desired focal length of the camera assembly based on state information received from the sensor unit, the state information including position and orientation information of the end effector of each of the plurality of robot arms; A surgical robot system comprising:

2. The surgical robot system of claim 1, wherein the autofocus unit is configured to autofocus the lens of each of the at least two cameras based on the desired focal length.

3. The surgical robot system of claim 1 , wherein the status information further includes a distance from each camera to each end effector of the plurality of robotic arms that is within a surgeon's field of view.

4. The surgical robot system of claim 1 , wherein the state information further includes position and orientation information for each camera.

5. The surgical robot system of claim 1 , wherein based on the calculated desired focal length, the controller is configured to determine a focus command according to a particular focal depth.

6. 6. The surgical robotic system of claim 5, wherein the controller is configured to send the focus command to the autofocus unit, and in response, the autofocus unit is configured to adjust a focus of each camera to focus the lens of each camera.

7. The surgical robotic system of claim 5 , wherein the controller is configured to filter the desired focal length to reduce rapid changes in focus data.

8. The surgical robotic system of claim 7 , wherein a strength of the filter for filtering the desired focal length is varied based on a degree of head movement of the surgeon.

9. The surgical robotic system of claim 1 , wherein a different desired focal length is calculated for each of the at least two cameras.

10. The surgical robotic system of claim 1 , wherein the desired focal length is further calculated using a weighting algorithm.

11. The surgical robotic system of claim 10 , wherein each robotic arm is weighted differently in the weighting algorithm.

12. The surgical robotic system of claim 11 , wherein the weight of each robotic arm is a function based on system parameters.

13. The surgical robotic system of claim 12 , wherein each robotic arm includes multiple joints.

14. The surgical robot system of claim 13 , wherein the plurality of joints includes a shoulder joint, an elbow joint, and a wrist joint.

15. The surgical robotic system of claim 14 , wherein the system parameters include a distance from a center of each end effector within the field of view of each camera, a state of each end effector, and a position of the elbow joint.

16. The surgical robotic system of claim 1 , wherein the focusing speed is increased as each end effector moves outward from the target position.

17. The surgical robotic system of claim 1 , wherein the focusing speed is decreased as each end effector moves towards the target location.

18. 1. A robotic subsystem comprising: a plurality of robotic arms, each having an end effector at a distal end thereof; 1. A camera assembly comprising: At least two cameras, A control device; an autofocus unit configured to autofocus a lens of each of the at least two cameras; Equipped with the controller is configured to calculate a desired focal length based on state information received from a sensor unit, the state information including position and orientation information of the end effector of each of the plurality of robot arms; The autofocus unit is configured to autofocus the lens of each of the at least two cameras based on the desired focal length.

19. The robotic subsystem of claim 18 , wherein a focusing speed is increased as the robotic arm moves outward from a target location and is decreased as the robotic arm moves inward toward the target location.

20. 20. The robotic subsystem of claim 18, wherein the status information further includes at least one of a distance from each camera to each end effector of the plurality of robotic arms within a surgeon's field of view, and position and orientation information of the at least two cameras.

Images