System and method for autofocusing camera assemblies of surgical robot systems

Abstract

The surgical robotic system includes a sensor unit, a controller, and a robot subsystem. The robot subsystem is in communication with the sensor unit and the controller. The robot subsystem further includes a plurality of robot arms, each having an end effector at a distal end thereof. The robot subsystem also includes a camera assembly having at least two cameras and an autofocus unit for autofocusing a lens of each of the cameras.

Description

Technical Field

[0001] Cross - Reference to Related Applications This disclosure claims priority to U.S. Provisional Patent Application No. 63 / 153,128, filed Feb. 24, 2021, and U.S. Provisional Patent Application No. 63 / 176,634, filed Apr. 19, 2021, the entire contents of which are incorporated herein by reference.

[0002] This disclosure is directed to minimally invasive surgical devices and related methods, and more particularly to robotic surgical systems that are insertable into a patient for performing selected surgical procedures therein.

Background Art

[0003] The field of minimally invasive surgery has expanded rapidly since its inception in the early 1990s. Although minimally invasive surgery significantly improves patient outcomes, this improvement comes at the expense of the surgeon's accurate and facile surgical capabilities. During laparoscopic surgery, the surgeon must insert laparoscopic instruments through small incisions in the patient's abdominal wall. Due to the nature of inserting the instruments through the abdominal wall, the movement of laparoscopic instruments is restricted because they cannot move laterally without damaging the abdominal wall. Standard laparoscopic instruments are limited to four axes of movement. These four axes of movement are the movement of the instrument inside and outside the trocar (axis 1), the rotation of the instrument within the trocar (axis 2), and the angular movement of the trocar in two planes (axes 3 and 4) while maintaining the pivot point where the trocar enters the abdomen. Most minimally invasive surgeries have been performed using only these four degrees of freedom of movement for over two decades.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

[0005] Existing robotic surgical devices have attempted to address many of these problems. Some existing robotic surgical devices replicate non-robot laparoscopic surgery by having additional degrees of freedom at the ends of the instruments. However, despite many costly modifications to surgical procedures, existing robotic surgical devices have failed to provide improved patient outcomes for the majority of procedures in which they are used. Furthermore, existing robotic devices increase the separation generated between the surgeon and the surgical end effector. This increased separation leads to surgeon misinterpretations of movements and injuries caused by the forces applied by the robotic device. Because the degrees of freedom of many existing robotic devices are unfamiliar to human operators, surgeons must undergo extensive training on robotic simulators before operating on patients to minimize the possibility of accidental injury.

[0006] A typical robotic system includes one or more robotic arms and one or more associated cameras. To control existing robotic systems, the surgeon sits at a console and controls the manipulators with both hands and feet, thereby controlling the cameras and robotic arms. Furthermore, the cameras may remain in a semi-fixed position and are moved by combined hand and foot movements from the surgeon. These semi-fixed cameras, with their limited field of view, consequently make visualization of the surgical field difficult.

[0007] Conventional surgical robotic systems offer several methods for focusing the camera on the intended surgical site. Typically, and conventionally, surgeons must adjust the focus either using a manual dial or based solely on the image data received from the camera's image sensor, or based on the system's autofocus features. Typical autofocus mechanisms in the medical field can also employ phase-detection autofocus, time-of-flight (light reflection), or some other light-based estimation method.

[0008] The drawback of these conventional systems is that they require surgeons to interrupt surgical procedures and manually change the camera's focus. This distracts the surgeon during the surgical procedure. In systems employing autofocus technology, the camera's focus or field of view often does not align with or cover the actual part of the surgical site that the surgeon needs to see and which requires a greater depth of field. Conventional cameras employing a greater depth of field require more light, which results in a lower overall resolution. [Means for solving the problem]

[0009] In the robotic surgical system of the present disclosure, the system utilizes the position of the camera in the camera assembly and the position of the gripping device or end effector of the robotic arm to determine the camera's field of view and focal length or focus. By using the positional information of the end effector relative to the camera, the system of the present disclosure can determine the camera's focal length and focus, while the system allows the surgeon to view important or desired portions of the surgical site in conjunction with the image data from the camera, without having to rely solely on the image data from the camera. Thus, the system has a more accurate focus with fewer false positives and the ability to always focus on the desired field of view.

[0010] According to one embodiment, the present disclosure provides a surgical robotic system comprising a sensor unit, a control unit, and a robot subsystem. The robot subsystem communicates with the sensor unit and the control unit. Furthermore, the robot subsystem includes a plurality of robotic arms, each having an end effector at its distal end, and a camera assembly having at least two cameras and an autofocus unit configured to autofocus each lens of the at least two cameras. The control unit may be configured to calculate a desired focal length based on camera and robotic arm statement information received from the sensor unit. In response, the autofocus unit may be configured to autofocus each lens of the at least two cameras based on the desired focal length.

[0011] State information may include the distance from each camera to each end effector of the robotic arm within the surgeon's field of view. Furthermore, state information may include positional and orientation information for each camera and each end effector. Based on the calculated desired focal length, the control unit may be configured to determine a focus command according to a specific depth of field and transmit the focus command to the autofocus unit. In response, the autofocus unit may be configured to adjust the physical focal length of each camera to focus each lens of the camera.

[0012] Furthermore, the control device may be configured to filter for a desired focal length to reduce abrupt changes in focus data. The strength of the filter used may vary based on the degree of the surgeon's head movement. Different desired focal lengths may also be calculated for each camera. The desired focal lengths may be further calculated using a weighting algorithm. Each robotic arm may be weighted differently in the weighting algorithm. The weight of each robotic arm is a function of system parameters.

[0013] In particular, each robotic arm may include multiple joints. These joints may include shoulder joints, elbow joints, and wrist joints. Therefore, system parameters may include the distance from the center of each end effector within the field of view of each camera, the state of each end effector, and the position of the elbow joint. In one embodiment, the focusing speed may increase as each end effector moves outward from the target position and decrease as each end effector moves toward the target position.

[0014] In another embodiment, the disclosure provides a robot subsystem comprising a plurality of robot arms, each having an end effector at its distal end, and a camera assembly. The camera assembly may include at least two cameras, a control unit, and an autofocus unit configured to autofocus each lens of the cameras. The control unit may be configured to calculate a desired focal length based on state information of at least two cameras and the plurality of robot arms received from a sensor unit. In response, the autofocus unit may be configured to autofocus each lens of the at least two cameras based on the desired focal length.

[0015] The focusing speed increases as the robotic arm moves outward from the target position and decreases as the robotic arm moves inward toward the target position. Furthermore, the state information includes, from each camera, the distance to each end effector of the multiple robotic arms within the surgeon's field of view, as well as at least one of the position and orientation information of at least two cameras and each end effector of the multiple robotic arms.

[0016] These and other features and advantages of this disclosure will be better understood by referring to the following detailed description in conjunction with the accompanying drawings, where similar reference numerals refer to similar elements throughout the various figures. The drawings illustrate the principles of this disclosure and show relative dimensions, although not to exact scale. [Brief explanation of the drawing]

[0017] [Figure 1] Figure 1 is a schematic diagram of the surgical robot system of the present disclosure. [Figure 2] Figure 2 is a schematic diagram of the camera assembly of the surgical robot system of Figure 1 according to the teachings of the present disclosure. [Figure 3] Figure 3 is a schematic diagram of the robot subsystem of Figure 1 according to the teachings of the present disclosure. [Figure 4] Figure 4 is a schematic diagram showing the processing of system state data by the weighting algorithm unit used by the system of the present disclosure. [Figure 5A] Figures 5A and 5B are schematic diagrams showing the processing of system state data by the weighting algorithm unit used by the surgical robot system according to the second embodiment of the present disclosure. [Figure 5B] The same as above. [Figure 6A] Figure 6A is a schematic diagram showing the processing of system state data by the focal length calculation device unit used by the surgical robot system according to the teachings of the present disclosure. [Figure 6B] Figure 6B is a schematic diagram showing the processing of system state data by the second embodiment of the focal length calculation unit used by the surgical robot system according to the teachings of the present disclosure. [Figure 6C] Figure 6C is a schematic diagram showing the processing of system state data by the third embodiment of the focal length calculation unit used by the surgical robot system according to the teachings of the present disclosure. [Figure 7] Figure 7 is a schematic diagram of the field of view of the camera assembly unit used by the surgical robot system according to the teachings of the present disclosure. [Figure 8] Figure 8 is a graph display of the weight graph used by the surgical robot system of the present disclosure. [Figure 9] Figure 9 is a graph display of the second embodiment of the weight graph used by the surgical robot system of the present disclosure. [Figure 10A]Figures 10A to 10C are schematic diagrams of the field of view of a camera assembly unit for limited focus adjustment as taught in this disclosure. [Figure 10B] Same as above. [Figure 10C] Same as above. [Figure 11] Figure 11 is a schematic diagram showing the processing of system state data according to a fourth embodiment of a focal length calculation unit used in a surgical robot system, as taught in this disclosure. [Figure 12] Figure 12 is a schematic flowchart illustrating a method for autofocusing the camera of a camera assembly according to the teachings of this disclosure. [Figure 13] Figures 13 and 14 are schematic diagrams illustrating the processing of system state data, such as motor and joint data, by the forward kinematics unit used in the surgical robotic system of this disclosure. [Figure 14] Same as above. [Figure 15] Figure 15 shows an exemplary focus curve that may be pre-stored and utilized by the control system of the surgical robot system of this disclosure. [Figure 16] Figure 16 shows an exemplary focal curve with multiple targets at different distances superimposed on each other. [Modes for carrying out the invention]

[0018] The following description includes numerous specific details regarding the systems and methods of this disclosure, as well as the environments in which such systems and methods may operate, in order to provide a thorough understanding of the disclosed subject matter. However, it will be apparent to those skilled in the art that the disclosed subject matter may be implemented without such specific details, and that certain features well known in the art are not described in detail in order to avoid complexity and to enhance clarity of the subject matter of this disclosure. Furthermore, it will be understood that any embodiments provided below are merely illustrative and not to be construed as limiting, and that the inventors assume that other systems, apparatus, and / or methods may be used to implement or complete the teachings of the invention and will be considered within the scope of the invention.

[0019] While exemplary embodiments are described as using multiple units to carry out exemplary processes, it is understood that exemplary processes may also be carried out by one or more modules. Furthermore, it is understood that the term control device / control unit refers to a hardware device including memory and a processor, which is specifically programmed to carry out the processes described herein. The memory is configured to store modules, and the processor is specifically configured to run said modules to carry out one or more processes, which are further described below.

[0020] Furthermore, the control logic of the present disclosure may be embodied as a non-temporary computer-readable medium on a computer-readable medium containing executable program instructions executed by a processor, control device / control unit, or similar. Examples of computer-readable media include, but are not limited to, ROM, RAM, compact disk (CD)-ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. Computer-readable recording media may also be distributed in a network-connected computer system so that the computer-readable medium is stored and executed in a distributed manner, for example by a telematics server or a control device area network (CAN).

[0021] The terms used herein are for the purpose of describing only specific embodiments and are not intended to limit the disclosure. The singular forms “a,” “an,” and “the” as used herein are intended to include plural forms unless the context explicitly suggests otherwise. The terms “comprises” and / or “comprising,” as used herein, specify the presence of the described features, integers, steps, actions, elements, and / or components, but it will be further understood that this does not preclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof. The terms “and / or” as used herein include any and all combinations of one or more of the related enumerated items.

[0022] Unless otherwise explicitly stated or evident from the context, the term “about” as used herein is understood to mean within the normal tolerances in the art, for example, within two standard deviations of the mean. “About” may be understood to mean within 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, 0.05%, or 0.01% of the stated value. Unless otherwise evident from the context, all numerical values ​​provided herein are modified by the term “about.”

[0023] While the systems and methods of this disclosure may be designed for use with one or more surgical robotic systems used as part of a virtual reality surgical system, the robotic systems of this disclosure may be used in connection with any type of surgical system, including, for example, robotic surgical systems, linear stick surgical systems, and laparoscopic systems. Furthermore, the systems of this disclosure may be used in other non-surgical systems where a user requires access to a vast amount of information while controlling a device or instrument.

[0024] The systems and methods disclosed herein may be incorporated into and used in conjunction with, for example, the robotic surgical apparatus and related systems disclosed in Patent Document 1 and PCT Patent Application No. PCT / US2020 / 39203, and / or the camera system disclosed in Patent Document 2, and all the content and teachings of the aforementioned patents, patent applications and publications are incorporated herein by reference. Surgical robotic systems forming part of this disclosure include surgical systems having a portable surgical robot subsystem comprising a user workstation, a robot-assisted system (RSS), a motor unit, and one or more robotic arms and one or more camera assemblies. The portable robotic arms and camera assemblies may form part of a single support axis robotic system or part of a split-arm architecture robotic system.

[0025] The robotic arm may have joints or regions that can be associated with movements associated with the user's shoulder, elbow, wrist, and fingers, for example, as shown in Figure 3, in order to provide human-like motion. For example, the robotic elbow joint may follow the position and orientation of a human elbow, and the robotic wrist joint may follow the position and orientation of a human wrist. The robotic arm may also have an associated end region, which may end with an end effector or gripper that follows the movement of one or more of the user's fingers, such as the index finger when the user is gripping the index finger and thumb together. The robotic shoulder may be fixed in place while the robotic arm follows the movement of the user's arm. In one embodiment, the position and orientation of the user's torso is subtracted from the position and orientation of the user's arm. This subtraction allows the user to move their torso without the robotic arm moving.

[0026] Figure 1 is a schematic block diagram of a surgical robot system 10 according to the teachings of the present invention. The system 10 includes a display device or unit 12, a virtual reality (VR) computing unit 14, a sensing and tracking unit 16, a computing unit 18, and a robot subsystem 20. The display unit 12 may be any selected type of display for displaying information, images, or videos generated by the VR computing unit 14, the computing unit 18, and / or the robot subsystem 20. The display unit 12 may include, or form part of, a head-mounted display (HMD), a screen or display, a three-dimensional (3D) screen, etc. The display unit may also include an optional sensor and tracking unit 16A, such as those found in commercially available head-mounted displays. The sensing and tracking units 16 and 16A may include one or more sensors or detectors coupled to a user of the system, such as a nurse or surgeon. Sensors may be coupled to the user's arm, and if a head-mounted display is not used, additional sensors may also be coupled to the user's head and / or neck area. The sensors in this configuration are represented by the sensor and tracking unit 16. When the user uses a head-mounted display, eye, head and / or neck sensors and associated tracking technology may be incorporated into or used within the device and thus may form part of an optional sensor and tracking unit 16A.

[0027] Sensors of the sensor and tracking unit 16 coupled to the surgeon's arm may preferably be coupled to selected areas of the arm, such as the shoulder area, elbow area, wrist or hand area, and preferably the fingers. The sensors generate position data indicating the location of the user's selected portion. The sensing and tracking unit 16 and / or 16A may be used to control the operation of the camera assembly 44 and robot arm 42 of the robot subsystem 20. The position data 34 generated by the sensors and tracking unit 16 may be transmitted to the computing unit 18 for processing by the processor 22. The computing unit 20 may be configured to determine or calculate the position and / or orientation of each portion of the surgeon's arm from the position data and transmit this data to the robot subsystem 20.

[0028] According to an alternative embodiment, the sensing and tracking unit 16 may use sensors coupled to the surgeon's torso or any other body part. Furthermore, in addition to sensors, the sensing and tracking unit 16 may use an inertial momentum unit (IMU) having, for example, an accelerometer, a gyroscope, a magnetometer, and an action processor. The addition of a magnetometer is a standard practice in the art because the magnetic orientation can reduce sensor drift around the vertical axis. Alternative embodiments also include sensors placed within surgical materials such as gloves, surgical scrubs, or surgical gowns. The sensors may be reusable or disposable. Furthermore, the sensors may be located outside the user, such as in a fixed location in a room, such as an operating room. The external sensors may be processed by a computing unit and thus configured to generate external data 36 that can be used by the system 10. According to another embodiment, if the display unit 12 is a head-mounted display using the associated sensors and tracking unit 16A, the device generates tracking and position data 34A that are received and processed by the VR computing unit 14. Furthermore, the sensor and tracking unit 16 may include a hand control device, if desired.

[0029] In embodiments where the display is an HMD, the display unit 12 may be a virtual reality head-mounted display such as, for example, an Oculus Rift, Varjo VR-1, or HTC Vive Pro Eye. The HMD may provide the user with a display coupled to or mounted on the user's head, lenses that enable focused viewing of the display, and sensors and / or tracking systems 16A that provide tracking of the display's position and orientation. The position and orientation sensor system may include, for example, an accelerometer, gyroscope, magnetometer, motion processor, infrared tracking, eye tracking, computer vision, alternating magnetic field emission and sensing, and any other method of tracking at least one of position and orientation, or any combination thereof. As is well known, the HMD may provide image data to both eyes of the surgeon from a camera assembly 44. To maintain a virtual reality experience for the surgeon, the sensor system may track the position and orientation of the surgeon's head and then relay the data to a VR computing unit 14, and optionally to a computing unit 18. The computing unit 18 can further adjust the panning and tilt of the robot's camera assembly 44 to follow the user's head movements.

[0030] For example, when associated with an HMD, such as being associated with a display unit 12 and / or a tracking unit 16A, sensor or position data 34A generated by the sensor can be transmitted to the computing unit 18, either directly or via the VR computing unit 14. Similarly, tracking and position data 34 generated by other sensors in the system, such as a sensing and tracking unit 16 that may be associated with the user's arms and hands, can be transmitted to the computing unit 18. The tracking and position data 34, 34A can be processed by the processor 22 and stored, for example, in a storage unit 24. The tracking and position data 34, 34A can also be used by a control unit 26, which can generate control signals in response to control the operation of one or more parts of the robot subsystem 20. The robot subsystem 20 may include a user workstation, a robot-assisted system (RSS), a motor unit 40, and a portable surgical robot including one or more robotic arms 42 and one or more camera assemblies 44. The portable robotic arm and camera assembly may form part of a single support axis robotic system as disclosed and described in Patent Document 1, or part of a segmented arm (SA) architecture robotic system as disclosed and described in PCT Patent Application PCT / US20 / 39203.

[0031] Control signals generated by the control unit 26 may be received by the motor unit 40 of the robot subsystem 20. The motor unit 40 may include a series of servo motors configured to drive the robot arm 42 and the camera assembly 44 separately. The robot arm 42 may be controlled to follow the reduced movement or motion of a surgeon's arm, as sensed by associated sensors. The robot arm 42 may have parts or regions that can be associated with the user's shoulder, elbow, and wrist joints, as well as the fingers. For example, the robot elbow joint may follow the position and orientation of a human elbow, and the robot wrist joint may follow the position and orientation of a human wrist. The robot arm 42 may also have an associated end region that may terminate with an end effector that follows the motion of one or more of the user's fingers, such as the index finger when the user is pinching their index finger and thumb together. While the robot arm follows the user's arm movements, the robot's shoulder is fixed in place. In one embodiment, the position and orientation of the user's torso are subtracted from the position and orientation of the user's arms. This subtraction allows the user to move their torso without the robotic arm moving.

[0032] The robotic camera assembly 44 is configured to provide the surgeon with image data 48, such as a live video feed of the surgery or surgical site, as well as to enable the surgeon to operate and control the cameras that form part of the camera assembly 44. The camera assembly 44 preferably includes a pair of cameras 60A, 60B, whose optical axes are spaced axially apart by a selected distance, known as the inter-camera distance, to provide a stereoscopic view or image of the surgical site. The surgeon can control the operation of the cameras 60A, 60B either by the operation of a head-mounted display, via sensors coupled to the surgeon's head, or by using a hand-mounted control device or sensor that tracks the movement of the user's head or arm, thereby enabling the surgeon to intuitively and naturally obtain a desired view of the surgical site. The cameras are, as is well known, movable in multiple directions, including, for example, yaw, pitch, and roll directions. The components of the stereoscopic camera may be configured to provide a natural and comfortable user experience. In some embodiments, the inter-axis distance between the cameras may be modified to adjust the depth of the surgical site as perceived by the user.

[0033] According to one embodiment, the camera assembly 44 may be actuated by the head movement of a surgeon. For example, if, during surgery, the surgeon wishes to view an object located above the current field of view (FOV), the surgeon looks upward, thereby rotating the stereoscopic camera upward around the pitch axis from the user's viewpoint. Images or video data 48 generated by the camera assembly 44 may be displayed on a display unit 12. If the display unit 12 is a head-mounted display, the display may include an integrated tracking and sensor system 16A that acquires raw orientation data of the HMD in the yaw, pitch, and roll directions, as well as position data of the HMD in Cartesian space (x,y,z). However, alternative tracking systems may be used to provide supplemental position and orientation tracking data for the display in place of, or in addition to, the HMD's integrated tracking system. Examples of camera assemblies suitable for use in this disclosure include the camera assemblies disclosed in Patent Documents 1 and 2, the contents of which are incorporated herein by reference.

[0034] Image data 48 generated by the camera assembly 44 may be transmitted to a virtual reality (VR) computing unit 14 and processed by a VR or image rendering unit 30. The image data 48 may include still photographs or image data as well as video data. The VR rendering unit 30 may include suitable hardware and software for processing the image data and then rendering the image data for display by the display unit 12, as is well known in the art. Furthermore, the VR rendering unit 30 may combine the image data received from the camera assembly 44 with information associated with the position and orientation of the camera in the camera assembly, as well as information associated with the position and orientation of the surgeon's head. With this information, the VR rendering unit 30 may generate an output video or image rendering signal, which may be transmitted to the display unit 12. That is, the VR rendering unit 30 may render the readings of the position and orientation of the hand control device and the position of the surgeon's head for display on a display unit, for example, in an HMD worn by the surgeon.

[0035] The VR computing unit 14 may also include a virtual reality (VR) camera unit 38 for generating one or more virtual reality (VR) cameras for use or installation in a VR world displayed within the display unit 12. The VR camera unit 38 may generate one or more virtual cameras in the virtual world, which may be used by the system 10 to render images for a head-mounted display. This ensures that the VR cameras always render the same view as the user wearing the head-mounted display sees on the cubemap. In one embodiment, a single VR camera may be used, and in another embodiment, separate binocular VR cameras may be used, rendering on separate binocular cubemaps within the display to provide stereoscopic vision. The FOV setting of the VR cameras may be self-configured relative to the FOV exposed by the camera assembly 44. In addition to providing a contextual background for live camera views or image data, cubemaps may be used to generate dynamic reflections on virtual objects. This effect causes reflective surfaces on virtual objects to capture reflections from the cubemap, making these objects appear to the user as if they were actually reflecting the real-world environment.

[0036] The robotic arm 42 may consist of multiple mechanically connected actuating sections or parts that form joints, which can be constructed and combined for rotational and / or hinged motion to mimic different parts of a human arm, such as the shoulder, elbow, and wrist regions. The actuator sections of the robotic arm may, for example, provide cable-driven rotational motion, but are constructed to remain within reasonable rotational limits. The actuator sections are configured to provide maximum torque and speed with minimum size.

[0037] Figure 2 is a further detail view of the camera assembly 44 of the robot subsystem 20 of this disclosure. The illustrated camera assembly 44 may include cameras 60A and 60B for providing stereoscopic views of the surgical site. The cameras may include known elements, such as lenses and associated optical lenses, image sensors, and control devices. Thus, the camera assembly may include, for example, an autofocus unit 62 for autofocusing the lenses of cameras 60A and 60B. The autofocus unit 62 is shown as a separate unit but may be included in each camera. The control device 64 may provide control signals to control the autofocus unit 62, as well as cameras 62A and 62B, in response to control signals received from the computing unit 18 and / or the motor unit 40.

[0038] The illustrated camera assembly 44 exhibits two additional characteristics that make the autofocus unit 62 more important than other commercially available devices. First, the camera assembly 44 has far more movement in normal operation, resulting in the need to focus more quickly on different positions. Second, the autofocus unit 62 can employ a lens system that utilizes a narrower depth of field than otherwise required. As a result, the autofocus unit 62 can use less expensive lens elements while providing better clarity in the focal region.

[0039] Figure 3 is a general schematic diagram of the robot arm 42 and camera assembly 44 of the robot subsystem 20. The robot arm 42 may include, for example, separate robot arms 42A and 42B. Each of the robot arms 42A and 42B may include an end effector or gripping device 46A and 46B, respectively.

[0040] The control unit 64 may use or employ positional information received from the motor unit 40 and any sensors associated with the system, such as from the sensor and tracking units 16, 16A, to calculate or determine a desired focal length. In one embodiment, the information utilized is the distance from the camera to the respective end-effector portions 46A, 46B of the robotic arms 42A, 42B, which are currently within the surgeon's field of view. The control unit 64 may also store the focal curves of each camera 60A, 60B, which are calibrated in distance space at the factory, and the focus of each camera may be adjusted by the autofocus unit 62 to match the location where the surgeon is viewing using the robotic arm 42 as the minimum position of the intended field of view depth. As the system 10 moves the robotic arm 42 or the camera assembly 44, the focus of the cameras 60A, 60B may be adjusted accordingly.

[0041] As shown in Figure 4, the desired focal length may be determined by determining the system state 110 (e.g., arm position and camera position), and then using a selected weighting algorithm 112, processing the system state data 110. The weighting algorithm technique 112 may be configured to match the surgeon's region of interest with high fidelity, and then have the cameras 40A, 40B focus on the region by the autofocus unit 62 without direct knowledge of the scene or direct input from the surgeon. The weighting algorithm may determine the desired focal length 114. The desired focal length 114 may then be sent to the control unit 64, which may then utilize various methods, including using a calibrated focus curve 100, to determine a selected focus command 116 for a given desired depth of field. The focus command 116 is sent to the autofocus unit 62 to change the physical focal length of the camera. The weighting algorithm and associated processing may occur within the calculation unit 18. Alternatively, the calculation may be performed within the camera control unit 64.

[0042] According to an alternative embodiment, the surgical robot system 10 may employ filters to process a desired focal length 114. For example, as shown in Figure 5A, the system via the computing unit 18 may employ a weighting algorithm technique 112 to generate a desired focal length 114. The desired focal length 114 may then pass through an optional filter, such as a low-pass filter 118, to reduce large or rapid changes in the focus data. The intensity of the low-pass filter 118 may be adjusted or varied depending on the degree of head movement (i.e., the greater the head movement, the weaker the filter intensity). The output of the low-pass filter 118 may be transmitted to a control device 64. Alternatively, the filter may be positioned at the output of an autofocus unit 62, as shown in Figure 5B. In particular, the filter is not limited to a low-pass filter, and this disclosure assumes other known filters.

[0043] According to another practice of this disclosure, the desired focal length 114 may vary between cameras 60A and 60B. By focusing cameras 60A and 60B at positions from the desired focal length 114 (for example, one camera closer and the other further away), the composite image can be created with a greater depth of field when the images from the cameras are superimposed.

[0044] The system can determine a desired focal length by taking the distance between the end effector portion of the robot arm and the camera, and by adopting selected weight values ​​associated with each robot arm. By mathematically combining the weight values ​​in a selected manner, the control unit 64 can then determine a desired focal length 114. The position and orientation of the camera assembly 44 relative to each end effector 46A, 46B can be determined by the camera assembly's control unit 64 and the control unit 26. The control unit 64 then generates a control signal to be received by the autofocus unit 62. In response, the autofocus unit 62 generates a signal to change, adjust, or control the focus or length of the camera 60A according to known techniques. As the distance between the end effector and the camera assembly changes, the autofocus unit 62 can automatically adjust the focus or length of the camera in response.

[0045] According to one embodiment, system state data is processed by a weighting algorithm unit to adjust the relative influence of the input on the output of system 10. For example, as shown in Figures 6A and 6B, system state data 110 defining the state of the robot arm and camera positions (e.g., posture) is generated by the control unit 26. The system state data 110 is then introduced into a focal length calculator 140 for determining a desired focal length 114. The focal length calculator 140 may form part of a calculation unit 18 or part of a robot subsystem 20. The focal length calculator 140 may include a data extraction unit 142 for extracting data of a selected type and generating processed system state data 144. The processed system state data 144 is transmitted to a weighting algorithm unit 146 for applying weighting algorithm techniques to the system state data 144. The weighting algorithm unit 146 generates a desired focal length 114 from the system state data. More specifically, the data extraction device unit 142 may employ multiple different distance calculation units for calculating the normal distance of the camera to each robot arm, including, for example, a left normal distance calculation unit 142A and a right normal distance calculation unit 142B. For example, the system state information 110 may include information on the position and orientation of the left end effector, as well as information on the position and orientation of the left camera introduced into the left normal distance calculation unit 142A.

[0046] Furthermore, the system state information 110 may include information on the position and orientation of the right end effector, as well as information on the position and orientation of the right camera, which is introduced into the right normal distance calculation unit 142A. The left normal distance calculation unit 142A calculates left distance data 144A, which indicates the distance between the left camera and the left end effector, from the input data. Similarly, the right normal distance calculation unit 142B calculates right distance data 144B, which indicates the distance between the right camera and the right end effector, from the corresponding input data. The distance data 144A and 144B are then introduced into the weighting algorithm unit 146 for further processing. The weighting algorithm unit 146 may include a focal length calculation unit that determines the desired focal length 114 using the following formula: Focal length=(W1×Z L +W2×Z R ) / (W1+W2) In the formula, W1 and W2 represent selected weight values ​​associated with the left and right robot arms, respectively, Z1 represents the distance value 144A from the left normal distance calculation unit 142A, and Z2 represents the distance value 144B from the right normal distance calculation unit 142B.

[0047] Therefore, according to the weighting algorithm technique, each robot arm is weighted separately, and then the weight values ​​are normalized by dividing by the sum of their weights so that the calculated desired focal length 114 is within a suitable range. According to one embodiment of the weighting algorithm technique, both weights W1 and W2 are fixed and equal to 1 so that the weighting algorithm effectively calculates the average of the two distances. According to another embodiment of the weighting algorithm technique, the weights W1 and W2 of the two robot arms can be varied so that one arm has a greater influence on the other in determining the desired focal length 114. In yet another embodiment of the weighting algorithm technique, the weights of the two robot arms may be a function (i.e., not fixed) based on other system parameters. System parameters may include, for example, how central the end effector is within the camera's field of view (FOV), the state of the end effector (i.e., whether the gripper is open, closed, or somewhere in between), the position of the elbow joint, or any other parameters that the system can measure and relay to the focal length calculation unit 140.

[0048] System parameters can be extracted from system state data 110 by a data extraction unit 142. An embodiment of this technique is shown, for example, in Figure 6C. In the illustrated embodiment, the data extraction unit 142 may include an additional computation unit for determining additional system state data 144. The additional system state data 144 is then processed by a weighting algorithm unit 146A. When utilizing how centrally an end effector is within the camera's field of view, the data extraction unit 142 uses geometric knowledge of the X and Y components of the end effector position and how it relates to the camera's FOV to calculate the distance of a given end effector to the center of the FOV. In this embodiment, the weight of a given end effector in the weighting algorithm technique is a function of the distance to the center of the FOV. For example, the closer an end effector is to the center, the stronger its weight. If an object is outside the field of view, its weight decreases. Such a dependency may be desirable because it increases the likelihood that the user is focusing on the center of the FOV, and therefore, an end effector being closer to the center may correlate with the user's desire to focus there.

[0049] An example of the field of view of the camera assembly is shown in Figure 7. The illustrated field of view 150 has a center 152. The robot arm 42 includes a robot arm 42A with an end effector or gripper 46A, and a robot arm 42B with a right end effector or gripper 46B. The system obtains center distance data R of the right and left robot arms from the image data. 左 and R 右 It is possible to determine this.

[0050] Figure 8 shows a graph 160 that may be used by a control unit of the present disclosure. Graph 160 shows weight values ​​along the Y-axis 162 and distances from the center of the field of view 152 along the X-axis 164. As can be seen, the weights by the end effector decrease as the distance from the center of the FOV increases. In other embodiments, the relationship between the distance from the center of the FOV and any associated weights may take other nonlinear forms, such as polynomic, logarithmic, inverse exponential, or any other relationship known in the art.

[0051] In yet another embodiment, the weights of each robot arm 42A, 42B may depend on the state of the gripper at the end effector (e.g., open or closed, or somewhere in between). For example, if the user is closing the gripper, this may indicate that the user is performing an action that they wish to observe at that end effector, and therefore it is desirable that the end effector be in focus and thus weighted more heavily. Figure 9 shows a graph 160 that may be used by a control unit of the present disclosure. Graph 170 shows weight values ​​along the Y-axis 172 and the state of the gripper along the X-axis 174. The state of the gripper may change between a closed state 176 and an open state 178. As can be seen, the weight by the end effector decreases as the gripper transitions from a closed state to an open state. In other embodiments, the relationship between the state of the gripper and any associated weights may take other nonlinear forms, such as polynomic, logarithmic, inverse exponential, or any other relationship known in the art.

[0052] In other embodiments, the weights that can be applied to more distant end effectors may be larger. This biases the output so that objects appear further away, making the background more visible than the foreground. This effectively creates a second-order (or higher-order) dependency on the depth of the end effectors.

[0053] As shown in Figures 10A–10C, it is desirable to limit autofocus adjustment in certain situations. For example, during suturing or other operations, it is desirable to reduce the speed of focus adjustment as the end effector moves toward the patient. Focus adjustment should be rapid as the end effector of the robotic arm moves away from the patient or target position (e.g., outward), while it should be slower as the end effector moves toward the patient or target position. This difference in autofocus speed provides stronger autofocus performance and a sharper background image. Small movements, such as dropping a suture, result in minimal change to the actual set focal position, while larger movements result in more abrupt changes. This behavior can be generated using weights that influence positive movements that are farther away (stronger than negative movements). Proportional-integral-derivative (PID) control with a highly weighted I / D term can influence negative movements in a way that provides less change at smaller movement changes. Figure 10A shows an example of such movement where the focus is slightly anterior to the tissue.

[0054] As shown in Figure 10B, the tissue is manipulated, and it is desirable that the focus remain close to the target. That is, Figure 10B shows an uncoordinated hand movement or suturing. For example, if a surgeon begins to discard a suture, it is undesirable to change the focus immediately, and the focus should remain closer to the tissue than to the center of the arm. Therefore, a large filter may be applied to the autofocus calculation during rapid movements. Figure 10C shows an example of a longer movement that results in a focus position shift toward the task. In this example, the focus may advance faster the closer the end effector (grasp) it is coming together with is. The proximity of the end effectors indicates coordinated movement.

[0055] Furthermore, weights W1 and W2 are functions of multiple system parameters and can be simple multiplicative functions of the individual weights of multiple parameters. An example of multiplicative weights is as follows: W1 = wLC × wLG × wLZ In the formula, W1 is the total weight of the left end effector, wLC is the weight from the distance of the left arm to the center of the FOV, wLG is the weight from the open / closed state of the left gripper, and wLZ is the weight from the depth to the left end effector.

[0056] In other embodiments, additional distances, such as the distance to the elbow, the background distance, and the distance from the autofocus algorithm, may be incorporated into the weighting algorithm technique. A suitable embodiment of the weighting algorithm technique is shown, for example, in Figure 11. The illustrated focal length calculation unit 140 may include the illustrated data extraction unit 142 and the illustrated weighting algorithm unit 146. In the illustrated present embodiment, the elbow joint of the robot arm is an additional point of interest to be weighted. According to one practice, the elbow joint may have an associated weight value smaller than the weight value associated with the end effector. This is desirable because, when only the elbow joint is viewed with the end effector out of the FOV, the output of the weighting algorithm technique may be dominated by the distance the elbow is from the camera.

[0057] Furthermore, if the user has the end effector portion with the elbow joint visible in the field of view, the weight associated with the elbow joint may be significantly smaller than the weight associated with the end effector, and therefore the output may be dominated by the distance to the end effector. In other embodiments, the focal length calculator 140 may receive image data from a camera assembly as input and utilize PDAF or other autofocus algorithms to calculate the autofocus distance to focus the image. This may be particularly useful when other weight values ​​fall to zero (or 1 in some embodiments). In yet another embodiment, the input to the weighting algorithm may include three-dimensional depth information calculated from the camera's stereoscopic image data to calculate the distance to the background of the image, and then use that information as another input in the calculation of the desired focal length 114. This may be useful in biasing the desired focal length 114 to be closer to the background, and thereby expanding the depth of view by focusing further away.

[0058] The field of view (FOV) can be determined by the optical elements within the camera. Each camera may also be tested at the factory to ensure that its FOV is within a normal range (e.g., within 60 / 601 ± 15% of the diagonal value of the stated value). Furthermore, the camera's FOV changes as the focus changes. A correction curve for FOV changes with respect to focus can be calculated based on the lens elements.

[0059] Therefore, the focal length calculation unit 140 takes the weight of each position of interest (e.g., elbow joint, gripper, background, etc.) and then multiplies it by the dot product of the positions of interest and the camera's pointing vector. In other embodiments, the system may calculate the spherical range of influence from the camera that provides weight values ​​based on positions within the spherical range. Regions of the field of view may be assigned to have larger weight values ​​associated with them, and the weight values ​​may peak at the center of the spherical range. Since the camera does not produce a spherical window but rather a rectangle, positions must be weighted not on a pure dot product, but therefore on axial positions. Thus, movement of the camera along the X-axis may have a weaker gradient than movement along the Y-axis because the X-axis has a larger field of view.

[0060] It should be noted that in some embodiments, the weighting algorithm may take a form different from the normalized linear combination described above. The weighting algorithm may also use any selected type of filter, combining the state data with the desired focal length, for example, a Kalman filter or even machine learning techniques, to help determine the desired focal length based on the system state data.

[0061] Figure 12 is a schematic flowchart illustrating the steps or methods used by the surgical robot system 10 to autofocus the camera of the camera assembly 44 based on one or more system parameters. As shown in the figure, raw positional data of selected components of the surgical robot system, such as sensors (e.g., magnetometer and IMU) and other control inputs, is aggregated and stored in the memory unit 24 (step 70). The positional data can be used by the control unit 26 and / or the control device 64 to determine the pose or position of the robot arms 42A, 42B and the camera assembly 44 in three-dimensional space (step 72).

[0062] In other words, the control device may calculate the position of the robot arm using either direct or indirect measurement techniques. According to the direct measurement technique, the robot arms 42A, 42B and cameras 60A, 60B each have an associated absolute sensor located at their furthest joint. The absolute sensor can measure the position and orientation of the furthest end of the robot arm in six degrees of freedom (e.g., X, Y, Z, yaw, pitch, and roll) with respect to a common spatial origin. This sensor allows the control device to easily determine the relative position and orientation of the robot arm and camera. Suitable embodiments of absolute sensors for use in this disclosure may include alternating magnetic field tracking (such as that incorporated in the technology owned by Mr. Polhemus), optical tracking methods and IMUs, and others known in the art.

[0063] According to indirect measurement techniques, the posture and position of the robot arm and camera are estimated from other known parameters in the system, such as the individual angles of each joint in the arm. The angles of each joint are then input into a forward kinematic model of the robot arm to calculate the position and orientation of the robot's most distal end or the camera in six degrees of freedom. The position of the robot arm or camera can be measured in several ways, all depending on the state of the system. According to one practical example, as shown in Figure 13, in a cable-driven surgical robot system, the control unit determines the state of the cable and then uses that information to estimate the position of each joint in the robot arm. As is well known, each cable used in the robot arm changes a specific joint position and therefore has a computable effect on the position of the robot arm and camera assembly.

[0064] For example, the system may employ a suitable position sensor, such as a Hall effect sensor or a potentiometer, positioned on the output of a motor unit 40 that controls the drive cable to a given joint of a robot arm or camera. In this embodiment, the ratio of the motor's motion to the robot joint's motion is known and fixed based on the shape (e.g., radius) of any motor pulley and drive pulley used for the joint of the robot arm. Thus, the system may determine that when the motor rotates by X degrees, the robot joint moves by Y degrees, where Y = X × R, and R is the ratio described above. For this reason, the control unit 26 may determine the motor position 120 for each joint of the robot arm. The control unit 26 also determines the joint position 124 for all joints of the robot arm by using a selected joint ratio 122, which may be defined as Rj / Rm, where Rj is the radius of the joint and Rm is the radius of the motor. Subsequently, the calculated joint position 124 and the associated motion of the joint can be input into the forward kinematics model 126 to calculate the position and orientation 128 of the most distal part of the robot arm. This method works best when the cable is assumed to be stiff and the friction acting on the cable is low.

[0065] According to one embodiment, as shown in Figure 14, the angular position of a robot joint can be measured via a sensor directly attached to the joint. This may include a Hall effect sensor array, a potentiometer, or other methods known in the art. This type of joint measurement may be useful in cable-driven systems when the cable is not very stiff or when there is high friction acting on the cable. In one embodiment, a Hall effect sensor array embedded in the joint of a robot arm measures the rotational position of a magnet embedded on the opposite side of the joint, thereby enabling direct measurement of the angular position of that joint. Thus, the system can determine the joint position 130 of the robot arm via the sensor. The joint position 130 can be introduced into a forward kinematics model 126 that processes the joint position data to calculate or determine the robot position and orientation 128 of the most distal part of the robot arm.

[0066] In all indirect measurement methods and techniques, the forward kinematic model 126 of the robotic arm and camera assembly refers to a common origin and coordinate frame. This can be done by utilizing knowledge of the geometric relationships or quantities between the robotic arm and the camera assembly, which can be adopted as additional parameters in the forward kinematic model 126. This is not necessary in direct measurement techniques, as they have already measured back to a common origin in essence.

[0067] Regardless of the technique employed to determine the robot's position and orientation, once the position and orientation of the robot arm's end effectors, as well as the camera's position and orientation, are calculated, the distance between the camera and each individual end effector can be calculated. Furthermore, indirect and direct measurement techniques may be combined. For example, the camera assembly may be measured directly, the robot arm indirectly, or vice versa. In other embodiments, the measurement techniques are combined within a single system. For example, the robot may utilize a Hall effect sensor for one joint and a motor cable measurement technique for another joint on the same robot.

[0068] Once the orientation or position of each of the robot arms 42A, 42B and the camera assembly 44 is determined by the system 10, the distance of the camera assembly 44 from both the end effectors 46A and 46B of the robot arms can be determined or calculated by the control unit 26. (Step 74) Once the position and orientation of the end effectors of the robot arms, as well as the position and orientation of the camera, are calculated, the distance between the camera and each individual end effector can be calculated. This can be calculated in different ways, including by employing known mathematical relationships such as the Pythagorean theorem (three-dimensional), quaternion rotations and translations, vector and dot products, and many other methods known in the art. When using the Pythagorean theorem, the theorem determines the X, Y, and Z coordinates and then calculates the distance between points in the following manner: Darm_to_Camera=sqrt[(XEE-XC)2+(YEE-YC)2+(ZEE-ZC)2], In the equation, XEE, YEE, and ZEE are the coordinates of the distal end effector of the robot arm, and XC, YC, and ZC are the coordinates of the camera. The vector and dot product method utilizes the dot product of the line of sight in vector form of the camera, which can be directly determined from the orientation and geometric shape of the camera in a given state, and a vector that starts at the camera and ends at the end effector. This gives the end effector a distance perpendicular to the camera's line of sight from the camera. This identical distance can be calculated using quaternion rotation and translational movement to position the end effector's position and orientation (X, Y, Z, yaw, pitch, roll) in the same coordinate frame as the camera. Once this operation is performed, the Z term of the end effector in this new coordinate frame (direction perpendicular to the camera) is the distance the end effector is perpendicular to the camera's line of sight from the camera.

[0069] Furthermore, the field of view (FOV) is determined by the optical elements within the camera assembly. Each camera 60A, 60B within the camera assembly 44 may be tested at the factory to ensure that the FOV is within an acceptable range. In addition, since the FOV changes in direct correlation with changes in focus, a correction curve for changes in FOV with respect to focus may be calculated based on the optical lens stack used by the camera.

[0070] Subsequently, the control device 64 may generate a control signal to be received by the autofocus unit 62, which may automatically adjust the focus of cameras 42A and 42B. (Step 84) Furthermore, the system 10 may allow a surgeon or user to manually adjust the focus of the camera. (Step 80) Furthermore, the camera focus curve of the camera assembly may be pre-stored in the memory element 24. (Step 82) The focus curve is an embodiment of how the focus control device converts a desired focal length 114 into a command that the system can use to achieve that desired focal length. For this purpose, the desired focal length 114 may be implemented via a lookup table or a suitable function of the position of the focal element relative to the focal length. It is known that the lens must be calibrated for each camera in order to enable the focus to be adjusted as a function of the distance to an object.

[0071] Figure 15 shows an example of one type of focus curve 90 that may be associated with cameras 60A and 60B of the surgical robot system 10. As illustrated in the focus curve 90, the optimal distance to focus changes as the distance-amplitude correction (DAC) value changes. As noted, larger changes are required for targets closer to the camera. This is due to the design of the voice coil module (VCM), where the current command has an inverse square relationship with lens movement, and the changes required by the VCM become smaller as the distance increases.

[0072] Figure 16 shows a curve representing the defined focal area of ​​the camera in camera assembly 44. The illustrated focal curve 100 includes multiple targets at different distances that overlap each other. The peaks of the illustrated curve 100 over all distances are shown above. Note that a larger peak at greater distances does not necessarily mean a sharper image. Rather, what is important is the width of the curve over that number of distances.

[0073] The camera focus curve of camera assembly 44 may be generated and pre-stored in distance space at the factory and applied, and the camera focus may be adjusted to the point the surgeon is fixated on using a robotic arm as the minimum position on the depth of the field of view of the intended scene. The term "distance space" as used herein is intended to describe physical distances in space.

[0074] The cameras 60A and 60B of the camera assembly 44 may be calibrated and configured at the factory before being incorporated into the surgical robot system 10 of the present disclosure. The camera assembly 44 of the present disclosure is configured to focus on the surgeon's surgical site (e.g., surgical site) while maintaining the maximum possible range of motion for the robot arm. Calibration of the camera assembly 44 may occur at the factory and may include selected tests to determine the resolution of the cameras 60A and 60B. Camera calibration may occur after final assembly due to the need to position the cameras and register camera data in selected memory.

[0075] The output data from each camera may be accumulated during calibration and stored in a lookup table. The lookup table may be associated, for example, with each camera and associated lens assembly. The process for calibrating the cameras is as follows: The cameras may be mounted on a motor unit and locked to a selected support that controls the orientation and position of the cameras and associated sensors. A single axial stage is used to change the distance from the camera to a target from approximately 5 cm to approximately 20 cm, and the cameras are calibrated at all distances within this range. A 10 mm spacing distance between cameras in the camera assembly may be used and can be adjusted. The calibration target is preferably a calibrated resolution target with sharp edges. Each position is swept with various VCM currents to obtain the best resolution score determined by the variance of the Laplacian in the central region of the image. The region of interest from the central 50% to the central 10% may be performed in a linear form from a distance of approximately 50 mm to approximately 200 mm. The calibration output data may be used to form the focal curve 90.

Claims

1. A surgical robotic system, Sensor unit and Multiple robot arms, each robot arm having an end effector at its distal end, A camera assembly comprising at least two cameras, a control device, and an autofocus unit configured to automatically focus the lenses of each of the at least two cameras, A robot subsystem that communicates with the sensor unit, Equipped with, The control device is configured to calculate a desired focal length based on the state information of at least two cameras and the plurality of robot arms, the state information including position and orientation information of each of the end effectors of the plurality of robot arms received from the sensor unit, the distance from each camera to each end effector of the plurality of robot arms within the surgeon's field of view, and a weighting algorithm, each robot arm being weighted differently in the weighting algorithm. A surgical robotic system wherein the autofocus unit is configured to autofocus each of the lenses of the at least two cameras based on the desired focal length.

2. The surgical robot system according to claim 1, wherein the state information further includes position information and orientation information for each camera.

3. The surgical robot system according to claim 1, wherein the control device is configured to determine a focus command according to a specific depth of focus based on the desired focal length, and the focus command changes the desired focal length.

4. The surgical robot system according to claim 3, wherein the control device is configured to transmit the focus command to the autofocus unit, and in response, the autofocus unit is configured to adjust the focus of each camera in order to focus the lens of each camera.

5. The surgical robot system according to claim 3, wherein the control device is configured to filter the desired focal length in order to reduce abrupt changes in the data of the desired focal length.

6. The surgical robot system according to claim 5, wherein the intensity of the filter for filtering the desired focal length is varied based on the degree of head movement of the surgeon.

7. The surgical robotic system according to claim 1, wherein different desired focal lengths are calculated for each of the at least two cameras.

8. The surgical robot system according to claim 1, wherein the weight of each robot arm is a function based on system parameters.

9. The surgical robot system according to claim 8, wherein each robotic arm includes a plurality of joints.

10. The surgical robot system according to claim 9, wherein the plurality of joints include a shoulder joint, an elbow joint, and a wrist joint.

11. The surgical robot system according to claim 10, wherein the system parameters include the distance from the center of each end effector within the field of view of each camera, the state of each end effector, and the position of the elbow joint.

12. The surgical robotic system according to claim 1, wherein the focusing speed is increased as each end effector moves outward from the target position.

13. The surgical robotic system according to claim 1, wherein the focusing speed is reduced as each end effector moves toward the target position.

14. It is a robot subsystem, Sensor unit and Multiple robot arms, each having an end effector at its distal end, It is a camera assembly, At least two cameras, Control device and A camera assembly including an autofocus unit configured to autofocus each of the lenses of at least two of the cameras, A robot subsystem that communicates with the sensor unit, Equipped with, The control device is configured to calculate a desired focal length based on state information of at least two cameras and the plurality of robot arms, which includes position information and orientation information of each of the plurality of robot arms' end effectors received from the sensor unit, the distance from each camera to each of the plurality of robot arms' end effectors within the surgeon's field of view, and a weighting algorithm, wherein each robot arm is weighted differently in the weighting algorithm. The autofocus unit is configured to autofocus each of the lenses of the at least two cameras based on the desired focal length, in a robotic subsystem.

15. The robot subsystem according to claim 14, wherein the focusing speed is increased as the robot arm moves outward from the target position and decreased as the robot arm moves inward toward the target position.

16. The robot subsystem according to claim 14, wherein the state information further includes the distance from each camera to each end effector of the plurality of robot arms within the surgeon's field of view, and at least one of the position information and orientation information of the at least two cameras.

Images