Systems and methods for determining in vivo depth perception in surgical robotic systems

Abstract

A system and method for generating a depth map from image data in a surgical robotic system uses a robotic subsystem having a camera assembly with first and second cameras for generating the image data. The system and method generates multiple depth maps based on the image data and then converts the multiple depth maps into a single composite depth map having associated distance data. The system and method can then control the camera assembly based on the distance data in the single composite depth map.

Description

[Technical field]

[0001] <Related Applications> This invention claims priority to U.S. Provisional Patent Application No. 62 / 971,097, filed February 6, 2020, entitled DEPTH PERCEPTION IN VIVO, the contents of which are incorporated herein by reference.

[0002] The present invention relates to surgical robotic systems, and more particularly to surgical robotic systems that use a camera assembly. [Background technology]

[0003] Minimally invasive surgery (MIS) has proven beneficial for patient outcomes when compared to open surgery, where surgeons operate by hand through large incisions, by significantly reducing patient recovery time, risk of infection, and future hernia rates, and allowing more outpatient procedures to be performed.

[0004] Despite the advent of manual and robotic MIS systems, open surgery remains the standard procedure for many indications due to the complexity of the procedure and general limitations of current MIS solutions, including the amount of training and practice required to become proficient in MIS and the limited abdominal access from a single insertion site.

[0005] U.S. Patent No. 10,285,765, entitled Virtual Reality Surgical Device, U.S. Patent Publication No. 2019 / 0142531, entitled Virtual Reality Wrist Assembly, and U.S. Patent Publication No. 2019 / 0076199, entitled Virtual Reality Surgical Camera System, disclose advanced surgical systems that can reach any location in the abdomen with a single MIS incision. Natural interfaces and poseable viewpoints create surgical systems that require minimal instrument-specific training. Surgeons interact with the robot as if it were their own hands and eyes, which, combined with high-quality manufacturing and lower per-system costs, allows surgeons to focus on providing high-quality care to their patients.

[0006] One of the biggest problems with traditional MIS systems is that injuries can occur outside the operator's field of view. Generating a detailed three-dimensional (3D) map of the surgical stage allows the system to actively prevent collisions and help surgeons plan optimal surgical paths for robotic components, such as robotic arms.

[0007] Furthermore, depth data will enable the use of autonomous surgical robots that can closely interact with human tissue while performing navigation and surgical procedures inside the patient in a safe and efficient manner. Intelligent, autonomous surgical systems capable of performing basic surgical procedures require detailed three-dimensional knowledge of the patient's interior. Knowledge of depth is therefore fundamental to this goal.

[0008] The robotic surgery system disclosed in the aforementioned publication blends miniaturized robotics with augmented reality. In some embodiments, the system includes two eight-degree-of-freedom (DOF) robotic arms, plus a stereoscopic camera assembly or head that is inserted through a single incision and deployed within the patient's abdominal cavity. The surgeon controls the robotic arms with two six-axis handheld controllers while visualizing the complete surgical situation via the robotic head and a virtual reality headset.

[0009] The unique architecture of this system offers opportunities and capabilities not realized by any other surgical method or system, yet the surgeon can only utilize the patient's internal visual feedback. The brain naturally uses a variety of low- and high-level sources to obtain reliable and robust distance estimates. This is because humans rely heavily on depth signatures to interact with their environment. Making these signatures accessible through the robotic head enhances the surgeon's capabilities, creating a much richer and more effective experience.

[0010] However, available conventional and commercially augmented reality and virtual reality systems are unable to provide sufficient depth information to the surgeon to ensure accurate visual representation and feedback. Summary of the Invention

[0011] The present invention is directed to a surgical robotic system that employs a depth perception subsystem that generates multiple depth maps and then combines or merges the depth maps into a single depth map. The depth perception subsystem also generates a set of confidence values ​​associated with the distance data in the single composite depth map. The confidence values ​​are an indication of the confidence or likelihood that the distance data associated with a selected point or portion of the depth map is correct or accurate. The depth perception subsystem generates depth maps associated with the cameras of a camera assembly. In particular, the depth perception subsystem generates depth maps associated with the autofocus mechanisms of the cameras, disparities associated with each camera, and(parallax) data, and between the image data from each camera (disparity) The depth perception subsystem then processes all of the depth maps to generate a single combined depth map. The depth map data and the confidence value can be used by the system to move one or more components of the robotic subsystem.

[0012] The present invention is directed to a surgical robotic system comprising: a robotic subsystem having a camera assembly having a first and second camera for generating image data; a computing unit having: a processor for processing the image data; a control unit for controlling the robotic subsystem; a depth perception subsystem for receiving image data generated by the first and second cameras, generating multiple depth maps based on the image data, and then converting the multiple depth maps into a single composite depth map with associated distance data. The robotic subsystem further comprises multiple robotic arms and a motor unit for controlling movement of the multiple robotic arms and the camera assembly. The control unit uses the distance data associated with the single combined depth map to control one of the camera assembly and the robotic arm. The depth perception subsystem further comprises a depth map conversion unit for receiving the multiple depth maps and then converting the depth maps into a single composite depth map. The depth map conversion unit generates the single composite depth map using a region convolutional neural network (R-CNN) technique.

[0013] Additionally, each of the first and second cameras includes an image sensor for receiving the optical data and generating image data in response thereto, a lens and optical system having one or more lens elements optically coupled to the image sensor for focusing the optical data onto the image sensor, and an autofocus mechanism associated with the lens and optical system for automatically adjusting the one or more lens elements and generating autofocus data.

[0014] The depth perception subsystem of the present invention includes any combination of the following: a first autofocus conversion unit for receiving autofocus data from a first camera and converting the autofocus data into a first autofocus depth map; a second autofocus conversion unit for receiving autofocus data from a second camera and converting the autofocus data into a second autofocus depth map; a first parallax conversion unit for receiving image data from the first camera and converting the image data into a first parallax depth map; a second parallax conversion unit for receiving image data from the second camera and converting the image data into a second parallax depth map; a second parallax conversion unit for receiving image data from the first camera and the second camera and converting the image data into a second parallax depth map accordingly. (disparity) Generate a depth map, (disparity) Conversion unit.

[0015] The first and second disparity units may be configured to obtain first and second successive images in the image data, and then measure the amount that each portion of the first image moves relative to the second image. Furthermore, each of the first and second disparity conversion units may include: a separation unit that receives image data from a respective camera, divides the image data into a plurality of segments, and then generates shifted image data in response to the plurality of segments; a movement determination unit that receives position data from a respective camera, and then generates camera movement data indicative of a position of the camera in response thereto; a distance conversion unit that receives the image data and the camera movement data, and then converts the image data and the camera movement data into a respective disparity depth map. The distance conversion unit uses a region convolutional neural network (R-CNN) technique to generate the respective disparity depth map. Also, (disparity) The conversion unit converts between an image of the image data received from the first camera and an image of the image data received from the second camera. (disparity) Analyze.

[0016] The depth perception subsystem includes a first autofocus depth map, a second autofocus depth map, a first disparity depth map, and a second disparity depth map.(disparity) The present invention further comprises a depth map conversion unit for receiving the depth maps, forming a received depth map, and then converting the received depth map into a single composite depth map. The depth map conversion unit also includes a depth map generation unit for converting the received depth maps into the single composite depth map after receiving the received depth maps, and a confidence value generation unit for generating from the received depth maps a confidence value associated with each of the distance values ​​associated with each point of the single composite depth map. The confidence value indicates a confidence of the distance values ​​associated with the single combined depth map.

[0017] The present invention also relates to a method for generating depth maps from image data in a surgical robotic system, the method comprising: providing a robotic subsystem having a camera assembly having first and second cameras for generating image data; generating a plurality of depth maps based on image data from the first and second cameras; converting the plurality of depth maps into a single composite depth map having distance data associated therewith; and controlling the camera assembly based on the distance data in the single composite depth map.

[0018] The method also includes one or more of the following, or any combination thereof: converting the autofocus data from the first camera into a first autofocus depth map; converting the autofocus data from the second camera into a second autofocus depth map; converting image data from the first camera into a first disparity depth map; converting image data from the second camera into a second disparity depth map; converting the image data from the first camera and the image data from the second camera into a second disparity depth map. (disparity) Generating a depth map.

[0019] The method also includes: a first autofocus depth map; a second autofocus depth map; a first disparity depth map; and a second disparity depth map. (disparity)The method includes receiving a depth map, forming a received depth map, and then converting the received depth map into a single composite depth map. The method further includes generating, from the received depth map, a confidence value associated with each of the distance values ​​associated with each point of the single composite depth map. The confidence value indicates a confidence of the distance values ​​associated with the single combined depth map. [Brief description of the drawings]

[0020] These and other features and advantages of the present invention will be more fully understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which like reference numbers refer to like elements throughout the different drawings, which illustrate the principles of the invention and, although not to scale, show relative dimensions. (Figure 1) FIG. 1 is a schematic block diagram of a surgical robotic system suitable for use with the present invention. (Figure 2) FIG. 2 is a schematic diagram of a depth perception subsystem in accordance with the teachings of the present invention. (Figure 3) FIG. 2 is a schematic block diagram of a disparity conversion unit of the depth perception subsystem of the present invention. (Figure 4) 2 is a schematic block diagram of a depth map conversion unit of a depth perception subsystem in accordance with the teachings of the present invention. (Figure 5) 2 is an exemplary schematic diagram of a processing technique employed by a depth map conversion unit of a depth perception subsystem in accordance with the teachings of the present invention. (Figure 6) 2 is an example depth map in accordance with the teachings of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] In the following description, numerous specific details are set forth with respect to the systems and methods of the present invention, as well as the environments in which the systems and methods may operate, to provide a thorough understanding of the disclosed subject matter. However, it will be apparent to those skilled in the art that the disclosed subject matter may be practiced without such specific details, and that certain features well known in the art will not be described in detail to avoid complicating the disclosed subject matter. In addition, it is understood that any examples provided below are merely illustrative and should not be construed as limiting, and that other systems, devices, and / or methods may be used to implement the teachings of the present invention and are considered to be within the scope of the present invention.

[0022] Although the systems and methods of the present invention can be designed for use with one or more surgical robotic systems used as part of a virtual reality surgery, the systems of the present invention can be used with any type of surgical system including, for example, robotic surgical systems, straight stick surgical systems, and laparoscopic systems. In addition, the systems of the present invention can be used in other non-surgical systems where a user requires access to a large amount of information while controlling a device or apparatus.

[0023] The systems and methods disclosed herein can be incorporated into and utilized, for example, the robotic surgical devices and associated systems disclosed in U.S. Patent No. 10,285,765 and PCT Patent Application No. PCT / US20 / 39203, and / or the camera system disclosed in U.S. Patent Application Publication No. 2019 / 0076199. The entire contents and teachings of the aforementioned applications and publications are incorporated herein by reference. A surgical robotic system forming part of the present invention can comprise a surgical system including a user workstation, a robotic support system (RSS), a motor unit, and an embedded surgical robot including one or more robotic arms and one or more camera assemblies. The embedded robotic arms and camera assemblies can form part of a single support axis robotic system or can form part of a split arm architecture robotic system.

[0024] The robotic arm can have movement-associated portions or regions that correspond to the user's shoulder, elbow, wrist, and fingers. For example, the robotic elbow can follow the position and orientation of a human elbow, and the robotic wrist can follow the position and orientation of a human wrist. The robotic arm can also have an associated end region, terminating in an end effector that follows the movement of one or more of the user's fingers, such as the index finger when the user pinches the index finger and thumb together. The robotic shoulder is fixed in place while the robotic arm follows the movement of the user's arm. In one embodiment, the position and orientation of the user's torso is subtracted from the position and orientation of the user's arm. This subtraction allows the user to move the torso without moving the robotic arm. The robotic arm can be configured to reach all areas of the surgical site and can operate in different configurations and tight spaces.

[0025] The present invention is directed to a surgical robotic system employing a depth perception subsystem that generates multiple depth maps and then combines or merges the depth maps into a single composite depth map. The depth perception subsystem also generates a set of confidence values ​​associated with distance data in the single composite depth map. The confidence values ​​are indicative of a confidence or likelihood that distance data associated with selected points or portions of the depth maps is correct or accurate. The depth perception subsystem generates depth maps associated with the cameras of a camera assembly. In particular, the depth perception subsystem generates depth maps associated with the autofocus mechanisms of the cameras, disparity data associated with each camera, and a depth map associated with the camera assembly. (parallax data) , and between the image data from each camera (disparity) The depth perception subsystem then processes all of the depth maps to generate a single combined depth map. The depth map data and the confidence value are used by the system to move one or more components of the robotic subsystem.

[0026] FIG. 1 is a schematic block diagram illustration of a surgical robotic system 10 according to the teachings of the present invention. The system 10 includes a display device or unit 12, a virtual reality (VR) computing unit 14, a sensing and tracking unit 16, a computing unit 18, and a robotic subsystem 20. The display unit 12 can be any selected type of display for displaying information, images, or videos generated by the VR computing unit 14, the computing unit 18, and / or the robotic subsystem 20. The display unit 12 can include, for example, a head mounted display (HMD), a screen or display, a three-dimensional (3D) screen, and the like. The display unit can also include an optional sensor and tracking unit 16A, such as found in commercially available head mounted displays. The sensing and tracking units 16 and 16A can include one or more sensors or detectors coupled to a user of the system, such as a nurse or surgeon. The sensors can be coupled to the user's arms, and additional sensors can also be coupled to the user's head and / or neck area if a head mounted display is not used. The sensors in this configuration are represented by the sensing and tracking unit 16. If the user uses a head mounted display, eye, head and / or neck sensors and associated tracking technology may be incorporated or used within that device and thus form part of the optional sensing and tracking unit 16A. The sensors of the sensing and tracking unit 16 coupled to the surgeon's arm may be preferably coupled to selected regions of the arm, such as the shoulder region, elbow region, wrist or hand region, and fingers if desired. The sensors generate position data indicative of the position of selected parts of the user. The sensing and tracking units 16 and / or 16A may be utilized to control the camera assembly 44 and the robot arm 42 of the robotic subsystem 20. The position data 34 generated by the sensors of the sensing and tracking unit 16 is conveyed to the computing unit 18 for processing by the processor 22.From the position data, the computing unit 18 can determine or calculate the position and / or orientation of each part of the surgeon's arm and communicate this data to the robotic subsystem 20. According to an alternative embodiment, the sensing and tracking unit 16 can use sensors coupled to the surgeon's torso or any other body part. Furthermore, the sensing and tracking unit 16 can use, in addition to the sensors, an inertial momentum unit (IMU) having, for example, an accelerometer, a gyroscope, a magnetometer, and a motion processor. The addition of a magnetometer is standard practice in the field, and the magnetic orientation allows for a reduction in sensor drift around the vertical axis. Alternative embodiments also include sensors placed within surgical materials such as gloves, surgical scrubs, or surgical gowns. The sensors may be reusable or disposable. Additionally, the sensors can be placed external to the user, such as in fixed positions in a room, such as an operating room. The external sensors can generate external data 36 that is processed by the computing unit and thus used by the system 10. According to another embodiment, when the display unit 12 is a head-mounted device using an associated sensing and tracking unit 16A, the device generates tracking and position data 34A that is received and processed by the VR computation unit 14. Additionally, the sensing and tracking unit 16 may include hand controllers, if desired.

[0027] In embodiments in which the display is an HMD, the display unit 12 may be, for example, a virtual reality head mounted display such as Oculus Rift, Varjo VR-1, or HTC Vive Pro Eye. The HMD may provide the user with a display coupled to or attached to the user's head, lenses that allow a focused view of the display, and a sensor and / or tracking system 16A that provides tracking of the position and orientation of the display. The position and orientation sensor system may include, for example, accelerometers, gyroscopes, magnetometers, motion processors, infrared tracking, eye tracking, computer vision, emitting and sensing alternating magnetic fields, and any other method or combination of tracking at least one of position and orientation. As is known, the HMD may provide image data from a camera assembly 44 to the right and left eyes of the surgeon. To maintain a virtual reality experience for the surgeon, the sensor system may track the position and orientation of the surgeon's head and then relay the data to the VR computing unit 14 and, if necessary, to the computing unit 18. The computing unit 18 can further adjust the pan and tilt of the camera assembly 44 of the robotic subsystem 20 via the motor unit 40 to follow the movements of the user's head.

[0028] When associated with the display unit 12, sensor or position data generated by the sensors can be communicated to the computing unit 18, either directly or via the VR computing unit 14. Similarly, tracking and position data 34 generated by other sensors in the system, such as from the sensing and tracking unit 16, which may be associated with the user's arms and hands, can be communicated to the computing unit 18. The tracking and position data 34, 34A can be processed by the processor 22 and stored, for example, in the storage unit 24. The tracking and position data 34, 34A can also be used by the control unit 26 to generate control signals for controlling one or more portions of the robotic subsystem 20 in response. The robotic subsystem 20 can include a user workstation, a robotic support system (RSS), a motor unit 40, and an implantable surgical robot including one or more robotic arms 42 and one or more camera assemblies 44. The embedded robotic arm and camera assembly may form part of a single support axis robotic system such as that shown and described in U.S. Pat. No. 10,285,765, or may form part of a split arm architecture robotic system such as that shown and disclosed in PCT Patent Application No. PCT / US20 / 39203, the contents of which are incorporated herein by reference.

[0029] The control signals generated by the control unit 26 can be received by a motor unit 40 of the robot subsystem 20. The motor unit 40 can include a series of servo motors configured to separately drive the robot arm 42 and the camera assembly 44. The robot arm 42 can be controlled to follow the scaled-down movement or motion of the surgeon's arm as sensed by associated sensors. The robot arm 42 can have portions or regions associated with the movements associated with the shoulder, elbow, wrist, and fingers of a user. For example, the robot elbow can follow the position and orientation of a human elbow, and the robot wrist can follow the position and orientation of a human wrist. The robot arm 42 can also have associated end regions, such as terminating in an end effector that follows the movement of one or more of the user's fingers, such as the index finger when the user pinches the index finger and thumb together. The robot shoulder is fixed in place while the robot arm follows the movement of the user's arm. In one embodiment, the position and orientation of the user's torso is subtracted from the position and orientation of the user's arm. This subtraction allows the user to move the torso without moving the robot arm.

[0030] The robotic camera assembly 44 is configured to provide image data 48, such as, for example, a live video feed of the procedure or surgical site, to the surgeon, and to allow the surgeon to operate and control the cameras that make up the camera assembly 44. The camera assembly 44 preferably includes a pair of cameras whose optical axes are axially spaced a selected distance (known as the inter-camera distance) to provide a stereoscopic view of the surgical site. The surgeon can control the movement of the cameras through the movement of a head-mounted display, or via sensors coupled to the surgeon's head, or by using a hand controller or sensors that track the movement of the user's head or arm, so that the surgeon can obtain a desired view of the surgical site in an intuitive and natural way. The cameras are movable in multiple directions, including, for example, yaw, pitch, and roll directions, as is known. The components of the stereoscopic camera can be configured to provide a user experience that feels natural and comfortable. In some embodiments, the inter-axial distance between the cameras can be modified to adjust the depth of the surgical site perceived by the user.

[0031] The camera assembly 44 is actuated by the movement of the surgeon's head. For example, during surgery, if the surgeon wants to see an object located above the current field of view, the surgeon looks upward, which results in the stereoscopic camera being rotated upward about the pitch axis from the user's perspective. The image or video data 48 generated by the camera assembly 44 can be displayed on the display unit 12. If the display unit 12 is a head-mounted display, the display can include a built-in tracking and sensor system that captures raw orientation data in the yaw, pitch, and roll directions of the HMD, as well as position data in Cartesian space (x, y, z) of the HMD. However, alternative tracking systems can be used to provide supplemental position and orientation tracking data of the display instead of or in addition to the built-in tracking system of the HMD. Examples of camera assemblies suitable for use in the present invention include the camera assemblies disclosed in U.S. Pat. No. 10,285,765 and U.S. Patent Application Publication No. 2019 / 0076199, the contents of which are incorporated herein by reference.

[0032] Image data 48 generated by the camera assembly 44 is communicated to the virtual reality (VR) computation unit 14 and processed by the VR or image rendering unit 30. The image data 48 may include still photographs or image data as well as video data. The VR rendering unit 30 may include appropriate hardware and software for processing the image data and then rendering the image data for display by the display unit 12, as known in the art. Additionally, the VR rendering unit 30 may combine image data received from the camera assembly 44 with information related to the position and orientation of the cameras in the camera assembly, as well as information related to the position and orientation of the surgeon's head. This information enables the VR rendering unit 30 to generate and transmit an output video or image rendering signal to the display unit 12. That is, the VR rendering unit 30 renders a reading of the position and orientation of the hand controllers, as well as the surgeon's head position, for display on the display unit, such as in an HMD worn by the surgeon.

[0033] The VR computation unit 14 may also include a virtual reality (VR) camera unit 38 for generating one or more virtual reality (VR) cameras for use or placement in the VR world displayed in the display unit 12. The VR camera unit may generate one or more virtual cameras in the virtual world, which may be used by the system 10 to render images for the head mounted display. This ensures that the VR camera always renders the same view that a user wearing the head mounted display would see in the cube map. In one embodiment, a single VR camera may be used, while in another embodiment, separate left and right eye VR cameras may be used to render onto separate left and right eye cube maps in the display to provide a stereo view. The FOV setting of the VR camera may be self-configuring relative to the FOV exposed by the camera assembly 44. In addition to providing a contextual background for the live camera view or image data, the cube map may be used to generate dynamic reflections on virtual objects. This effect allows reflective surfaces on the virtual objects to pick up reflections from the cube map, making these objects appear to the user as if they were actually reflecting the real world environment.

[0034] The robotic arm 42 may be comprised of multiple mechanically linked actuating sections or portions that may be constructed and combined for rotational and / or hinged motion to emulate different portions of a human arm, such as, for example, the shoulder region, elbow region, and wrist region of the arm. The actuator sections of the robotic arm may be configured, for example, to provide cable-driven rotational motion, but within reasonable rotational limits. The actuator sections may be configured to provide maximum torque and maximum speed at a minimum size.

[0035] The present invention relates to generating and providing depth perception related data (e.g., distance data and / or depth map data) to a robotic subsystem so that the data can be used to assist a surgeon in controlling the movement of one or more components, such as a robotic arm or camera, of the subsystem. The depth perception related data is important because it allows a surgeon to determine the amount of movement the robot can safely perform at the surgical site before and during a surgical procedure. In addition, the data can be used for automated operations without a surgeon. The present invention also relates to the generation and provision of depth perception related data (e.g., lens focus data, image data, image (disparity) Software and hardware (e.g., processors, memory, storage devices, etc.) may be used to calculate or determine a three-dimensional (3D) distance map, or depth map, from a number of different data and image sources, including image data, and image disparity related data. According to other embodiments, other types of depth cues may be used as inputs to depth perception subsystem 50 of the present invention.

[0036] The present invention can use a variety of computing elements and sensors to determine or extract depth and related distance information. There are a variety of hardware and software that can be used with the system of the present invention to extract depth perception information or distance data. For example, hardware sensors such as structured light or time-of-flight sensors measure changes in physical parameters to estimate distance. Software sensors are used to infer distance by analyzing specific features in one or more images in time and space. The system can use a variety of computing elements and sensors to generate or convert input images or other types of data into depth-related data. (disparity), epipolar geometry, Structure from Motion (SfM), and other techniques can be used. While the inventive system can extract depth-related information from a single cue or source, the inventive system can also consider additional inputs or data sources when constructing a final combined three-dimensional (3D) depth map. The inventive final composite depth map essentially combines multiple lower quality depth maps into a single final composite depth map of any selected scene that is more robust to noise, occlusion, and ambiguity, e.g., the interior of a human body.

[0037] The computing unit 18 of the surgical robotic system 10 may include a depth perception subsystem 50, as shown in FIG. 2, for example. The depth perception subsystem 50 is configured to interact with one or more components of the robotic subsystem 20, such as, for example, the camera assembly 44 and the robot arm 42. The camera assembly 44 may include, for example, a pair of stereo cameras, including a left camera 44A and a right camera 44B. The left camera 44A may include, for example, a lens and optics system 54A including one or more lenses and associated optical elements for receiving optical or image information, among other components. The camera 44A may also include an image sensor 58A for capturing optical or image data, and an autofocus mechanism 62A for providing the camera with an autofocus function. The autofocus mechanism 62A interacts with and automatically changes or adjusts the optics, such as the lenses in the lens and optics system 54A, to focus an image on the image sensor 58A. The image sensor surface may typically correspond to a focal plane. Similarly, the right camera 44B may include a lens and optics system 54B including one or more lenses and associated optical elements. The camera 44B may also include an image sensor 58B for capturing optical or image data and an autofocus mechanism 62B for providing the camera with autofocus functionality.

[0038] Cameras 44A, 44B with autofocus capabilities can provide information that can be converted into an initial rough depth map of the area directly monitored by the camera. Thus, each camera 44A, 44B constantly monitors the input stream of image data from the image sensor 58A, 58B to maintain the observed environment or object in focus. For each image portion in the image data stream, a subset of the pixels of the image can be kept in focus by the corresponding image sensor. As is known, the autofocus mechanism can generate effort signals, which are adjustments required by the autofocus hardware and software, and the effort signals can be converted into control signals that can be used to mechanically or electrically change the geometry of the optical system. Furthermore, for a given image, any subset of pixels that are in focus can be associated with a control signal. The control signals can be converted to approximate depths by the autofocus conversion units 70A, 70B, thereby generating a depth map of the in-focus pixels.

[0039] The illustrated depth perception subsystem 50 may further include an autofocus conversion unit 70A for receiving autofocus data 64A generated by the autofocus mechanism 62A. The autofocus conversion unit 70A serves to convert the autofocus data into distance data, which is displayed as or forms part of a depth map 72A. As used herein, the term "depth map" or "distance map" is intended to include an image, image channel, or map that includes information regarding or about the distance between one or more objects or surfaces of an image from a selected viewpoint or a viewpoint in the overall scene. The depth map may be created from source image or image data and may be presented in any selected color, such as grayscale, and may include one or more color variations or hues, each variation or hue corresponding to various or different distances of the image or object from a viewpoint in the overall scene. Similarly, the autofocus mechanism 62B generates autofocus data received by the autofocus conversion unit 70B. In response, the autofocus conversion unit 70B converts the autofocus data into distance data, which is displayed as or forms part of a separate depth map 72B. In some embodiments, the focal depth can be purposefully varied over time to generate a more detailed depth map of different pixels.

[0040] The depth perception subsystem 50 may further include parallax conversion units 80A and 80B for converting image data into distance data. In particular, the left and right cameras 44A, 44B generate camera data 74A, 74B, respectively, which may include, for example, image data and camera position data, which are forwarded to and received by the parallax conversion units 80A, 80B, respectively, which then convert the data into distance data that may form part of the separate depth maps 76A, 76B. As is known, the parallax effect is typically present in both natural and artificial optical systems, and can make objects that are farther away from the image sensor appear to move slower than objects closer to the image sensor when the image sensor is moved. In some embodiments, the measurement of the parallax effect is achieved by measuring, for two images taken at successive intervals, how much each part of the image has moved relative to its corresponding part in the previous interval. The more a part of the image moves between intervals, the closer it is to the camera.

[0041] Details of the parallax conversion units 80A, 80B are shown in FIG. 3, for example. Since the parallax conversion units 80A, 80B are identical, only the parallax conversion unit 80A will be described below for the sake of brevity and clarity. The camera data 74A generated by the camera 44A may include image data 74C and camera position data 74D. The camera position data 74D corresponds to the vertical and horizontal position of the camera based on the position measured or commanded by the on-board sensors and electronics. The image data 74C is introduced and received by the separation unit 130. The separation unit 130 divides the image data 74C into a number of patches or segments, and then generates shifted image data 132 in response thereto by comparing a set of typically consecutive images in the image data. The camera position data 74D generated by the camera 44A is received by a movement determination unit 134, which determines the position of the camera, and then generates camera movement data 136. The camera movement data 136 relates to the amount of movement or rotation of the camera, measured by on-board sensors, estimated based on kinematics, or simply estimated based on commands. The shifted image data 132 and the movement data 136 are then introduced to a distance conversion unit that converts the two types of input data 132, 136 into distance data, which may be represented in the form of a separate depth map 76A. An example of how to determine distance data from image data is described in: Active estimation of distance in a robotic system that replicates human eye movement, Santini et al, Robotics and Autonomous Systems, August 2006, the contents of which are incorporated herein by reference.

[0042] The distance transformation unit 140 may use known processing techniques, such as, for example, a multi-layer domain convolutional neural network (R-CNN) technique. For example, according to one embodiment, training data for the network is generated using image pairs selected from the image data. The separation unit 130 may further segment an image (e.g., an image taken at time t) into smaller image segments. For each image segment, a position on an image from the same image sensor, but possibly at a different time (e.g., time t+1), is calculated using a normalized cross-correlation technique. Since the depth perception subsystem 50 can easily determine the motion actuated during the preceding time interval, and the position difference of the image segment before and after the motion, the distance of the 3D points included in the image segment may be calculated or determined via known optical considerations and known analytical formulations and techniques.

[0043] Referring again to Figure 2, depth perception subsystem 50 converts image data 78A received from camera 44A and image data 78B received from camera 44B into distance data. (disparity) It further includes a transformation unit 90 , and the distance data may form part of a depth map 92 . (disparity) The conversion unit 90 converts the image data received from the cameras 44A and 44B into a difference between the images. (disparity) Specifically, (disparity) The transformation unit 90 analyzes the same image section of each input image and determines the difference between them. The difference between the images recorded by the optical system and the image sensors of the cameras 44A, 44B by observing the scene from different viewpoints is (disparity) This can be used in conjunction with the known geometry and arrangement of the camera optical systems to convert the information into distance information. According to one embodiment, the distance between the images from the left and right cameras 44A, 44B can be calculated using the known geometry and arrangement of the camera optical systems to convert the information into distance information. (disparity) is computed using a well-layered regional convolutional neural network (R-CNN) that simultaneously considers and processes all pixels from the image. (disparity)The transformation unit 90 can be trained, for example, using images from each camera 44A, 44B selected from a real-time image feed. (disparity) can be calculated using the formula: (disparity) The value (d) can be converted to a depth value (Z) by the following formula: Z = R * f / d where f is the focal length of the cameras and T is the baseline distance between the cameras.

[0044] (disparity) The depth map or distance data 92 generated by the transformation unit 90 and corresponding to the input image data under consideration can be generated and refined by using epipolar geometry. For example, the likely location of point A on the left image received by the left camera 44A can be estimated on the right image received from the right camera 44B. A normalized cross-correlation between the left and right parts of the image around a selected point, such as point A, is then performed to obtain a more accurate location estimate. (disparity) and depth information is derived using well-known analytical formulas. (disparity) The depth map generated by transformation unit 90 can be further improved by a manual refinement process that removes artifacts and outliers that are not easily detectable by the automated functions of the depth perception subsystem.

[0045] The inventors have provided autofocus conversion units 70A and 70B, parallax conversion units 80A and 80B, and (disparity)It has been recognized that the depth maps 72A, 72B, 76A, 76B, and 92 generated by the conversion unit 90 may be inherently unreliable when used separately to determine distance. The individual depth maps may not contain all of the image data and associated position data necessary to properly and appropriately control the robotic subsystem 20. To address this unreliability, the depth perception subsystem 50 may use a depth map conversion unit 100. The depth map conversion unit 100 is configured to receive all of the depth maps and associated distance data generated by the depth perception subsystem 50, and combine or merge the depth maps into a single composite depth map 122. The depth map conversion unit 100 may use one or more different types of processing techniques, including, for example, a regional convolutional neural network (R-CNN) based encoder-decoder architecture.

[0046] The depth map conversion unit 100 is shown in detail in FIG. 3 and FIG. 4. As shown in FIG. 3, the depth map conversion unit 100 may include a depth map generation unit 120 that combines input depth maps and generates a single combined output depth map 122 from the depth maps. The depth map conversion unit 100 also includes a confidence value generation unit 110 for generating one or more confidence values ​​112 associated with each distance or point on the depth map or associated with a portion or segment of the depth map. As used herein, the term "confidence value" or "likelihood value" is intended to include any value that quantifies the accuracy or veracity of a given parameter and provides a way to communicate its certainty or reliability. In this embodiment, the value is associated with the confidence of a distance measurement or value, such as a distance value, associated with the depth map. The value may be expressed in any selected range, preferably in the range of 0 to 1, with zero representing the lowest or minimum confidence level or value and one representing the highest or maximum confidence level or value. Furthermore, the confidence value may be expressed as a distance or distance range for a given depth. The confidence interval can be determined by a statistical analysis of the data, which can take into account the spread of depth values ​​from the depth map from various depth cues within a given region of the composite depth map, or their variation over time.

[0047] See FIG. 5. The depth map conversion unit 100 can use a regional convolutional neural network (CNN) based encoder-decoder architecture 114 that imports multiple input data streams 116, such as depth maps, and then processes the depth map data using a series of CNN filters or stages. The CNN filters can be placed at the input as an encoder stage or a series of CNN filters 118A. In this case, the data on the depth map is downsampled so that the distance and image data of the best or highest quality pixel or image segment is small. The data is then upsampled in a decoder stage of the CNN filter 118B that uses a series of aligned CNN filters, and the data is combined with other data from the input side to form or create a combined image, such as a single combined depth map 122. The encoder-decoder CNN architecture 114 removes noise from the input data, which helps to generate more accurate output data, such as a single composite depth map 122. The encoder-decoder CNN architecture 114 can also have a parallel upsampling or decoding stage of the CNN filter 18C, which also upsamples the input data and removes the associated noise. The data can then be processed through a Softmax function 124 to generate or create a confidence value 112. As is known, the Softmax function is a multi-dimensional generalization of the logistic function and can be used in multinomial logistic regression as the final activation function of a neural network to normalize the output of the network into a probability distribution over the predicted output classes. In this embodiment, the output of the Softmax function can be used to calculate the probability that the depth value for a particular point in the depth map is accurate. The encoder-decoder CNN architecture 114 merges the input depth maps with a probabilistic approach that minimizes or reduces noise in a single combined depth map 122 and estimates the true distance value associated therewith better than the data contained in each individual input depth map.In some other embodiments, the estimates of 112, 118A may also take into account any reliability data generated by 72A, 72B, 76A, 76B and 92 during processing. The encoder-decoder CNN architecture 114 may be trained using expected outputs of samples in one or more training sets of input depth or distance cues or maps, and the resulting depth and likelihood maps may be analytically calculated from the input depth maps. Other methods of combining or recombining depth maps known in the art, such as Kalman filters, particle filters, etc., may also be utilized.

[0048] As mentioned above, noise in the input depth map can occur in a variety of ways. For example, with respect to autofocus mechanisms, cameras tend to follow the laws of physics and optics, such as focal length and lens mechanisms, to abruptly amplify measurement errors and thus invalidate estimates. The system and method of the present invention uses the depth maps 72A, 72B in conjunction with other depth maps to remove spurious or outlier readings. Additionally, (disparity) Noise in the computation of is generally related to the diversity of the observed environment. A large amount of unique features in an image strongly reduces the probability of ambiguity (e.g., sources of noise in this type of cue or depth map). When considered independently from other input data sources, (disparity) Depth sources using the fusion method do not have a means to resolve ambiguity. The system of the present invention can resolve this ambiguity by considering which other sources are estimating the same region of the image and discarding erroneous or unlikely possibilities. The disparity cue or depth map is computationally complex because it depends on several noisy parameters, such as accurate measurements of actuated camera motion and knowledge of the geometric relationship between the left and right imaging sensors 58A, 58B. The fusion approach of the present invention allows the system 10 to reduce the effect of noise in this cue or depth map and produce a single composite depth map that is less noisy than if the input sources or depth maps were considered individually.

[0049] FIG. 6 is an example of a composite depth map 122 generated by the depth map conversion unit 100 according to the teachings of the present invention. The illustrated depth map 122 is formed by combining all of the input depth maps. The depth map 122 includes a scene 144 that includes a series of pixels forming an image within the scene. The image can have various hues that indicate different distances or depths from the viewpoint. The current scene 144 is represented in grayscale, but other colors can be used. Lighter hues 146 can represent pixels or segments of the image within the scene 144 that are closer to the viewpoint, and darker hues 148 can represent pixels or segments of the image that are farther from the viewpoint. Thus, pixels in the depth map are associated with distance values, and the depth map generator can also generate confidence values ​​112 associated with each distance value. Thus, each point or pixel of the depth map can have a depth value and a confidence value associated with it. The confidence values ​​can be stored separately within the system.

[0050] Referring again to FIG. 2, the depth map 122 and the confidence value 112 generated by the depth map conversion unit 100 are introduced to the control unit 26. The distance value and the confidence value are used by the control unit 26 to control the movement of the cameras 44a, 44b and / or the robot arm 42 of the robot subsystem 20. The confidence value provides a reasonable degree of confidence for the distances shown in the depth map 122 that when the surgeon moves the robot arm, for example, the arm will not touch a surface before or after the distance measurement. For delicate surgical procedures, having confidence in the distance values ​​in the depth map is important since the surgeon needs to know if the instructions sent to the robot arm and camera are accurate. The depth map is important to automatically warn or prevent the surgeon from accidentally touching the surgical environment or anatomical structures, thus allowing the system to automatically traverse the surgical environment and interact with anatomical components without intervention by the surgeon. Furthermore, the depth perception subsystem can allow "guide rails" to be virtually placed in the surgical environment to aid in the control and / or direct movement of the robot subsystem. Similarly, depth maps allow the surgical environment to be augmented or virtual objects to be more accurately placed within the environment (e.g., overlaying a preoperative scan of the patient's vitals on the surgical site, etc.), as disclosed or disclosed by way of example in International Patent Application No. PCT / US2020 / 059137, the contents of which are incorporated herein by reference. Depth maps can also be used with computer vision and artificial intelligence to help identify anatomical and abnormal structures. Additionally, depth maps can be used in combination with advanced sensory information (e.g., multi-wavelength images to detail the vasculature) or patient images (e.g., MRI, CAT scans, etc.) to create rich three-dimensional maps that surgeons can use to plan future procedures.

[0051] It will thus be seen that the present invention efficiently attains the objects set forth above, among those objects which will become apparent from the foregoing description. Because certain changes can be made in the above-described construction without departing from the scope of the invention, it is intended that all matter contained in the above description or shown in the accompanying drawings be interpreted as illustrative and not limiting.

[0052] It will be understood that the claims are intended to cover all of the generic and specific features of the invention that have been individually described, as well as any statements of the scope of the invention that may be omitted as a matter of language.

[0053] Having described the invention, what is claimed as new and desired to be protected by Letters Patent is set forth in the appended claims.

Claims

1. a robotic subsystem having a camera assembly having first and second cameras for generating image data; A computing unit; Equipped with The computing unit is a processor for processing the image data; a control unit for controlling the robotic subsystem; a depth perception subsystem for receiving the image data generated by the first and second cameras, generating two different types of depth maps based on the image data, selected from an autofocus depth map, a parallax depth map, or a disparity depth map, and converting the two different types of depth maps into a single composite depth map having associated distance data; having Surgical robotic system.

2. The robotic subsystem further comprises: a plurality of robot arms; and a motor unit for controlling movement of the plurality of robot arms and the camera assembly. The surgical robot system of claim 1 .

3. The surgical robotic system of claim 2 , wherein the control unit controls one of the camera assembly or the robotic arm using the distance data associated with the single composite depth map.

4. The depth perception subsystem further comprises: a depth map conversion unit for receiving the two different types of depth maps and converting the depth maps into the single composite depth map. The surgical robot system of claim 1 .

5. The surgical robot system of claim 4 , wherein the depth map conversion unit uses a region-specific convolutional neural network (R-CNN) technique to generate the single composite depth map.

6. The first and second cameras each include an image sensor for receiving optical data and responsively generating said image data; a lens and optical system having one or more lens elements optically coupled to the image sensor for focusing the optical data onto the image sensor; an autofocus mechanism associated with the lens and optical system that automatically adjusts the one or more lens elements and generates autofocus data; Equipped with The surgical robot system of claim 1 .

7. The depth perception subsystem includes: a first autofocus conversion unit for receiving the autofocus data from the first camera and converting the autofocus data into a first camera autofocus depth map; a second autofocus conversion unit for receiving the autofocus data from the second camera and converting the autofocus data into a second camera autofocus depth map; Equipped with The surgical robot system of claim 6.

8. The depth perception subsystem further comprises: a first disparity transformation unit for receiving image data from the first camera and transforming the image data into a first camera disparity depth map; a second disparity transformation unit for receiving image data from the second camera and transforming the image data into a second camera disparity depth map; Equipped with The surgical robot system of claim 7.

9. The depth perception subsystem further comprises: a disparity transformation unit for receiving image data from the first camera and image data from the second camera and for generating the disparity depth map in response thereto. The surgical robot system of claim 8.

10. The depth perception subsystem includes: a first autofocus conversion unit for receiving the autofocus data from the first camera and converting the autofocus data into a first camera autofocus depth map; a second autofocus conversion unit for receiving the autofocus data from the second camera and converting the autofocus data into a second camera autofocus depth map; a first disparity transformation unit for receiving image data from the first camera and transforming the image data into a first camera disparity depth map; a second disparity transformation unit for receiving image data from the second camera and transforming the image data into a second camera disparity depth map; a disparity transformation unit for receiving image data from the first camera and for receiving image data from the second camera and for generating a disparity depth map in response thereto; comprising one or more of: The surgical robot system of claim 6.

11. 11. The surgical robot system of claim 10, wherein each of the first and second parallax units is configured to acquire first and second successive images in the image data and measure an amount that each portion of the first image moves relative to the second image.

12. Each of the first and second cameras generates position data, and each of the first and second parallax conversion units: a separation unit for receiving the image data from each of the cameras, dividing the image data into a plurality of segments, and generating shifted image data in response to the plurality of segments; a movement determination unit for receiving the position data from each of the cameras and responsively generating camera movement data indicative of the positions of the cameras; a distance transformation unit for receiving the image data and the camera movement data and transforming the image data and the camera movement data into the respective disparity depth maps; Equipped with The surgical robotic system of claim 11.

13. The surgical robotic system of claim 12 , wherein the distance transformation unit uses a region-specific convolutional neural network (R-CNN) technique to generate each of the disparity depth maps.

14. The surgical robot system of claim 10 , wherein the image difference conversion unit analyzes image difference between an image in the image data received from the first camera and an image in the image data received from the second camera.

15. 15. The surgical robotic system of claim 14, wherein the image difference between the images from the first and second cameras is determined using a Layered Local Convolutional Neural Network (R-CNN) technique.

16. The depth perception subsystem further comprises: a depth map conversion unit configured to receive the first camera autofocus depth map, the second camera autofocus depth map, the first camera disparity depth map, the second camera disparity depth map, and the disparity depth map, form a received depth map, and convert the received depth map into the single composite depth map. The surgical robotic system of claim 10.

17. 17. The surgical robotic system of claim 16, wherein the depth map conversion unit uses a regional convolutional neural network (R-CNN) based encoder-decoder architecture to generate the single composite depth map.

18. Each point in each of the received depth maps has a distance value associated with it, and the depth map conversion unit a depth map generation unit for receiving the received depth maps and converting the received depth maps into the single composite depth map; a confidence value generation unit for generating, from the received depth map, a confidence value associated with each of the distance values ​​associated with each point of the single composite depth map; Equipped with the confidence value indicates a confidence in the distance values ​​associated with the single composite depth map.

20. The surgical robotic system of claim 17.

19. 13. A method of generating a depth map from image data in a surgical robotic system, comprising the steps of: providing a robotic subsystem having a camera assembly having first and second cameras for generating image data; generating two different types of depth maps based on image data from the first and second cameras, selected from an autofocus depth map, a parallax depth map, or a disparity depth map; converting the two different types of depth maps into a single composite depth map having distance data associated therewith; controlling the camera assembly based on the distance data in the single composite depth map; The method according to claim 1,

20. the robotic subsystem further comprising a plurality of robotic arms and a motor unit for controlling movement of the plurality of robotic arms and the camera assembly; The method further comprises controlling the robot arm based on the distance data in the single composite depth map.

20. The method of claim 19.

21. Each of the first and second cameras an image sensor for receiving optical data and generating said image data in response thereto; a lens and optical system having one or more lens elements optically coupled to the image sensor for focusing the optical data onto the image sensor; an autofocus mechanism associated with the lens and optical system for automatically adjusting the one or more lens elements and generating autofocus data; Equipped with 20. The method of claim 19.

22. The method further comprises: converting the autofocus data from the first camera into a first camera autofocus depth map; The autofocus data from the second camera is converted into a second camera autofocus depth map. Step 2:

22. The method of claim 21 , comprising:

23. The method further comprises: converting the image data from the first camera into a first camera disparity depth map; converting the image data from the second camera into a second camera disparity depth map; 23. The method of claim 22, comprising:

24. 24. The method of claim 23, further comprising generating a disparity depth map from the image data from the first camera and the image data from the second camera.

25. The method further comprises: converting the autofocus data from the first camera into a first camera autofocus depth map; converting the autofocus data from the second camera into a second camera autofocus depth map; converting the image data from the first camera into a first camera disparity depth map; converting the image data from the second camera into a second camera disparity depth map; generating a disparity depth map from the image data from the first camera and the image data from the second camera; 22. The method of claim 21, comprising one or more of:

26. Transforming the image data from the first camera into a first camera disparity depth map includes: acquiring first and second successive images within the image data; measuring the amount that each portion of the first image moves relative to the second image; 26. The method of claim 25, comprising:

27. Transforming the image data from the second camera into a second camera disparity depth map includes: acquiring first and second successive images within the image data; measuring the amount that each portion of the first image moves relative to the second image; 27. The method of claim 26, comprising:

28. The first camera generates position data, and the method further comprises: dividing the image data from the first camera into a plurality of segments and generating shifted image data responsive to the plurality of segments; generating from the first camera movement data indicative of a position of the camera in response to the position data; converting the image data and the camera motion data into the first disparity depth map; 26. The method of claim 25, comprising:

29. The second camera generates position data, and the method further comprises: dividing the image data from the second camera into a plurality of segments and generating shifted image data responsive to the plurality of segments; generating from the second camera movement data indicative of a position of the camera in response to the position data; converting the image data and the camera movement data into the second disparity depth map; 30. The method of claim 28, comprising:

30. 26. The method of claim 25, wherein generating the disparity depth map from the image data from the first camera and the image data from the second camera further comprises analyzing the disparity between an image in the image data received from the first camera and an image in the image data received from the second camera.

31. The method further comprises: receiving the first camera autofocus depth map, the second camera autofocus depth map, the first camera disparity depth map, the second camera disparity depth map, and the disparity depth map to form a received depth map; converting the received depth maps into the single composite depth map; 26. The method of claim 25, comprising:

32. Each point in each of the receive depth maps has a distance value associated therewith, the method further comprising: generating from the received depth maps a confidence value associated with each of the distance values ​​associated with each point of the single composite depth map; the confidence value indicates a confidence in the distance values ​​associated with the single composite depth map.

32. The method of claim 31.

33. 11. The surgical robot system of claim 10, wherein the two different types of depth maps include at least two of the first camera autofocus depth map, the second camera autofocus depth map, the first camera parallax depth map, the second camera parallax depth map, and a disparity depth map.

34. 26. The method of claim 25, wherein the two different types of depth maps include at least two of the first camera autofocus depth map, the second camera autofocus depth map, the first camera disparity depth map, the second camera disparity depth map, and a disparity depth map.

Images