Systems and methods for creating virtual boundaries in robotic surgical systems

Abstract

A system for clinical workspace augmentation includes an augmented reality device. The augmented reality headset comprises: an imaging device configured to capture an image of a real-world environment; a display configured to display the composite view; a processor; and a memory. The memory includes instructions stored on the memory that, when executed by the processor, cause the system to identify input objects in the captured real-world environment; tracking a path of the input object in the captured real-world environment; generating a boundary region based on a tracking path of the input object in the captured real-world environment; rendering a coverage map including a boundary region; and displaying a composite view of the captured real-world environment and overlay on a display of the augmented reality headset.

Description

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 432,432, filed on December 14, 2022, the entire content of which is incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to systems and methods for workspace augmentation. In particular, this disclosure relates to a virtual or augmented reality simulation setup for a robotic surgery system having a boundary zone. Background Art

[0003] Robotic surgery systems are used in minimally invasive medical procedures. Some robotic surgery systems include a surgical console that controls surgical arms and a surgical instrument having an end effector (e.g., a clamp or grasping instrument) coupled to and actuated by the arms. In operation, the arms move to a position above the patient and then guide the surgical instrument through a small incision via a surgical port or natural orifice of the patient to position the end effector at a working site within the patient.

[0004] However, in some conventional systems, when the arms move around, the arms may move "outside" the physical or virtual environment, which degrades the user experience. Summary of the Invention

[0005] In accordance with aspects of this disclosure, a computer-implemented method for clinical workspace augmentation is presented. The method includes: capturing, by an imaging device, a real-world environment including an input object; identifying the input object in the captured real-world environment; tracking the path of the input object in the captured real-world environment; generating at least one boundary zone based on the tracked path of the input object in the captured real-world environment; rendering an overlay map including the at least one boundary zone; and displaying, on a first display of an augmented reality device, a composite view of the captured real-world environment and the overlay map.

[0006] In one aspect of this disclosure, the input object may include a user's hand and / or an input controller.

[0007] In another aspect of this disclosure, the path of the input object may be tracked based on gestures, pose recognition, eye gaze tracking, and / or voice commands.

[0008] In yet another aspect of this disclosure, the input object in the captured real-world environment may be identified based on object detection.

[0009] In another aspect of the present disclosure, object detection can be performed by: generating a spatial grid based on the captured real-world environment; determining the boundaries of the input object based on the spatial grid; and identifying the input object based on a machine learning model, wherein the determined boundaries are provided as input to the machine learning model.

[0010] In yet another aspect of the present disclosure, the method can further include modifying at least one boundary region by: selecting a generated shape configured to be part of at least one boundary region; positioning the generated shape in the captured real-world environment; receiving input to transform the generated shape by moving, rotating, and / or scaling the generated shape; and adding the transformed generated shape to at least one boundary region.

[0011] In yet another aspect of the present disclosure, the method can further include modifying at least one boundary region by: selecting a plurality of waypoints to be part of at least one boundary region; receiving input to position the plurality of waypoints; connecting the waypoints from a second boundary region; and adding the second boundary region to at least one boundary region.

[0012] In another aspect of the present disclosure, the method can further include: receiving 3D volume data of the robotic arm of the robotic surgery system in the real-world environment; receiving the spatial position of the robotic arm; determining, based on the spatial position and the 3D volume data, that the robotic arm crosses the boundary of at least one boundary region; and preventing the robotic arm from entering at least one boundary region.

[0013] In yet another aspect of the present disclosure, the method can further include: providing force feedback to an input device of the surgical console of the robotic surgery system in response to determining that the robotic arm crosses the boundary of at least one boundary region.

[0014] In another aspect of the present disclosure, the method can further include: displaying, on a second display, a composite view of at least one boundary region and the real-world environment from the perspective of the second display. The perspective of the second display is different from the perspective of the first display.

[0015] In accordance with aspects of the present disclosure, a system for clinical workspace augmentation including an augmented reality device is presented. The augmented reality device includes: an imaging device configured to capture an image of a real-world environment; a display configured to display a composite view; a processor; and a memory. The memory includes instructions stored thereon that, when executed by the processor, cause the system to identify an input object in the captured real-world environment; track the path of the input object in the captured real-world environment; generate at least one boundary zone based on the tracked path of the input object in the captured real-world environment; render an overlay including the at least one boundary zone; and display a composite view of the captured real-world environment and the overlay on a first display of the augmented reality device.

[0016] In one aspect of the present disclosure, the input object may include the user's hand and / or an input controller.

[0017] In another aspect of the present disclosure, the path of the input object may be tracked based on gestures, pose recognition, eye gaze tracking, and / or voice commands.

[0018] In yet another aspect of the present disclosure, the input object in the captured real-world environment may be identified based on object detection.

[0019] In another aspect of the present disclosure, object detection may be performed by generating a spatial grid based on the captured real-world environment; determining the boundaries of the input object based on the spatial grid; and identifying the input object based on a machine learning model, wherein the determined boundaries are provided as an input to the machine learning model.

[0020] In another aspect of the present disclosure, the instructions, when executed by the processor, may further cause the system to modify at least one boundary zone by: selecting a generated shape configured to be part of the at least one boundary zone; positioning the generated shape in the captured real-world environment; receiving an input to transform the generated shape by moving, rotating, and / or scaling the generated shape; and adding the transformed generated shape to the at least one boundary zone.

[0021] In yet another aspect of the present disclosure, the instructions, when executed by the processor, may further cause the system to modify at least one boundary zone by selecting a plurality of waypoints to be part of the at least one boundary zone; receiving an input to position the plurality of waypoints; connecting the waypoints from a second boundary zone; and adding the second boundary zone to the at least one boundary zone.

[0022] In one aspect of the present disclosure, when executed by a processor, these instructions may further cause the system to: receive 3D volume data of the robotic arm of a robotic surgery system in a real-world environment; receive the spatial position of the robotic arm; determine that the robotic arm crosses the boundary of at least one boundary zone based on the spatial position and the 3D volume data; and prevent the robotic arm from entering at least one boundary zone.

[0023] In another aspect of the present disclosure, when executed by a processor, these instructions may further cause the system to provide force feedback to an input device of a surgical console of the robotic surgery system in response to determining that the robotic arm crosses the boundary of at least one boundary zone.

[0024] In accordance with aspects of the present disclosure, a non-transitory computer-readable medium is presented. The non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform a method including capturing a real-world environment including an input object by an imaging device; identifying the input object in the captured real-world environment; tracking the path of the input object in the captured real-world environment; generating at least one boundary zone based on the tracked path of the input object in the captured real-world environment; rendering an overlay map including the at least one boundary zone; and displaying a composite view of the captured real-world environment and the overlay map on a first display of an augmented reality device. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Aspects of the present disclosure are described herein with reference to the accompanying drawings, in which:

[0026] Figure 1 is a schematic illustration of a robotic surgery system according to one aspect of the present disclosure, the robotic surgery system including a control tower, a console, and one or more surgical robotic arms;

[0027] Figure 2 is according to one aspect of the present disclosure Figure 1 a perspective view of a surgical robotic arm of a robotic surgery system;

[0028] Figure 3 is a perspective view of a placement arm according to one aspect of the present disclosure, the placement arm having Figure 1 a surgical robotic arm of a robotic surgery system;

[0029] Figure 4 is according to one aspect of the present disclosure Figure 1 a schematic diagram of a computer architecture of a robotic surgery system;

[0030] Figure 5 is according to an embodiment of the present disclosure Figure 1 a schematic illustration of a robotic surgery system that is positioned around an operating table;

[0031] Figure 6 is a flowchart of a computer-implemented method for clinical workspace enhancement according to one aspect of the present disclosure; and

[0032] Figure 7 is an image of a composite view of a clinically enhanced workspace showing boundaries according to one aspect of the present disclosure. DETAILED DESCRIPTION

[0033] Aspects of the robotic surgical system disclosed herein are described in detail with reference to the accompanying drawings, in which like reference numerals represent the same or corresponding elements in each of the several views. As used herein, the term "distal" refers to the portion of the robotic surgical system and / or the surgical instrument coupled thereto that is closer to the patient, while the term "proximal" refers to the portion that is farther from the patient.

[0034] The term "application" may include a computer program designed to perform functions, tasks, or activities for the benefit of a user. An application may refer to, for example, software running locally or remotely, software running as a stand-alone program or in a web browser, or other software understood by those skilled in the art as an application. An application may run on a controller or on a user device, including, for example, on a mobile device, a personal computer, or a server system.

[0035] As will be described in detail below, the present disclosure relates to a robotic surgical system that includes a surgical console, a control tower, and one or more mobile carts having surgical robotic arms coupled to mounting arms. The surgical console receives user input through one or more interface devices, and the user input is interpreted by the control tower as a movement command for moving the surgical robotic arms. The surgical robotic arms include a controller configured to process the movement command and configured to generate torque commands for activating one or more actuators of the robotic arms, and the one or more actuators in turn move the robotic arms in response to the movement command.

[0036] Referring Figure 1 , the robotic surgical system 10 generally includes an augmented reality device 600 (e.g., an AR headset), a control tower 20 that is connected to all components of the robotic surgical system 10, including a surgical console 30 and one or more robotic arms 40. Each robotic arm 40 includes a surgical instrument 50 removably coupled thereto. Each robotic arm 40 is also coupled to a mobile cart 60.

[0037] The augmented reality device 600 is configured to display a composite view and generally includes a controller 602, an imaging device 604, and a display 608. The controller 602 includes: a memory configured to have instructions stored thereon; and a processor configured to execute those instructions. The augmented reality device 600 can overlay virtual objects, such as a virtual robotic arm ( Figure 7 ). For example, the augmented reality device 600 can provide a user with suggestions on how to position various virtual objects to assist in setting up an operating room for a surgery. It is envisioned that the augmented reality device 600 can be a full virtual reality headset (such as the Quest from of Menlo Park, California) or an augmented reality (mixed reality) headset (such as the from of Seattle, Washington).

[0038] The surgical instrument 50 is configured for use during a minimally invasive surgical procedure. In various aspects, the surgical instrument 50 can be configured for an open surgical procedure. In various aspects, the surgical instrument 50 can be an endoscope (such as the endoscope camera 51) configured to provide video feedback to a user. In additional aspects, the surgical instrument 50 can be an electrosurgical clamp configured to seal tissue by pressing the tissue between jaw members and applying an electrosurgical current thereto. In yet additional aspects, the surgical instrument 50 can be a surgical stapler that includes a pair of jaws configured to grasp and clamp tissue while deploying a plurality of tissue fasteners (e.g., staples) and cutting the stapled tissue.

[0039] One of the robotic arms 40 can include an endoscope camera 51 configured to capture video of the surgical site. The endoscope camera 51 can be a stereoscopic endoscope configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The endoscope camera 51 is coupled to a video processing device 56, which can be disposed within the control tower 20. The video processing device 56 can be any of the computing devices described below, configured to receive video feedback from the endoscope camera 51, perform image processing based on the depth estimation algorithm of the present disclosure, and output the processed video stream.

[0040] The surgical console 30 includes a first display 32 and a second display 34. The first display shows video feedback of the surgical site provided by the camera 51 of the surgical instrument 50 disposed on the robotic arm 40, and the second display shows a user interface for controlling the robotic surgical system 10. The first display 32 and the second display 34 are touchscreens that allow for the display of different graphical user inputs.

[0041] The surgical console 30 further includes a plurality of user interface devices, such as a foot pedal 36 and a pair of handle controllers 38a and 38b, which are used by a user to remotely control the robotic arm 40. The surgical console further includes an armrest 33, which is used to support the clinician's arm when operating the handle controllers 38a and 38b.

[0042] The control tower 20 includes a display 23 (which may be a touch screen) and outputs on a graphical user interface (GUI). The control tower 20 also serves as an interface between the surgical console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arm 40, such as to move the robotic arm 40 and the corresponding surgical instrument 50 based on a set of programmable instructions and / or input commands from the surgical console 30 in such a way that the robotic arm 40 and the surgical instrument 50 perform a desired sequence of movements in response to inputs from the foot pedal 36 and the handle controllers 38a and 38b.

[0043] Each of the control tower 20, the surgical console 30, and the robotic arm 40 includes a corresponding computer 21, 31, 41. The computers 21, 31, 41 are interconnected with each other by using any suitable communication network based on a wired or wireless communication protocol. As used herein, the term "network", whether plural or singular, refers to a data network, including but not limited to the Internet, intranet, wide area network, or local area network, and is not limited to the entire scope of the definition of the communication network covered by this disclosure. Suitable protocols include but are not limited to Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol / Internet Protocol (UDP / IP), and / or Datagram Congestion Control Protocol (DCCP). Wireless communication can be achieved through one or more wireless configurations, such as radio frequency, light, Wi-Fi, Bluetooth (an open wireless protocol used to exchange data from fixed and mobile devices over short lengths of radio waves), creating a personal area network (PAN), (a specification of a set of advanced communication protocols using small low-power digital radios based on the IEEE 122.15.4-2003 wireless personal area network (WPAN) standard).

[0044] Computers 21, 31, 41 may include any suitable processor (not shown) operably connected to a memory (not shown), which may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuitry) adapted to execute the operations, calculations, and / or instruction sets described in this disclosure, including but not limited to a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor, and combinations thereof. Those skilled in the art will understand that the processor may be replaced by any logical processor (e.g., control circuitry) adapted to execute the algorithms, calculations, and / or instruction sets described herein.

[0045] Reference Figure 2 , each robotic arm 40 may include a plurality of links 42a, 42b, 42c that are interconnected at joints 44a, 44b, 44c, respectively. Joint 44a is configured to secure robotic arm 40 to a movable cart 60 and define a first longitudinal axis. Reference Figure 3 , the movable cart 60 includes a lift 61 and a mounting arm 62 that provides a base for mounting robotic arm 40. The lift 61 allows the mounting arm 62 to move vertically. The movable cart 60 also includes a display 69 for displaying information about robotic arm 40.

[0046] The mounting arm 62 includes a first link 62a, a second link 62b, and a third link 62c that provide lateral maneuverability of robotic arm 40. Links 62a, 62b, 62c are interconnected at joints 63a and 63b, and each joint may include an actuator (not shown) for rotating links 62b and 62b relative to each other and relative to link 62c. In particular, links 62a, 62b, 62c are capable of moving in their respective lateral planes that are parallel to each other, thereby allowing robotic arm 40 to extend relative to a patient (e.g., an operating table). In various aspects, robotic arm 40 may be coupled to an operating table (not shown). The mounting arm 62 includes a control device 65 for adjusting the movement of links 62a, 62b, 62c and lift 61.

[0047] The third link 62c includes a rotatable base 64 that has two degrees of freedom. In particular, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a is capable of rotating about a first fixed arm axis perpendicular to the plane defined by the third link 62c, and the second actuator 64b is capable of rotating about a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b allow for a full three-dimensional orientation of the robotic arm 40.

[0048] The actuator 48b of joint 44b is coupled to joint 44c via a belt 45a, and joint 44c is in turn coupled to joint 46c via a belt 45b. Joint 44c may include a transfer case that couples belts 45a and 45b such that the actuator 48b is configured to rotate each of links 42b, 42c and the holder 46 relative to one another. More specifically, links 42b, 42c and the holder 46 are passively coupled to the actuator 48b that forces rotation about a pivot point “P” that is located at the intersection of a first axis defined by link 42a and a second axis defined by the holder 46. Thus, the actuator 48b controls the angle θ between the first axis and the second axis, thereby allowing for the orientation of the surgical instrument 50. Due to the interconnection of links 42a, 42b, 42c and the holder 46 via belts 45a and 45b, the angles between links 42a, 42b, 42c and the holder 46 are also adjusted to achieve the desired angle θ. In various aspects, some or all of joints 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0049] Joints 44a and 44b include actuators 48a and 48b that are configured to drive joints 44a, 44b, 44c relative to one another via a series of belts 45a and 45b or other mechanical linkages such as drive rods, cables, or levers, etc. In particular, the actuator 48a is configured to rotate the robotic arm 40 about a longitudinal axis defined by link 42a.

[0050] Reference Figure 2 , the robotic arm 40 further includes a holder 46 that defines a second longitudinal axis and is configured to receive an instrument drive unit (IDU) 52 ( Figure 1)。The IDU 52 is configured to be coupled to the actuation mechanisms of the surgical instrument 50 and the camera 51 and is configured to move (e.g., rotate) and actuate the instrument 50 and / or the camera 51. The IDU 52 transfers the actuation force from its actuator to the surgical instrument 50 to actuate components of the surgical instrument 50 (e.g., the end effector). The holder 46 includes a sliding mechanism 46a that is configured to move the IDU 52 along a second longitudinal axis defined by the holder 46. The holder 46 also includes a joint 46b that rotates the holder 46 relative to the link 42c. During an endoscopic procedure, the instrument 50 can be inserted through the endoscopic port 55 provided by the holder 46 ( Figure 3 )。

[0051] The robotic arm 40 also includes a plurality of manual override buttons 53 ( Figure 1 and Figure 5 ) that are provided on the IDU 52 and the mounting arm 62 and can be used in a manual mode. A user can press one or more of these buttons 53 to move the components associated with the buttons 53.

[0052] Reference Figure 4 , each of the computers 21, 31, 41 of the robotic surgery system 10 can include a plurality of controllers that can be implemented in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and a safety observer 21b. The controller 21a receives data from the computer 31 of the surgical console 30 regarding the current position and / or orientation of the handle controllers 38a and 38b and the status of the foot pedal 36 and other buttons. The controller 21a processes these input positions to determine the desired drive commands for each joint of the robotic arm 40 and / or the IDU 52 and transmits these desired drive commands to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by the encoders of the actuators 48a and 48b and uses this information to determine a force feedback command that is transmitted back to the computer 31 of the surgical console 30 to provide haptic feedback through the handle controllers 38a and 38b. The safety observer 21b validates the data going into and coming out of the controller 21a and, if an error in data transmission is detected, notifies the system fault handler to place the computer 21 and / or the robotic surgery system 10 in a safe state.

[0053] The computer 41 includes a plurality of controllers, namely, the cart main controller 41a, the mounting arm controller 41b, the robotic arm controller 41c, and the instrument drive unit (IDU) controller 41d. The cart main controller 41a receives and processes joint commands from the controller 21a of the computer 21 and transmits them to the mounting arm controller 41b, the robotic arm controller 41c, and the IDU controller 41d. The cart main controller 41a also manages instrument exchange and the overall status of the mobile cart 60, the robotic arm 40, and the IDU 52. The cart main controller 41a also transmits the actual joint angles back to the controller 21a.

[0054] The mounting arm controller 41b controls each of the joints 63a and 63b, as well as the rotatable base 64 of the mounting arm 62, and calculates the desired motor movement commands (e.g., motor torque) for the pitch axis and controls the brakes. The robotic arm controller 41c controls each of the joints 44a and 44b of the robotic arm 40 and calculates the desired motor torques required for gravity compensation, friction compensation, and closed-loop position control of the robotic arm 40. The robotic arm controller 41c calculates movement commands based on the calculated torques. The calculated motor commands are then transmitted to one or more of the actuators 48a and 48b in the robotic arm 40. The actual joint positions are then transmitted back to the robotic arm controller 41c by the actuators 48a and 48b.

[0055] The IDU controller 41d receives the desired joint angles (such as wrist and jaw angles) of the surgical instrument 50 and calculates the desired current of the motors in the IDU 52. The IDU controller 41d calculates the actual angles based on the motor positions and transmits the actual angles back to the cart main controller 41a.

[0056] The manipulator 40 is controlled in response to the pose of a handle controller (e.g., handle controller 38a) that controls the manipulator 40, and this pose is transformed into a desired pose of the manipulator 40 by the hand-eye transformation function executed by the controller 21a. The hand-eye function and other functions described herein are implemented in software executable by the controller 21a or any other suitable controller described herein. The pose of a handle controller 38a can be implemented as a coordinate position and a roll-pitch-yaw ("RPY") orientation relative to a coordinate reference system (which is fixed to the surgical console 30). The desired pose of the instrument 50 is with respect to a fixed system on the manipulator 40. Then, the pose of the handle controller 38a is scaled by a scaling function executed by the controller 21a. In various aspects, by the scaling function, the coordinate position is reduced, and the orientation is amplified. In addition, the controller 21a also executes a clutch function, thus disengaging the handle controller 38a from the manipulator 40. In particular, if certain movement limits or other bounds are exceeded, the controller 21a stops transmitting movement commands from the handle controller 38a to the manipulator 40 and substantially acts like a virtual clutch mechanism, e.g., restricting mechanical input from affecting mechanical output.

[0057] The desired pose of the manipulator 40 is based on the pose of the handle controller 38a and is then passed through an inverse kinematics function executed by the controller 21a. The inverse kinematics function calculates the angles of the joints 44a, 44b, 44c of the manipulator 40 that achieve the scaled and adjusted pose input by the handle controller 38a. The calculated angles are then passed to the manipulator controller 41c, which includes a joint axis controller with a proportional derivative (PD) controller, a friction estimator module, a gravity compensator module, and a bilateral saturation block configured to limit the commanded torque of the motors of the joints 44a, 44b, 44c.

[0058] The video processing device 56 is configured to process video feedback from the endoscope camera 51 and output the processed video stream on the first display 32 of the surgical console 30 and / or the display 23 of the control tower 20.

[0059] Reference Figure 5 , the robotic surgical system 10 is installed around the operating table 90. The system 10 includes mobile carts 60a-d, which can be numbered from "1" to "4". During installation, each of the carts 60a-d is positioned around the operating table 90. The position and orientation of the carts 60a-d depend on multiple factors, such as the placement of multiple access ports 55a-d, which in turn depend on the surgery being performed. Once the port placement is determined, the access ports 55a-d are inserted into the patient's body, and the carts 60a-d are positioned to insert the instrument 50 and the laparoscopic camera 51 into the corresponding ports 55a-d.

[0060] Figure 6 A flowchart is shown that illustrates various operations of an exemplary method for enhancing a clinical workspace having a boundary region. Those skilled in the art will understand that one or more operations of method 650 may be performed, repeated, and / or omitted in any suitable order without departing from the scope of this disclosure. In various aspects, the illustrated method 650 may operate in one or more controllers (i.e., controllers 21a, 31a, 41a, 602), in a remote device, or in another server or system. These operations may be implemented as software or instructions executable by a controller. Although the operations of method 650 are described below with respect to the controller 602 ( Figure 1 ) of the augmented reality device 600 ( Figure 1 ), other components and controllers of system 10 may be used. With respect to Figure 7 Method 650 is described, and the figure shows a composite view of the clinical workspace.

[0061] Initially, at step 652, the controller 602 causes the robotic surgical system 10 to capture the real-world environment via the imaging device 604 of the augmented reality device 600 ( Figure 1 ) or any other imaging device (e.g., coupled to a mobile device / tablet computer). The imaging device 604 may include a stereoscopic imaging device. The real-world environment includes the input object 702. The input object 702 may be the hand of a user (e.g., a clinician) and / or an input controller. The input object 702 is used to draw a virtual boundary, as described in further detail below. The input controller may include a set of sensors for tracking the position of the controller. For example, the tracking may include optical tracking using the imaging device 604 of the augmented reality device 600. The input controller may include a set of infrared LEDs located, for example, on a ring of the input controller. The imaging device 604 detects the LEDs, and then the controller 602 triangulates the position of the controller in space.

[0062] Next, at step 654, the controller 602 causes the robotic surgical system 10 to identify the input object 702 in the captured real-world environment. For example, the input object 702 may be identified as a hand. Edge detection and / or image segmentation may be used to detect real-world objects such as patients, users, the input object 702, and / or the operating table 90. For example, the controller 602 may extract the edges of the input object 702 in the captured image by detecting depth discontinuities, surface orientation discontinuities, and / or material property changes of the input object 702 in the captured image. The extracted edges may be used to determine the boundary of the input object 702. Then, the controller may identify the input object 702 as, for example, a hand based on the determined boundary.

[0063] In various aspects, an input object 702 in a captured real-world environment can be identified based on object detection. Object detection can be performed by generating a spatial grid based on the captured real-world environment, determining the boundaries of the object based on the spatial grid, and identifying the input object 702 based on a machine learning model. These determined boundaries can be provided as input to the machine learning model. The machine learning model can be trained on labeled images of an object (e.g., a hand).

[0064] Next, at step 656, the controller 602 enters a boundary generation mode during which the user creates a virtual boundary (e.g., Figure 7 a first boundary zone 740) that is used to track the movement of the robotic arm 40 and other objects. The boundary can be created by the robotic surgery system 10 to track the path 746 of the input object 702 in the captured real-world environment. For example, the user can virtually "draw" a custom trajectory by moving their hand, with the 3D trajectory following the movement of the input object 702. Tracking the path 746 of the object can be based on at least one of gesture recognition, pose recognition, eye gaze tracking, and / or voice commands, and can be done in 3D. The resulting path 746 can be adjusted using the input object 702. The path 746 can have one or more waypoints that can be moved by the input object 702 to modify the length, direction, shape, curvature, etc. of the path.

[0065] The controller 602 can cause the robotic surgery system 10 to detect the width and / or volume of the input object 702. For example, tracking the path 746 can include the volume and / or width of the user's hand.

[0066] Next, at step 658, the controller 602 causes the robotic surgery system 10 to generate a first boundary zone 740 ( Figure 7 ) based on the tracking path 746 of the input object 702 in the captured real-world environment. In the case where the boundary zone 740 is not drawn as a complete and / or closed shape, the boundary zone can be automatically connected to itself.

[0067] In various aspects, the user 704 can place the generated shapes 742 (e.g., cubes, spheres, etc.) and can move, scale, and / or rotate them into position to form a custom boundary zone 740 and / or modify an existing boundary zone 740. The controller 602 can cause the robotic surgery system 10 to modify the first boundary zone 740 by selecting the generated shapes 742 that are configured to be part of the first boundary zone; positioning the generated shapes 742 in the captured real-world environment; receiving input to transform the generated shapes 742 by moving, rotating, and / or scaling the generated shapes 742; and adding the transformed generated shapes 742 to the boundary zone. In various aspects, multiple boundary zones can be generated.

[0068] The user may place waypoints 750 to be connected to form a custom boundary region or modify an existing boundary region. The controller 602 may cause the robotic surgical system 10 to modify the first boundary region 740 by selecting a plurality of waypoints 750 that are part of the first boundary region 740. The controller 602 may receive input to locate the plurality of waypoints. The controller 602 may connect the waypoints from the second boundary region 760 and add the second boundary region 760 to the first boundary region 740.

[0069] In various aspects, object detection may be used to identify a person (e.g., a clinician), and the controller 602 may cause the robotic surgical system 10 to generate a moving boundary around the person (e.g., to prevent collisions with a bedside assistant).

[0070] Next, at step 660, the controller 602 causes the robotic surgical system 10 to render an overlay map including the first boundary region 740. At step 662, the controller 602 causes the robotic surgical system 10 to display on the display 608 of the augmented reality device 600 ( Figure 1 ) a composite view 700 of the captured real-world environment and the overlay map ( Figure 7 ).

[0071] One or more users may view the boundary region 740 via the augmented reality device 600 or another display and / or projection device. For example, a user (such as a bedside assistant) may see the area displayed on the display 608 of the augmented reality device 600 that is reserved for an anesthesiologist (or any other clinical staff), and the user knows to avoid the area based on the displayed boundary region 740. The boundary region 740 may be displayed as a grid, a holographic object, a transparent object, a translucent object, and / or a solid object. The boundary region 740 may be displayed as a color, a color gradient (changing color as an object approaches the boundary region 740), and / or a shadow.

[0072] The robotic surgical system 10 can prevent the robotic arm 40 from entering the boundary area via movement restrictions set on the motors controlling the robotic arm 40. In particular, the kinematic pipeline that provides movement commands based on user input is restricted by the boundary area 740, i.e., when a portion of the robotic arm 40 moves outside the boundary area 740, the robotic arm 40 can stop and / or slow down. In particular, the controller 602 can cause the robotic surgical system 10 to receive or access 3D volume data of the robotic arm of the robotic surgical system 10 in the real-world environment and receive or access the spatial position of the robotic arm 40. This can be achieved by sending the 3D volume data to the control tower 20 via Data Distribution Service (DDS) messages or another publish-subscribe communication protocol. Then, the main controller 21a can prevent the robotic arm 40 from entering the boundary area 740 based on the 3D volume data and apply force feedback to the input device at the surgical console 30 to resist control movement in that direction. For example, the controller 21a can cause the robotic surgical system 10 to determine that the robotic arm has crossed the boundary of the first boundary area 740 based on the spatial position and the 3D volume data and prevent the robotic arm from entering the first boundary area 740.

[0073] In addition to controlling the movement of the robotic arm 40, the controller 21a can also output an alarm in response to a portion of the robotic arm 40 moving outside the boundary area 740. The alarm can be audio or visual and can be displayed on the augmented reality device 600 and / or the displays 23, 32, and / or 34.

[0074] Figure 7 is an image of a composite view 700 that shows a clinical workspace augmentation of the boundary area 740. The user 704 is shown generating the 3D boundary area 740 using the method described herein. One or more mobile carts 60a-b can be positioned around the operating table 90. The robotic surgical system 10 can prevent the robotic arm 40 of the mobile carts 60a-b from entering the (multiple) boundary areas 740.

[0075] It will be understood that various modifications can be made to the aspects disclosed herein. Accordingly, the above description should not be construed as limiting, but merely as illustrative of the various aspects. Those skilled in the art will envision other modifications within the scope and spirit of the appended claims.

Claims

1. A computer-implemented method for clinical workspace enhancement, the method comprising: Capturing a real-world environment by an imaging device, wherein the real-world environment includes an input object; Identifying the input object in the captured real-world environment; Tracking the path of the input object in the captured real-world environment; Generating at least one boundary zone based on the tracked path of the input object in the captured real-world environment; Rendering an overlay map including the at least one boundary zone; and Displaying a composite view of the captured real-world environment and the overlay map on a first display of an augmented reality device.

2. The computer-implemented method according to claim 1, wherein, The input object includes at least one of a user's hand or an input controller.

3. The computer-implemented method according to claim 1, wherein, Tracking the path of the input object is based on at least one of gesture, pose recognition, eye gaze tracking, and / or voice commands.

4. The computer-implemented method according to claim 1, wherein, Identifying the input object in the captured real-world environment is based on object detection.

5. The computer-implemented method according to claim 4, wherein, The object detection is performed by: Generating a spatial grid based on the captured real-world environment; Determining the boundaries of the input object based on the spatial grid; And Identifying the input object based on a machine learning model, wherein the determined boundaries are provided as inputs to the machine learning model.

6. The computer-implemented method according to claim 1, further comprising modifying the at least one boundary zone by: Selecting a generated shape configured to be part of the at least one boundary zone; Positioning the generated shape in the captured real-world environment; Receiving an input to transform the generated shape by at least one of moving, rotating, or scaling the generated shape; and Adding the transformed generated shape to the at least one boundary zone.

7. The computer-implemented method according to claim 1, further comprising modifying the at least one boundary zone by: Selecting a plurality of waypoints to be part of the at least one boundary zone; Receiving an input to position the plurality of waypoints; Connecting the waypoints from a second boundary zone; and Adding the second boundary zone to the at least one boundary zone.

8. The computer-implemented method according to claim 1, further comprising: Receiving 3D volume data of a robotic arm of a robotic surgery system in the real-world environment; Receiving the spatial position of the robotic arm; Determining the boundaries of the robotic arm crossing the at least one boundary zone based on the spatial position and the 3D volume data; And Preventing the robotic arm from entering the at least one boundary zone.

9. The computer-implemented method according to claim 8, further comprising: Providing force feedback to an input device of a surgical console of the robotic surgery system in response to determining that the robotic arm crosses the boundaries of the at least one boundary zone.

10. The computer-implemented method according to claim 1, further comprising: Displaying a composite view of the at least one boundary zone and the real-world environment from the perspective of a second display on the second display, wherein the perspective of the second display is different from the perspective of the first display.

11. A system for clinical workspace enhancement, the system comprising: An augmented reality device, the augmented reality device comprising: An imaging device configured to capture an image of a real-world environment; A display configured to display a composite view; A processor; and A memory including instructions stored thereon that, when executed by the processor, cause the system to: Identify an input object in the captured real-world environment; Track the path of the input object in the captured real-world environment; Generate at least one boundary region based on the tracked path of the input object in the captured real-world environment; Render an overlay map including the at least one boundary region; and Display a composite view of the captured real-world environment and the overlay map on the display of the augmented reality device.

12. The system according to claim 11, wherein, The input object includes at least one of a user's hand or an input controller.

13. The system according to claim 11, wherein, Tracking the path of the input object is based on at least one of gesture or pose recognition.

14. The system according to claim 11, wherein, Identifying the input object in the captured real-world environment is based on object detection.

15. The system according to claim 14, wherein, The object detection is performed by: Generating a spatial grid based on the captured real-world environment; Determining the boundaries of the input object based on the spatial grid; And Identifying the input object based on a machine learning model, wherein the determined boundaries are provided as inputs to the machine learning model.

16. The system according to claim 11, wherein The instructions, when executed by the processor, further cause the system to modify the at least one boundary region by: Selecting a generated shape configured to be part of the at least one boundary region; Positioning the generated shape in the captured real-world environment; Receiving an input to transform the generated shape by at least one of moving, rotating, or scaling the generated shape; And Adding the transformed generated shape to the at least one boundary region.

17. The system according to claim 11, wherein The instructions, when executed by the processor, further cause the system to modify the at least one boundary region by: Selecting a plurality of waypoints to be part of the at least one boundary region; Receiving an input to position the plurality of waypoints; Connecting the waypoints from a second boundary region; And Adding the second boundary region to the at least one boundary region.

18. The system according to claim 11, wherein, The instructions, when executed by the processor, further cause the system to: Receive 3D volume data of a robotic arm of a robotic surgery system in the real-world environment; Receive the spatial position of the robotic arm; Determine the boundaries of the robotic arm crossing the at least one boundary region based on the spatial position and the 3D volume data; And Prevent the robotic arm from entering the at least one boundary region.

19. The system according to claim 18, wherein, The instructions, when executed by the processor, further cause the system to: Provide force feedback to an input device of a surgical console of the robotic surgery system in response to determining that the robotic arm has crossed the boundaries of the at least one boundary region.

20. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method including: Capturing a real-world environment by an imaging device, wherein the real-world environment includes an input object; Identifying an input object in the captured real-world environment; Tracking the path of the input object in the captured real-world environment; Generating at least one boundary region based on the tracked path of the input object in the captured real-world environment; Render an overlay map including the at least one boundary region; and Display a composite view of the captured real-world environment and the overlay map on a display of the augmented reality device.

Images