Augmented reality simulation setup and control for robotic surgical systems with instrument overlay

Abstract

A system for workspace augmentation includes an augmented reality device. The augmented reality headset comprises: an imaging device configured to capture an image of a real-world environment; a display configured to display the composite view; a processor; and a memory. The memory includes instructions stored on the memory that, when executed by the processor, cause the system to: capture, by the imaging device, a real-world environment including an object; identifying an object in the captured image; determining information related to the object; rendering an overlay including information related to the object; and displaying information related to the object on the display. The object includes a surgical instrument of the robotic surgical system.

Description

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 432,431, filed on Dec. 14, 2022, the entire content of which is incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to systems and methods for workspace augmentation. In particular, this disclosure relates to an augmented reality simulation setup for a robotic surgery system with instrument overlays. Background Art

[0003] Robotic surgery systems are currently being used in minimally invasive medical procedures. Some robotic surgery systems include a surgical console that controls robotic arms and a surgical instrument having an end effector (e.g., a clamp or grasping instrument) coupled to and actuated by the robotic arms. In operation, the robotic arms move to a position above the patient and then guide the surgical instrument through a surgical port or natural orifice of the patient into a small incision to position the end effector at a working site within the patient. Summary of the Invention

[0004] In aspects according to this disclosure, a computer-implemented method for workspace simulation is presented. The method includes capturing a real-world environment by an imaging device. The real-world environment includes objects. The objects include surgical instruments of a robotic surgery system. The method further includes: identifying the objects in the captured real-world environment; determining information associated with the objects; rendering an overlay including the information associated with the objects; and displaying the information associated with the objects on a display of an augmented reality device, wherein the display is configured to display a composite view.

[0005] In one aspect of this disclosure, the information displayed may include the service life, remaining usage amount, number of uses, time of object use, total force, maximum force, name, serial number, lot number, batch number, expiration date, and / or total time of delivered energy and total time of object use of the objects.

[0006] In another aspect of this disclosure, the objects in the captured real-world environment may be identified based on object detection.

[0007] In yet another aspect of this disclosure, object detection may be performed by: generating a spatial grid based on the captured real-world environment; determining the boundaries of the objects based on the spatial grid; and identifying the objects based on a machine learning model, wherein the determined boundaries are provided as an input to the machine learning model.

[0008] In another aspect of the present disclosure, identifying an object in a captured real-world environment can be based on: identifying a machine-readable identifier of the object; comparing the machine-readable identifier with a predetermined database of machine-readable identifiers associated with the object; and identifying the object based on the comparison.

[0009] In yet another aspect of the present disclosure, identifying an object in a captured real-world environment can be based on: receiving a wireless signal from the object, where the wireless signal includes information; and identifying the object based on the information included in the wireless signal.

[0010] In yet another aspect of the present disclosure, the method can further include receiving a command to display an object dashboard and displaying the object dashboard on a display.

[0011] In another aspect of the present disclosure, the object dashboard can include object history, current object status, and / or object usage instructions.

[0012] In yet another aspect of the present disclosure, the method can further include: determining that the object is inserted into a patient's abdomen, where the identified object includes a surgical port; and displaying information related to the surgical port on a display based on the determination.

[0013] In another aspect of the present disclosure, the method can further include displaying a prompt that indicates instructions for a reloador and / or staple cartridge for replacing a surgical instrument.

[0014] According to aspects of the present disclosure, a system for workspace augmentation is presented. The system includes an augmented reality device (e.g., an AR head-mounted device). The augmented reality head-mounted device includes: an imaging device configured to capture an image of a real-world environment; a display configured to display a composite view; a processor; and a memory. The memory includes instructions stored on the memory that, when executed by the processor, cause the system to: capture an image of a real-world environment including an object by the imaging device; identify the object in the captured image; determine information related to the object; render an overlay including the information related to the object; and display the information related to the object on the display. The object includes a surgical instrument of a robotic surgical system.

[0015] In one aspect of the present disclosure, the information displayed can include the service life of the object, remaining usage amount, number of uses, time of object use, total force, maximum force, name, serial number, lot number, batch number, expiration date, and / or total time of delivered energy versus total time of object use.

[0016] In another aspect of the present disclosure, the object in the captured image can be identified based on object detection.

[0017] In yet another aspect of the present disclosure, object detection can be performed by: generating a spatial grid based on the captured real-world environment; determining the boundaries of the object based on the spatial grid; and identifying the object based on a machine learning model, wherein the determined boundaries are provided as an input to the machine learning model.

[0018] In another aspect of the present disclosure, identifying an object in a captured image can be based on: identifying a machine-readable identifier of the object; comparing the machine-readable identifier with a predetermined database of machine-readable identifiers associated with the object; and identifying the object based on the comparison.

[0019] In another aspect of the present disclosure, identifying an object in a captured image can be based on: receiving a wireless signal from the object, wherein the wireless signal includes information; and identifying the object based on the information included in the wireless signal.

[0020] In yet another aspect of the present disclosure, when executed by a processor, the instructions can further cause the system to receive a command to display an object dashboard and display the object dashboard on a display.

[0021] In one aspect of the present disclosure, the object dashboard can include object history, current object status, and / or object usage instructions.

[0022] In another aspect of the present disclosure, when executed by a processor, the instructions can further cause the system to display a prompt that indicates instructions for replacing at least one of a reloador and / or a staple cartridge of a surgical instrument.

[0023] In accordance with aspects of the present disclosure, a non-transitory computer-readable medium is presented. The non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform a method including: capturing a real-world environment by an imaging device, wherein the real-world environment includes an object; identifying the object in the captured real-world environment; determining information related to the object; rendering an overlay including the information related to the object; and displaying the information related to the object on a display of an augmented reality head-mounted device. The object includes a surgical instrument of a robotic surgical system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Aspects of the present disclosure are described herein with reference to the drawings, in which:

[0025] Figure 1 is a schematic illustration of a robotic surgical system according to one aspect of the present disclosure, the robotic surgical system including a control tower, a console, and one or more surgical robotic arms;

[0026] Figure 2is of an aspect according to the present disclosure Figure 1 Perspective view of the surgical robotic arm of a robotic surgery system;

[0027] Figure 3 is a perspective view of a placement arm of a robotic surgery system according to an aspect of the present disclosure, the placement arm having Figure 1 the surgical robotic arm of a robotic surgery system;

[0028] Figure 4 is of an aspect according to the present disclosure Figure 1 Schematic diagram of the computer architecture of a robotic surgery system;

[0029] Figure 5 is of an embodiment according to the present disclosure Figure 1 Schematic diagram of a robotic surgery system that is positioned around an operating table;

[0030] Figure 6 is a flowchart of a computer-implemented method for workspace enhancement according to an aspect of the present disclosure;

[0031] Figure 7 is an image of a composite view of workspace enhancement with a surgical instrument having an overlay; and

[0032] Figure 8 is an image of a composite view of workspace enhancement with a surgical port having an overlay. DETAILED DESCRIPTION

[0033] Aspects of the robotic surgery system disclosed herein are described in detail with reference to the accompanying drawings, in which like reference numerals represent the same or corresponding elements in each of the several views. As used herein, the term "distal" refers to the portion of the robotic surgery system and / or the surgical instrument coupled thereto that is closer to the patient, while the term "proximal" refers to the portion that is farther from the patient.

[0034] The term "application" may include a computer program designed to perform a function, task, or activity for the benefit of a user. An application may refer to, for example, software that runs locally or remotely, software that runs as a stand-alone program or in a web browser, or other software that would be understood by those skilled in the art as an application. An application may run on a controller or on a user device, including, for example, on a mobile device, a personal computer, or a server system.

[0035] As will be described in detail below, the present disclosure relates to a robotic surgical system that includes a surgical console, a control tower, and one or more mobile carts having surgical robotic arms coupled to mounting arms. The surgical console receives user input via one or more interface devices, and the user input is interpreted by the control tower as movement commands for moving the surgical robotic arms. The surgical robotic arms include a controller configured to process the movement commands and configured to generate torque commands for activating one or more actuators of the robotic arms, and the one or more actuators in turn move the robotic arms in response to the movement commands.

[0036] Reference Figure 1 , the robotic surgical system 10 generally includes an augmented reality head-mounted device 600, a control tower 20 that is connected to all components of the robotic surgical system 10, including a surgical console 30 and one or more robotic arms 40. Each robotic arm 40 includes a surgical instrument 50 removably coupled thereto. Each robotic arm 40 is also coupled to a mobile cart 60.

[0037] The augmented reality head-mounted device 600 configured to display a composite view generally includes a controller 602, an imaging device 604, and a display 608. The controller 602 includes: a memory configured to have instructions stored thereon; and a processor configured to execute the instructions. The augmented reality head-mounted device 600 can overlay virtual objects, such as virtual robotic arms ( Figure 7 ). For example, the augmented reality head-mounted device 600 can provide suggestions to the user on how to position various virtual objects to assist in setting up the operating room for the surgery. It is envisioned that the augmented reality head-mounted device 600 can be a full virtual reality head-mounted device (such as the Quest from Menlo Park, California) or an augmented reality (mixed reality) head-mounted device (such as the from Seattle, Washington).

[0038] The surgical instrument 50 is configured for use during minimally invasive surgical procedures. In aspects, the surgical instrument 50 can be configured for open surgical procedures. In aspects, the surgical instrument 50 can be an endoscope (such as an endoscope camera 51) configured to provide video feedback to the user. In additional aspects, the surgical instrument 50 can be an electrosurgical clamp configured to seal tissue by pressing the tissue between jaw members and applying an electrosurgical current thereto. In yet additional aspects, the surgical instrument 50 can be a surgical stapler that includes a pair of jaws configured to grasp and clamp tissue while deploying a plurality of tissue fasteners (e.g., staples) and cutting the stapled tissue.

[0039] One of the robotic arms 40 can include an endoscopic camera 51 configured to capture video of the surgical site. The endoscopic camera 51 can be a stereoscopic endoscope configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The endoscopic camera 51 is coupled to a video processing device 56, which can be disposed within the control tower 20. The video processing device 56 can be any computing device as described below, configured to receive video feedback from the endoscopic camera 51, perform image processing based on the depth estimation algorithm of the present disclosure, and output the processed video stream. The processing of the video feedback from the endoscopic camera 51 can be translated into valuable information to be displayed on an overlay showing augmented reality instrument tags. For example, the video feedback from the endoscopic camera 51 can be processed, and augmented reality instrument tags can be displayed to indicate whether the surgical instrument 50 can be safely withdrawn, or for example, whether the surgical instrument 50 is still grasping tissue.

[0040] The surgical console 30 includes a first display 32 and a second display 34. The first display shows video feedback of the surgical site provided by the camera 51 of the surgical instrument 50 disposed on the robotic arm 40, and the second display shows a user interface for controlling the robotic surgery system 10. The first display 32 and the second display 34 are touchscreens, allowing different graphical user inputs to be displayed.

[0041] The surgical console 30 also includes a plurality of user interface devices, such as a foot pedal 36 and a pair of handle controllers 38a and 38b, which are used by the user to remotely control the robotic arm 40. The surgical console further includes an armrest 33 for supporting the user's arm when operating the handle controllers 38a and 38b.

[0042] The control tower 20 includes a display 23 (which can be a touchscreen) and outputs on a graphical user interface (GUI). The control tower 20 also serves as an interface between the surgical console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arm 40, such as moving the robotic arm 40 and the corresponding surgical instrument 50 based on a set of programmable instructions and / or input commands from the surgical console 30 in such a way that the robotic arm 40 and the surgical instrument 50 perform a desired sequence of movements in response to inputs from the foot pedal 36 and the handle controllers 38a and 38b.

[0043] Each of the control tower 20, the surgical console 30, and the robotic arm 40 includes a respective computer 21, 31, 41. The computers 21, 31, 41 are interconnected with each other by using any suitable communication network based on a wired or wireless communication protocol. As used herein, the term "network", whether plural or singular, refers to a data network, including but not limited to the Internet, intranet, wide area network or local area network, and is not limited to the entire scope of the definition of the communication network covered by this disclosure. Suitable protocols include but are not limited to Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol / Internet Protocol (UDP / IP), and / or Datagram Congestion Control Protocol (DCCP). Wireless communication can be achieved through one or more wireless configurations, such as radio frequency, light, Wi-Fi, Bluetooth (an open wireless protocol used to exchange data from fixed and mobile devices over short lengths of radio waves), creating a personal area network (PAN), (a specification of an advanced communication protocol using a small low-power digital radio based on the IEEE 122.15.4-2003 wireless personal area network (WPAN) standard).

[0044] The computers 21, 31, 41 can include any suitable processor (not shown) that is operably connected to a memory (not shown), which can include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor can be any suitable processor (e.g., control circuit) adapted to execute the operations, calculations, and / or instruction sets described in this disclosure, including but not limited to a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor, and combinations thereof. Those skilled in the art will understand that a processor can be replaced by using any logical processor (e.g., control circuit) adapted to execute the algorithms, calculations, and / or instruction sets described herein.

[0045] Reference Figure 2 , each robotic arm 40 can include a plurality of links 42a, 42b, 42c, which are interconnected at joints 44a, 44b, 44c, respectively. The joint 44a is configured to fix the robotic arm 40 to the movable cart 60 and define a first longitudinal axis. Reference Figure 3 , the movable cart 60 includes a lift 61 and a mounting arm 62, and the mounting arm provides a base for mounting the robotic arm 40. The lift 61 allows the mounting arm 62 to move vertically. The movable cart 60 also includes a display 69 for displaying information about the robotic arm 40.

[0046] The mounting arm 62 includes a first link 62a, a second link 62b, and a third link 62c, which provide lateral maneuverability of the robotic arm 40. The links 62a, 62b, 62c are interconnected at joints 63a and 63b, and each joint may include an actuator (not shown) for rotating the links 62b and 62b relative to each other and relative to the link 62c. In particular, the links 62a, 62b, 62c are capable of moving in their respective laterally parallel planes, thereby allowing the robotic arm 40 to extend relative to a patient (e.g., an operating table). In various aspects, the robotic arm 40 may be coupled to an operating table (not shown). The mounting arm 62 includes a control device 65 for regulating the movement of the links 62a, 62b, 62c and the elevator 61.

[0047] The third link 62c includes a rotatable base 64 having two degrees of freedom. In particular, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a is capable of rotating about a first fixed arm axis perpendicular to the plane defined by the third link 62c, and the second actuator 64b is capable of rotating about a second fixed arm axis transverse to the first fixed arm axis. The first actuator 64a and the second actuator 64b allow for full three-dimensional orientation of the robotic arm 40.

[0048] The actuator 48b of the joint 44b is coupled to the joint 44c via a belt 45a, and the joint 44c is in turn coupled to the joint 46c via a belt 45b. The joint 44c may include a differential gearbox that couples the belts 45a and 45b such that the actuator 48b is configured to rotate each of the links 42b, 42c and the holder 46 relative to each other. More specifically, the links 42b, 42c and the holder 46 are passively coupled to the actuator 48b, which forces rotation about a pivot point "P" located at the intersection of a first axis defined by the link 42a and a second axis defined by the holder 46. Thus, the actuator 48b controls the angle θ between the first axis and the second axis, thereby allowing for orientation of the surgical instrument 50. Due to the interconnection of the links 42a, 42b, 42c and the holder 46 via the belts 45a and 45b, the angles between the links 42a, 42b, 42c and the holder 46 are also adjusted to achieve the desired angle θ. In various aspects, some or all of the joints 44a, 44b, 44c may include actuators to eliminate the need for mechanical linkages.

[0049] Joints 44a and 44b include actuators 48a and 48b that are configured to drive joints 44a, 44b, 44c relative to each other via a series of belts 45a and 45b or other mechanical linkages such as drive rods, cables, or levers. In particular, actuator 48a is configured to rotate robotic arm 40 about a longitudinal axis defined by link 42a.

[0050] Also referring Figure 2 , robotic arm 40 further includes a holder 46 that defines a second longitudinal axis and is configured to receive an instrument drive unit (IDU) 52 ( Figure 1 ). IDU 52 is configured to couple to the actuation mechanisms of surgical instrument 50 and camera 51 and is configured to move (e.g., rotate) and actuate instrument 50 and / or camera 51. IDU 52 transfers actuation force from its actuator to surgical instrument 50 to actuate components of surgical instrument 50 (e.g., an end effector). Holder 46 includes a sliding mechanism 46a that is configured to move IDU 52 along the second longitudinal axis defined by holder 46. Holder 46 further includes a joint 46b that rotates holder 46 relative to link 42c. During an endoscopic procedure, instrument 50 may be inserted through an access port 55 ( Figure 3 ) held by holder 46.

[0051] Robotic arm 40 further includes a plurality of manual override buttons 53 ( Figure 1 and Figure 5 ) that are disposed on IDU 52 and mounting arm 62 and may be used in a manual mode. A user may press one or more of these buttons 53 to move components associated with buttons 53.

[0052] Referring Figure 4, each of the computers 21, 31, 41 of the robotic surgical system 10 may include a plurality of controllers, which may be implemented in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and a safety observer 21b. The controller 21a receives data from the computer 31 of the surgical console 30 regarding the current position and / or orientation of the handle controllers 38a and 38b and the status of the foot pedal 36 and other buttons. The controller 21a processes these input positions to determine the desired drive commands for each joint of the robotic arm 40 and / or the IDU 52, and transmits these desired drive commands to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by the encoders of the actuators 48a and 48b, and uses this information to determine the force feedback commands, which are transmitted back to the computer 31 of the surgical console 30 to provide haptic feedback through the handle controllers 38a and 38b. The safety observer 21b checks the validity of the data entering and exiting the controller 21a, and if an error in data transmission is detected, notifies the system fault handler to place the computer 21 and / or the robotic surgical system 10 in a safe state.

[0053] The computer 41 includes a plurality of controllers, namely, a cart main controller 41a, a mounting arm controller 41b, a robotic arm controller 41c, and an instrument drive unit (IDU) controller 41d. The cart main controller 41a receives and processes the joint commands from the controller 21a of the computer 21, and transmits them to the mounting arm controller 41b, the robotic arm controller 41c, and the IDU controller 41d. The cart main controller 41a also manages instrument exchange and the overall status of the movable cart 60, the robotic arm 40, and the IDU 52. The cart main controller 41a also transmits the actual joint angles back to the controller 21a.

[0054] The mounting arm controller 41b controls each of the joints 63a and 63b, as well as the rotatable base 64 of the mounting arm 62, and calculates the desired motor movement commands (e.g., motor torque) for the pitch axis and controls the brakes. The robotic arm controller 41c controls each joint 44a and 44b of the robotic arm 40, and calculates the desired motor torque required for gravity compensation, friction compensation, and closed-loop position control of the robotic arm 40. The robotic arm controller 41c calculates the movement commands based on the calculated torque. The calculated motor commands are then transmitted to one or more of the actuators 48a and 48b in the robotic arm 40. The actual joint positions are then transmitted back to the robotic arm controller 41c by the actuators 48a and 48b.

[0055] The IDU controller 41d receives the desired joint angles (such as wrist and jaw angles) of the surgical instrument 50 and calculates the desired current of the motors in the IDU 52. The IDU controller 41d calculates the actual angles based on the motor positions and transmits the actual angles back to the cart main controller 41a.

[0056] The manipulator 40 is controlled in response to the posture of a handle controller (e.g., the handle controller 38a) that controls the manipulator 40, and the posture is transformed into the desired posture of the manipulator 40 by the hand-eye transformation function executed by the controller 21a. The hand-eye function and other functions described herein are implemented in software that can be executed by the controller 21a or any other suitable controller described herein. The posture of a handle controller 38a can be implemented as the coordinate position and roll-pitch-yaw ("RPY") orientation relative to a coordinate reference system (which is fixed to the surgical console 30). The desired posture of the instrument 50 is relative to the fixed system on the manipulator 40. Then, the posture of the handle controller 38a is scaled by the scaling function executed by the controller 21a. In various aspects, by the scaling function, the coordinate position is reduced, and the orientation is amplified. In addition, the controller 21a also executes a clutch function, so as to disengage the handle controller 38a from the manipulator 40. In particular, if certain movement limits or other boundaries are exceeded, the controller 21a stops transmitting the movement commands from the handle controller 38a to the manipulator 40 and substantially acts like a virtual clutch mechanism, for example, restricting the mechanical input from affecting the mechanical output.

[0057] The desired posture of the manipulator 40 is based on the posture of the handle controller 38a and is then passed through the inverse kinematics function executed by the controller 21a. The inverse kinematics function calculates the angles of the joints 44a, 44b, 44c of the manipulator 40 that achieve the scaled and adjusted posture input by the handle controller 38a. The calculated angles are then passed to the manipulator controller 41c, which includes a joint axis controller with a proportional derivative (PD) controller, a friction estimator module, a gravity compensator module, and a bilateral saturation block configured to limit the commanded torques of the motors of the joints 44a, 44b, 44c.

[0058] The video processing device 56 is configured to process the video feedback from the endoscope camera 51 and output the processed video stream on the first display 32 of the surgical console 30 and / or the display 23 of the control tower 20.

[0059] Reference Figure 5, the robotic surgical system 10 is installed around the operating table 90. The system 10 includes mobile carts 60a-d, which can be numbered from "1" to "4". During installation, each of the carts 60a-d is positioned around the operating table 90. The position and orientation of the carts 60a-d depend on multiple factors, such as the placement of multiple access ports 55a-d, which in turn depend on the surgery being performed. Once the port placement is determined, the access ports 55a-d are inserted into the patient, and the carts 60a-d are positioned to insert the instruments 50 and the laparoscopic camera 51 into the corresponding ports 55a-d.

[0060] Figure 6 A flowchart is shown that illustrates various operations of an exemplary method for workspace enhancement using an augmented reality instrument overlay. Those skilled in the art will understand that one or more operations of method 650 may be performed in a different order, repeated, and / or omitted without departing from the scope of this disclosure. In various aspects, the illustrated method 650 may operate in a controller 602( Figure 1 ), in a remote device, or in another server or system. Other variations are considered to be within the scope of this disclosure. The operations of method 650 will be described with respect to a controller (e.g., the controller 602( Figure 1 ) of the augmented reality head-mounted device 600( Figure 1 ))), but it will be understood that the illustrated operations are also applicable to other systems and their components.

[0061] Initially, at step 652, the controller 602 causes the robotic surgical system 10 to capture the real-world environment via the imaging device 604( Figure 1 ) of the augmented reality head-mounted device 600 (or a mobile device / tablet computer). The imaging device 604 may include a stereoscopic imaging device. The real-world environment includes objects 55. The objects 55 may be surgical instruments 50, access ports 55, or any other instrument or accessory of the robotic surgical system 10. In various aspects, the controller 602 may render a 3D representation of the captured real world.

[0062] Next, at step 654, the controller 602 causes the robotic surgical system 10 to identify an object 755 in the captured real-world environment. For example, the robotic surgical system 10 may identify the object 755 as a surgical instrument adapter or a surgical port( Figure 7 ). Image-based object identification, machine-readable identifier tags, machine-readable identifiers (e.g., barcodes or machine-readable identifier tags), wireless identifiers (e.g., RFID) may be used to identify the surgical object 755. In various aspects, the controller 602 may enable manual registration and / or detection of machine-readable identifiers / tags on the object 755.

[0063] In various aspects, objects 755 in a captured real-world environment can be identified based on object detection. Object detection can be performed by generating a spatial grid based on the captured real-world environment, determining the boundaries of the objects based on the spatial grid, and identifying the objects 755 based on a machine learning model (e.g., a convolutional neural network). These determined boundaries can be provided as input to the machine learning model. The machine learning model can be trained on labeled images of the objects (e.g., surgical instruments).

[0064] In various aspects, identifying objects in a captured real-world environment can be based on identifying a machine-readable identifier of the object, comparing the machine-readable identifier 704 with a predetermined database of machine-readable identifiers associated with the object, and identifying the object based on the comparison.

[0065] In various aspects, identifying objects in a captured real-world environment can be based on receiving a wireless signal from the object, where the wireless signal includes information, and identifying the object based on the information included in the wireless signal.

[0066] Next, at step 656, the controller 602 causes the robotic surgical system 10 to determine information related to the object 755. This information can include the useful life of the object, remaining usage amount, number of uses, time of object use, total force, maximum force, name, serial number, lot number, batch number, expiration date, total time of delivered energy and total time of object use, and / or other relevant information.

[0067] At step 658, the controller 602 causes the robotic surgical system 10 to render an overlay 702 that includes information related to the object ( Figure 7 ), the overlay including information overlaid on the real-world environment. The composite view 700 can represent a clinical workspace simulation, which can be used, for example, to guide a staff member in setting up the robotic surgical system 10 based on information related to the object 755. In another example, the composite view 700 can represent a clinical workspace augmentation, which can be used to guide tasks during surgery. Figure 7 is an image of a composite view 700 of a clinical workspace augmentation with a surgical instrument having an overlay 702. For example, the overlay 702 can display information such as what the identified object 755 is ("stapler adapter") and / or other information related to the object 755, such as the number of uses (e.g., "use count: 5").

[0068] Next, at step 660, the controller 602 causes the robotic surgical system 10 to display on an augmented reality head-mounted device 600 configured to display the composite view ( Figure 1) The information related to the object 755 is displayed on the display 608. For example, the displayed information may include the remaining usage amount and the number of uses. This provides the benefit of not having to insert (i.e., connect) the surgical instrument into the robotic surgical system 10, which can be detected as a "usage" count, thereby inadvertently reducing the available lifespan of the instrument 50 or other accessories. In various aspects, if the object 755 is moved, the displayed overlay can move with the object 755.

[0069] The controller 602 can also recommend the optimal placement of the object based on the function of the object 755 and the type of surgery. In various aspects, the controller 602 can display the composite view 700 on a user device (such as a mobile device and / or a tablet computer), the display 608 of the augmented reality headset 600, or one of the displays 23, 32, 34 of the robotic system 10.

[0070] In various aspects, the controller 602 can receive a command to display the object dashboard 706 and display the object dashboard 706 on the display ( Figure 8 ). The object dashboard 706 can include the object history, the object status (e.g., "expired", "grasping tissue", and / or "straight"), and / or object usage instructions and other information that provides relevant information to the staff. In various aspects, the controller 602 can synchronize with the inventory management system and enable the display of the available quantity and / or the option to request inventory, such as additional reloads. Figure 8 is an image of the composite view 700 of the clinical workspace enhancement of the surgical port with the overlay 702. The composite view 700 can include a dashboard 706 that shows information related to the object 755, such as the usage history of the object 755. The overlay 702 can also indicate information related to the type of the object 755, such as (for example, "plastic 11mm" and "long length"). The dashboard can include a button 708 for turning on or off the display of the dashboard 706.

[0071] In various aspects, the controller 602 can cause the robotic surgical system 10 to display a prompt that indicates an instruction for replacing at least one of the reloaders and / or staple cartridges of the surgical instrument. For example, the controller 602 can cause the robotic surgical system 10 to capture an image of the surgical instrument. The controller 602 can cause the robotic surgical system 10 to render an overlay 702 that has information related to the object ( Figure 7 ) overlaid on the real-world environment. The overlay can display, for example, a prompt indicating an instruction for replacing the reloader of the surgical instrument. The prompt can be displayed on the display 608 of the augmented reality headset 600 ( Figure 1 ). In various aspects, the overlay can provide guidance to the user on how to connect the cable from the generator to the surgical instrument.

[0072] In various aspects, the controller 602 can detect a patient (or clinician) in the real-world environment by the imaging device and display the detected patient through the display 608 of the augmented reality head-mounted device 600( Figure 1 ). Edge detection and / or image segmentation can be used to detect real-world objects such as patients, users, and / or the operating table 90. For example, the controller 602 can extract the edges of the object 755 in the captured image by detecting depth discontinuities, surface orientation discontinuities, and / or material property changes of the object 755 in the captured image. The extracted edges can be used to determine the boundaries of the object 755. Then, the controller can identify the object 755 as, for example, a surgical instrument or a clinician based on these determined boundaries.

[0073] In various aspects, the controller 602 can cause the robotic surgical system 10 to determine the insertion of an object into the patient's abdomen. For example, the identified object 755 can be an access port( Figure 8 ). The controller 602 can cause the robotic surgical system 10 to display information related to the surgical port on the display based on this determination. For example, the overlay 702 can indicate the type of surgical port (e.g., metal vs. plastic, diameter, length, etc.) when inserted into the patient's abdomen. In various aspects, the controller 602 can clearly distinguish between plastic ports and metal ports, lengths, and diameters. For example, the controller 602 can detect which arm the port is connected to, e.g., RH, LH, ENDO, RES.

[0074] In various aspects, the controller 602 can cause the robotic surgical system 10 to provide visualization of the sterile area for the user through the operating table 90. This provides the benefit of enabling the user (e.g., clinician) to see what is sterile and what is not. The visualization of the sterile area can include colors, gradients, and / or shadows. For example, the controller 602 can cause the robotic surgical system 10 to display a red shaded area on the display 608 indicating that the operating table 90 is not sterile. In another example, the sterile area can be shown as green.

[0075] During port placement in the procedure, the controller 602 can cause the robotic surgical system 10 to generate an overlay 702 that displays measurements overlaid on the surgical port, umbilicus, and / or other structures, for example, in response to the curvature of the patient's inflated abdomen. For example, the controller 602 can access measurements based on the patient's inflated abdomen and, in response to the measurements, display the measurements overlaid on the surgical port.

[0076] The controller 602 can cause the robotic surgical system 10 to generate an overlay that displays a virtual monitor for the augmented reality head-mounted device 600( Figure 1) The endoscopic video is displayed in real time on the display 608. In all aspects, the disclosed technology can be extended to controlling robotic arm movement. For example, a user wearing the augmented reality head-mounted device 600( Figure 1 ) will be able to directly visualize the endoscopic feedback in front of him or her and be able to switch between 2D / 3D visualization, adjust the scale, position, and / or rotation of the visualization. In all aspects, the augmented reality head-mounted device 600 can enable the user to adjust display settings, such as the brightness and / or contrast of the visualization.

[0077] The controller 602 can provide enhanced feedback to clinical staff by overlaying information (such as the recommended surgical port entry points on the patient's abdomen) on the composite view 700. The surgical port entry points can be based on the patient's body build. In all aspects, the controller 602 can render real-time measurements or suggestions of the surgical port entry points based on the patient's body build at different locations on the patient. For example, the controller 602 can display an indication that the surgical port should be approximately 5 cm above the navel and approximately 5 cm to the left of the navel.

[0078] It will be understood that various modifications can be made to the aspects disclosed herein. Therefore, the above description should not be construed as restrictive, but merely as illustrations of the various aspects. Those skilled in the art will envision other modifications within the scope and spirit of the appended claims.

Claims

1. A computer-implemented method for workspace enhancement, the method comprising: Capturing a real-world environment by an imaging device, wherein the real-world environment includes objects, and wherein the objects include surgical instruments of a robotic surgical system; Identifying the objects in the captured real-world environment; Determining information associated with the objects; Rendering an overlay including the information associated with the objects; and Displaying the information associated with the objects on a display of an augmented reality device, wherein the display is configured to display a composite view.

2. The computer-implemented method according to claim 1, wherein, The displayed information includes at least one of the following: the useful life of the object, the remaining usage amount, the number of uses, the time of use of the object, the total force, the maximum force, the name, the serial number, the lot number, the batch number, the expiration date, or the total time of delivered energy and the total time of use of the object.

3. The computer-implemented method according to claim 1, wherein, The identifying the objects in the captured real-world environment is based on object detection.

4. The computer-implemented method according to claim 3, wherein, The object detection is performed by: Generating a spatial grid based on the captured real-world environment; Determining the boundaries of the object based on the spatial grid; and Identifying the object based on a machine learning model, wherein the determined boundaries are provided as input to the machine learning model.

5. The computer-implemented method according to claim 1, wherein, The identifying the objects in the captured real-world environment is based on: Identifying a machine-readable identifier of the object; Comparing the machine-readable identifier with a predetermined database of machine-readable identifiers associated with objects; and Identifying the object based on the comparison.

6. The computer-implemented method according to claim 1, wherein, The identifying the objects in the captured real-world environment is based on: Receiving a wireless signal from the object, wherein the wireless signal includes information; and Identifying the object based on the information included in the wireless signal.

7. The computer-implemented method according to claim 1, further comprising: Receiving a command to display an object dashboard; And Displaying the object dashboard on the display.

8. The computer-implemented method according to claim 7, wherein, The object dashboard includes object history, current object status, or object usage instructions.

9. The computer-implemented method according to claim 1, further comprising: Determining that the object is inserted into a patient's abdomen, wherein the identified object includes a surgical port; and Displaying information associated with the surgical port on the display based on the determination.

10. The computer-implemented method according to claim 1, further comprising: Displaying a prompt that indicates an instruction for replacing at least one of a reloador or a staple cartridge of the surgical instrument.

11. A system for workspace enhancement, the system comprising: An augmented reality head-mounted device, the augmented reality head-mounted device comprising: An imaging device configured to capture an image of a real-world environment; A display configured to display a composite view; A processor; and A memory including instructions stored on the memory, the instructions when executed by the processor cause the system to: Capture an image of the real-world environment by the imaging device, wherein the real-world environment includes objects, and wherein the objects in the captured image include surgical instruments of a robotic surgical system; Identify the objects in the captured image; Determine information associated with the objects; Render an overlay including information related to the object; and Display information related to the object on the display.

12. The system according to claim 11, wherein, The information displayed includes at least one of the following: the useful life of the object, the remaining usage amount, the number of uses, the time the object has been used, the total force, the maximum force, the name, the serial number, the lot number, the batch number, the expiration date, and / or the total time of delivering energy and the total time the object has been used.

13. The system according to claim 11, wherein, The identification of the object in the captured image is based on object detection.

14. The system according to claim 13, wherein, The object detection is performed by: Generating a spatial grid based on the captured real-world environment; Determining the boundaries of the object based on the spatial grid; and Identifying the object based on a machine learning model, wherein the determined boundaries are provided as input to the machine learning model.

15. The system according to claim 11, wherein, The identification of the object in the captured image is based on the following: Identifying a machine-readable identifier of the object; Comparing the machine-readable identifier with a predetermined database of machine-readable identifiers associated with the object; and Identifying the object based on the comparison.

16. The system according to claim 11, wherein The identification of the object in the captured image is based on the following: Receiving a wireless signal from the object, wherein the wireless signal includes information; and Identifying the object based on the information included in the wireless signal.

17. The system of claim 11, wherein When executed by the processor, the instructions further cause the system to: Receive a command to display an object dashboard; and Display the object dashboard on the display.

18. The system according to claim 17, wherein, The object dashboard includes the object history, the current object status, or object usage instructions.

19. The system according to claim 11, wherein, When executed by the processor, the instructions further cause the system to: Display a prompt that indicates instructions for replacing at least one of a reloador or a staple cartridge for a surgical instrument.

20. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method that includes: Capturing a real-world environment by an imaging device, wherein the real-world environment includes an object; Identifying the object in the captured real-world environment; Determining information related to the object; Rendering an overlay including information related to the object; and Displaying information related to the object on a display of an augmented reality head-mounted device.

Images