Augmented reality headset for surgical robots

Abstract

An augmented reality (AR) headset is disclosed that provides a wearer with spatial, system, and temporal contextual information of a surgical robotic system to guide the wearer in configuring, operating, or troubleshooting the surgical robotic system before, during, or after a surgical procedure. Spatial contextual information can be rendered in a coordinate system of the AR headset to display a spatially fixed 3D generated virtual model of the robotic arms, instruments, table, and other components of the surgical robotic system that matches the actual position or orientation of the surgical robotic system. The AR headset can communicate with the surgical robotic system to receive real-time status information of the components of the surgical robotic system. The AR headset can use the real-time status information to display context-sensitive user interface information, such as reminders, suggestions, visual or audio cues about maneuvering the robotic arms and table to their target positions and orientations or for troubleshooting purposes.

Description

TECHNICAL FIELD

[0001] The subject technology relates generally to robotic and surgical systems, and more particularly to an augmented reality headset for guiding a user of a robotic-assisted surgical system in setting up a surgical robotic arm or other components of the robotic-assisted surgical system when preparing for or performing a minimally invasive surgical procedure. BACKGROUND

[0002] Minimally invasive surgery (MIS), such as laparoscopic surgery, involves techniques intended to reduce tissue damage during a surgical procedure. For example, laparoscopic surgery often involves making a plurality of small incisions in a patient (e.g., in the abdomen) and introducing one or more surgical tools (e.g., an end effector and an endoscope) into the patient’s body through the incisions. The surgical procedure can then be performed using the introduced surgical tools, with visualization assistance provided by the endoscope.

[0003] Generally, MIS provides multiple benefits, such as reduced patient scarring, reduced patient pain, shortened patient recovery periods, and reduced medical expenses associated with patient recovery. Recent technological developments have allowed more MIS to be performed using robotic systems that include one or more robotic arms for manipulating surgical tools based on commands from a remote operator. The robotic arms can, for example, support various devices at their distal ends, such as surgical end effectors, imaging devices, cannulas for providing access to a patient’s body cavities and organs, etc. In robotic MIS systems, it can be desirable to establish and maintain a high positioning accuracy for surgical instruments supported by the robotic arms.

[0004] Medical personnel and staff typically set up the robotic arms before starting a surgical procedure. It is important that the robotic arms must be in the correct pose relative to the patient; otherwise, complications can arise during the surgical procedure. Setup of the robotic arms, and more generally the robotic system, is a complex and tedious task that often requires a large team of specialists to train new operating room staff on how to properly configure the pose of the robotic arms for a surgical procedure. The medical personnel can also need to perform fault diagnosis on the setup of the robotic arms or attached surgical tools when problems arise. Prior training on how to perform fault diagnosis can be easily forgotten, especially when the scenario is uncommon.

[0005] During a surgical procedure, control of a robotic arm can require a user (e.g., a surgeon or other operator) to make control inputs via one or more user interface devices that translate the user's manipulations or commands into control of the robotic arm. For example, when a surgical tool is positioned at a surgical site of a patient, a tool driver having one or more motors can actuate one or more degrees of freedom of the surgical tool in response to user commands. At times, during a surgical procedure, medical personnel must quickly move a robotic arm away from a patient, such as when the patient's vital signs are rapidly declining. Complications can arise when prior training on how to manipulate the robotic arm in emergency situations can be easily forgotten. Moving an erroneous robotic arm can cause serious injury to the patient, other equipment, or bedside personnel.

[0006] Accordingly, it is desirable to have a tool to mitigate the complexity of new user training, reduce the expense of hiring and training a dedicated education team, prevent delays in robot system setup, and eliminate accidents that can be caused by user confusion during fault diagnosis or emergency scenarios. The tool can be used by medical personnel to minimize errors when preparing and performing a robot-assisted surgical procedure. SUMMARY

[0007] Disclosed herein is an augmented reality headset that can be worn to provide real-time spatial and contextual information to guide medical personnel when setting up and operating a robot-assisted surgical system, also referred to as a surgical robotic system, which is a software-controlled electromechanical system designed for surgeons to perform minimally invasive surgical procedures. The augmented reality headset can overlay a spatially fixed 3D generative model of the surgical robotic system and context-sensitive user interface information onto a pair of goggles. The wearer of the augmented reality headset can receive intuitive guidance about tasks needed to prepare components of the surgical robotic system, such as a robotic arm, for a surgical procedure. For example, the wearer can receive spatial information such as the position and orientation of the robotic arm, system state information such as arm mode, tool type, tool grip state, and other real-time information of the robotic arm, and temporal context information for effecting movement of the robotic arm.

[0008] The augmented reality headset can include a sensor, a processor, and a display. The sensor can capture image data of a component of the surgical robotic system, such as a robotic arm or a surgical table. The processor can establish a common coordinate system between the augmented reality device and the surgical robotic system based on the image data. The augmented reality headset can communicate with the surgical robotic system to receive spatial information and real-time system state information of the surgical robotic system. The processor can create a 3D virtual model of another component of the surgical robotic system, which can be the same or a different component than the component for which image data was acquired. The 3D virtual model is created in a coordinate system of the augmented reality headset based on the spatial information and the real-time system state information of the surgical robotic system and based on the common coordinate system between the augmented reality device and the surgical robotic system. The display can present the 3D virtual model of the component to a wearer of the augmented reality headset.

[0009] A method for an augmented reality headset to interact with a surgical robotic system is disclosed. The method includes the augmented reality headset capturing image data of a component of the surgical robotic system, such as an arm or a surgical table. The method also includes establishing, by the augmented reality headset, a 3D position and orientation of the surgical robotic system in a coordinate system of the augmented reality headset based on the image data. The method also includes receiving, by the augmented reality headset, spatial information and real-time system state information of the surgical robotic system. The method also includes creating, by the augmented reality headset, a 3D virtual model of another component of the surgical robotic system in the coordinate system of the augmented reality headset based on the spatial information and the real-time system state information of the surgical robotic system and based on the 3D position and the orientation of the surgical robotic system in the coordinate system of the augmented reality headset. The method also includes the augmented reality headset maintaining the 3D virtual model of the component as the coordinate system of the AR device changes relative to a coordinate system of the surgical robotic system. BRIEF DESCRIPTION OF DRAWINGS

[0010] For a more complete understanding of the present application, the following description of various aspects and embodiments of the present subject matter is provided, taken together with the accompanying drawings. The drawings and embodiments are illustrative of the present application and are not intended to limit the scope of the present application. It is understood that one of ordinary skill in the art can modify the drawings to generate other drawings of other embodiments that would still fall within the scope of the present application.

[0011] Figure 1 is a pictorial view of an exemplary surgical robotic system 1 in a surgical field in accordance with aspects of the present subject matter.

[0012] Figure 2The present invention illustrates information exchange between an augmented reality headset and a surgical robotic system according to various aspects of the present subject matter, so that the augmented reality headset can display spatial, systemic, and temporal information of the components of the surgical robotic system based on the establishment of a common coordinate system between the augmented reality headset and the surgical robotic system using an image sensor.

[0013] Figure 3 A hologram rendered by an augmented reality headset is shown, illustrating various aspects of the technology in this subject matter, showing the actual and target poses of a robotic arm to guide medical personnel in moving the robotic arm to the target pose.

[0014] Figure 4 The actual and target poses of a robotic arm according to various aspects of the subject matter are shown. These poses can be used to generate waypoints, such as those rendered on an augmented reality headset, to guide medical personnel to move the robotic arm from the actual pose to the target pose.

[0015] Figure 5 A hologram rendered by an augmented reality headset is shown, illustrating various aspects of the technology in this subject matter. The hologram highlights the workspace of the robotic arm and the attached endoscope to help medical personnel position the robotic arm.

[0016] Figure 6 The present invention illustrates information exchange between an augmented reality headset and a surgical robotic system according to various aspects of the present subject matter, so that the augmented reality headset can display spatial, systemic, and temporal information of the components of the surgical robotic system in the coordinate system of the augmented reality headset based on the establishment of a common coordinate system between the augmented reality headset and the surgical robotic system using fixed reference markers.

[0017] Figure 7 This is a block diagram illustrating exemplary hardware components of an augmented reality headset and a surgical robot system according to various aspects of the art in this subject matter.

[0018] Figure 8 This is a flowchart illustrating a method for exchanging information between a surgical robotic system and an augmented reality headset according to various aspects of the subject matter, to establish a common coordinate system between the augmented reality headset and the surgical robotic system, and to transmit spatial, system, and temporal information of the components of the surgical robotic system to the augmented reality headset for rendering the component and system information as a virtual image.

[0019] Figure 9This is a flowchart illustrating a method for using fixed reference markers to establish a common coordinate system between the augmented reality headset and the surgical robot system, and for receiving spatial, systemic, and temporal information of the components of the surgical robot system for rendering images, according to various aspects of the subject matter. Detailed Implementation

[0020] Examples of various aspects and variations of this subject matter are described herein and illustrated in the accompanying drawings. The following description is not intended to limit the invention to these embodiments, but rather to enable those skilled in the art to make and use the invention.

[0021] An augmented reality (AR) headset is disclosed that provides the wearer with spatial, systemic, and temporal contextual information about the components of a surgical robotic system to guide the wearer in configuring and troubleshooting the surgical robotic system before, during, or after surgery. The spatial contextual information can be rendered to display a spatially fixed 3D generated virtual model of the surgical robotic system's robotic arm, instruments, bed, and other components, which is matched to the real-time actual position or orientation of the surgical robotic system in the coordinate system of the AR headset. A simultaneous localization and mapping (SLAM) algorithm can be run on the AR headset to localize the headset's position and orientation, such that the virtual model of the surgical robotic system is rendered to maintain the actual position and orientation of the surgical robotic system as the wearer moves around in the operating room. In one embodiment, a virtual model representing the desired or target position and orientation of the robotic arm can be rendered as an overlay of the actual position and orientation of the robotic arm. The virtual model can be used to guide the wearer of the AR headset to move the robotic arm from its current position and orientation to a target position and orientation.

[0022] The AR headset can also communicate with the surgical robot system to receive real-time status information of the system's components. The AR headset can use this status information to display context-sensitive user interface information to guide the wearer in configuring, operating, or diagnosing the surgical robot system. In one embodiment, the status information of the surgical robot system may include joint angles of the robotic arm, tool type, tool gripping status, active tool energy, arm mode, fault diagnosis codes, etc. In another embodiment, the status information may include the position, orientation, angle, and operating mode of the bed. The AR headset can display reminders, suggestions, visual or audio cues, etc., on how to manually move the robotic arm or execute automated sequences to robotically guide the robotic arm to a target location without colliding with obstacles, patients, beds, or other robotic arms.

[0023] Figure 1This is a drawing view of an exemplary surgical robotic system 1 in an operating room, according to various aspects of the subject matter. The robotic system 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 at a surgical robotic platform 5 (e.g., table, bed, etc.). The system 1 can be combined with any number of devices, tools, or accessories for performing surgery on a patient 6. For example, the system 1 may include one or more surgical tools 7 for performing surgery. The surgical tool 7 may be an end effector attached to the distal end of the surgical arm 4 for performing surgical procedures.

[0024] Each surgical tool 7 can be manually manipulated, robotically manipulated, or both during surgery. For example, a surgical tool 7 can be a tool for accessing, viewing, or manipulating the internal anatomy of a patient 6. In one embodiment, the surgical tool 7 is a gripper capable of grasping the patient's tissues. The surgical tool 7 can be manually controlled by a bedside operator 8; or it can be robotically controlled by the actuated movement of its attached surgical robotic arm 4. The robotic arm 4 is shown as a tabletop system, but in other configurations, the arm 4 can be mounted on a trolley, ceiling, or sidewall, or in another suitable structural support.

[0025] Generally, a remote operator 9 (such as a surgeon or another person) can use the user console 2 to remotely manipulate the arm 4 and / or the attached surgical instruments 7, for example, through remote operation. The user console 2 may be located in the same operating room as the rest of the system 1, such as... Figure 1 As shown. However, in other environments, the user console 2 may be located in an adjacent or nearby room, or it may be located in a remote location, such as in different buildings, cities, or countries. The user console 2 may include a seat 10, foot controls 13, one or more handheld user input devices (UIDs) 14, and at least one user display 15 configured to display a view, for example, of a surgical site within a patient 6. In the exemplary user console 2, a remote operator 9 sits in the seat 10 and views the user display 15 while manipulating the foot controls 13 and the handheld UID 14 to remotely control the arm 4 and the surgical instrument 7 (which is mounted on the distal end of the arm 4).

[0026] In some variations, the bedside operator 8 can also operate the system 1 in a "bedside" mode, where the bedside operator 8 (the user) is now positioned to one side of the patient 6 and simultaneously manipulates robot-driven tools (end-effectors attached to arm 4), for example, holding a handheld UID 14 and a manual laparoscopic tool with one hand. For instance, the bedside operator's left hand can manipulate the handheld UID to control the robotic components, while the bedside operator's right hand can manipulate the manual laparoscopic tool. Therefore, in these variations, the bedside operator 8 can perform both robot-assisted minimally invasive surgery and manual laparoscopic surgery on the patient 6.

[0027] During the exemplary procedure (surgical procedure), patient 6 is prepared for surgery and anesthetized by aseptically covering the patient with a sterile drape. Initial access to the surgical site can be manually performed (to facilitate access to the surgical site) while the arm of robotic system 1 is in a retracted or withdrawn configuration. Once access is complete, initial positioning or preparation of robotic system 1, including its arm 4, can be performed. For example, a remote operator 9 or bedside operator 8 at user console 2 can use a handheld UID 14 to move arm 4 from the retracted configuration to a ready position above patient 6 during preoperative setup. Alternatively, a surgeon or bedside person at direct viewing table 5 can wear the AR headset disclosed herein to receive instructions on moving arm 4. For example, the AR headset can render a virtual image of the actual retracted configuration of arm 4, a virtual image of the desired ready position, and a series of waypoints to guide the surgeon or bedside person on how to move arm 4 from the current retracted configuration to the ready position.

[0028] Next, the surgical procedure continues, with the remote operator 9 at user console 2 using foot controls 13 and UID 14 to manipulate various end effectors and, possibly, imaging systems to perform the procedure. Manual assistance can also be provided by a bedside person (e.g., bedside operator 8) wearing sterile surgical gowns at the operating table or operating table 5, who can perform tasks on one or more arms of the robotic arm 4, such as tissue retraction, manual repositioning, and tool changes. Non-sterilized personnel may also be present to assist the remote operator 9 at user console 2. When the procedure or surgical procedure is completed, system 1 and user console 2 can be configured or set to facilitate postoperative procedures, such as cleaning or sterilization, and inputting or printing healthcare records via user console 2. During the surgical procedure or postoperative procedure, the bedside person can wear the AR headset disclosed herein to receive guidance during manual or automated repositioning of the robotic arm 4.

[0029] In one embodiment, the remote operator 9 holds and moves UID 14 to provide input commands, thereby moving the robotic arm actuator 17 in the robotic system 1. UID 14 may be communicatively coupled to the rest of the robotic system 1, for example, via a console computer system 16. UID 14 may generate spatial state signals corresponding to the movement of UID 14, such as the position and orientation of the UID's handheld housing, and the spatial state signals may be input signals for controlling the movement of the robotic arm actuator 17. The robotic system 1 may use control signals derived from the spatial state signals to control the proportional movement of the actuator 17. In one embodiment, a console processor of the console computer system 16 receives the spatial state signals and generates corresponding control signals. Based on these control signals controlling how the actuator 17 is energized to move a segment or connector of the arm 4, the movement of a corresponding surgical tool attached to the arm may simulate the movement of UID 14. Similarly, the interaction between the remote operator 9 and UID 14 may generate, for example, a clamping control signal that causes the jaws of the gripper of the surgical tool 7 to close and clamp the tissue of the patient 6.

[0030] The surgical robot system 1 may include a plurality of UIDs 14, wherein each UID generates a corresponding control signal controlling the actuators and surgical instruments (end-effectors) of the corresponding arm 4. For example, a remote operator 9 may move a first UID 14 to control the movement of an actuator 17 located in the left robotic arm, wherein the actuator responds by moving links, gears, etc. in the arm 4. Similarly, movement of a second UID 14 by the remote operator 9 controls the movement of another actuator 17, which in turn moves other links, gears, etc. of the robot system 1. The robot system 1 may include a right arm 4 fixed to a bed or table 5 on the right side of the patient, and a left arm 4 located on the left side of the patient. The actuators 17 may include one or more motors controlled to drive joints of the arm 4 to rotate, for example, relative to the patient, changing the orientation of an endoscope or gripper of a surgical instrument 7 attached to the arm. The movement of a plurality of actuators 17 in the same arm 4 may be controlled by spatial state signals generated from a particular UID 14. The UID 14 may also control the movement of the corresponding surgical instrument gripper. For example, each UID 14 can generate a corresponding clamping signal to control the movement of an actuator (e.g., a linear actuator) that opens or closes the jaws of the gripper at the distal end of the surgical tool 7 to grasp tissue in the patient 6.

[0031] In some respects, communication between platform 5 and user console 2 can be achieved via control tower 3, which translates user commands received from user console 2 (and more specifically from console computer system 16) into robot control commands transmitted to arm 4 on robot platform 5. Control tower 3 can also transmit status and feedback from platform 5 back to user console 2. The communication connection between robot platform 5, user console 2, and control tower 3 can be via wired and / or wireless links, using any suitable data communication protocol from a variety of data communication protocols. Any wired connection can optionally be integrated into the floor and / or walls or ceiling of the operating room. Robot system 1 can provide video output to one or more displays, including displays within the operating room and remote displays accessible via the Internet or other networks. Video output or feeds can also be encrypted to ensure privacy, and all or part of the video output can be stored on a server or electronic healthcare record system.

[0032] Before initiating surgery using the surgical robotic system 1, the surgical team can perform preoperative setup. During preoperative setup, the main components of the surgical robotic system (stage 5 and robotic arm 4, control tower 3 and user console 2) are positioned, connected, and powered in the operating room. Stage 5 and robotic arm 4 can be in a fully retracted configuration, with arm 4 positioned below stage 5 for storage and / or transport. The surgical team can extend the arm from its retracted position for sterile draping. After draping, arm 4 can be partially retracted until needed for reuse. Several routine laparoscopic steps may need to be performed, including cannula placement and insufflation. For example, each cannula may be inserted into a small incision and through the body wall using a tampon. The cannulas and tampon allow light to enter to visualize tissue layers during insertion, thereby minimizing the risk of injury during placement. Typically, an endoscope is placed first to provide visualization for a handheld camera for placing other cannulas. After insufflation, if necessary, manual instruments can be inserted through the cannulas to perform any laparoscopic steps manually.

[0033] Next, the surgical team can position the robotic arms 4 above the patient and attach each arm to its corresponding cannula. In one implementation, guidance for manipulating the arms 4 to safely attach to the corresponding cannulas or tools can be provided by an AR headset worn by members of the surgical team. The AR headset renders a virtual image of the target position and orientation of the arms 4, as well as a series of waypoints for manipulating the arms 4 from their current position and orientation to the target position and orientation. The surgical robotic system 1 has the ability to immediately and uniquely identify each tool (endoscope and surgical instrument) once attached and displays the tool type and arm position on an open or immersive display 15 at the user console 2 and a touchscreen display on the control tower 3. The corresponding tool function is enabled and activated using the master UID 14 and foot pedal 13. The patient-side assistant can attach and detach tools as needed throughout the procedure. The surgeon, seated at the user console 2, can begin the surgery using tools controlled by two master UIDs 14 and foot pedal 13. The system translates the surgeon's hand movements, wrist movements, and finger movements into precise, real-time movements of the surgical tools via the master UID 14. Therefore, the system continuously monitors each surgical procedure performed by the surgeon and pauses instrument movement if the system cannot accurately reflect the surgeon's hand movements. During surgery, if the endoscope is moved from one arm to the other, the system can adjust the master UID 14 for instrument calibration and continue instrument control and movement. The foot pedal 13 can be used to activate various system modes, such as endoscope control and various instrument functions, including monopolar and bipolar ablation, without requiring the surgeon's hand to be removed from the master UID 116.

[0034] Table 5 can be repositioned intraoperatively. For safety reasons, all instruments must be within the surgeon's field of vision and under their active control at user console 2. Instruments not under the surgeon's active control must be removed, and the table legs must be locked. During table movement, the integrated robotic arm 4 can passively follow the table's movement. Audio and visual cues are available to guide the surgical team during table movement. Audio cues may include tone and voice prompts. Visual messaging on displays at user console 2 and control tower 3 informs the surgical team of the table movement status.

[0035] Figure 2This paper illustrates information exchange between an augmented reality headset and a surgical robotic system according to various aspects of the subject matter, allowing the AR headset to display spatial, systemic, and temporal information of the components of the surgical robotic system based on the establishment of a common or global coordinate system between the AR headset and the surgical robotic system using image sensors. The AR headset may have one or more cameras that capture color and depth information of objects in a real-world scene. For example, the AR headset may have RGB and depth (RGBD) sensors to capture color and depth image information of the arm 4 and platform 5 of the surgical robotic system 1 from the perspective of the wearer of the AR headset. Thus, the RGBD image captured by the AR headset is an image of the real-world arm 4 and platform 5 based on the coordinate system of the AR headset. In one embodiment, the AR headset may run an object recognition algorithm to identify the arm 4 and platform 5. The surgical robotic system 1 may have a set of RGBD sensors mounted at various locations to capture color images and depth information of the configuration of the arm 4 and platform 5. Thus, the RGBD image captured by the surgical robotic system 1 is an image of the arm 4 and platform 5 based on the coordinate system of the surgical robotic system 1. To enable the AR headset to render a virtual reconstruction of arms 4 and platform 5 that matches the real-time realistic scene position and orientation of arms 4 and platform 5, or to render a virtual image of arms 4 and platform 5 that can be integrated with the real-time realistic scene position and orientation of arms 4 and platform 5, a common coordinate system can be established between the AR headset and the surgical robot system 1. In one embodiment, the surgical robot system 1 may have other types of sensors, such as infrared sensors, to capture images and other information of arms 4 and platform 5 of the surgical robot system 1 or the patient.

[0036] The AR headset can transmit RGBD images (also known as point clouds) to a surgical robotic system 1, such as a control tower 3, to request assistance in analyzing the point cloud. The surgical robotic system 1 can process the analyzed point cloud from the AR headset, the RGBD images captured by the surgical robotic system 1, and a real-time data stream describing the position and orientation information of arms 4 and stage 5 to identify and virtually establish the 3D position and orientation of arms 4 and stage 5 in the coordinate system of the AR headset. Therefore, the surgical robotic system 1 can establish a common or global coordinate system between the AR headset and the surgical robotic system 1. Based on the common coordinate system, the surgical robotic system 1 can transmit information to the AR headset to create 3D virtual models of arms 4 and stage 5 that match their actual positions and orientations, or to create 3D virtual models of arms 4 and stage 5 that can be integrated with their real-time real-world scene positions and orientations. The surgical robotic system 1 can also transmit a global coordinate system to the AR headset.

[0037] The information transmitted from the surgical robot system 1 to the AR headset may include: real-time information on the position and orientation of arm 4 and table 5, analysis results of point clouds received from the AR headset, 3D mesh models of components of the surgical robot system 1 or the operating room, and real-time system status information of the surgical robot system 1. In one embodiment, the real-time system status information may include: joint angles and rotation of arm 4, type of tool attached to arm 4, tool gripping state, active tool energy, arm mode (e.g., retracted configuration, ready pose, clutch mode, teleoperation control, etc.), fault diagnosis error codes, etc. Using the information received from the surgical robot system 1, the AR headset can render virtual images of arm 4, table 5, or other components of the surgical robot system 1 or the operating room and fuse them with real-world scene objects captured by RGBD sensors in the coordinate system of the AR headset.

[0038] Simultaneous Localization and Mapping (SLAM) algorithms can be run on the AR headset to locate its position and orientation, allowing virtual images of the components of the surgical robot system 1 to be spatially fixed to maintain their virtual position and orientation as the wearer moves around the operating room. For example, as the coordinate system of the AR headset changes relative to the coordinate system of the surgical robot system 1, the virtual reconstructions of arm 4 and stage 5 can be matched with their real-time, realistic scene positions and orientations. In addition to rendering real-time spatial information of the surgical robot system 1, the AR headset can render real-time system information that cannot be detected by the RGBD sensors and computer vision algorithms of the surgical robot system 1 using state information received from the surgical robot system 1. The spatial, system, and temporal context-sensitive information about the surgical robot system 1 provided by the AR headset can be used to guide the wearer in configuring, operating, or diagnosing faults in the surgical robot system 1 before, during, or after surgery.

[0039] Figure 3 A hologram rendered by an augmented reality headset, according to various aspects of the subject matter, is shown, illustrating the actual and target poses of a robotic arm to guide medical personnel in moving the robotic arm to the target pose. The robotic arm can be extended in a draped pose for sterile draping. The AR headset can reconstruct a virtual image of the arm (labeled as the real arm) to match the arm's real-time position and orientation in the draped pose. In one embodiment, the AR headset can project a real-scene arm captured by an RGBD sensor. It is desirable to move the arm to its ready pose above the patient during preoperative setup, such as in manipulations to align the arm with a cannula.

[0040] The AR headset receives information from the surgical robotic system regarding the target position and orientation of an arm in a ready pose, allowing the AR headset to render a virtual image of the arm in the ready pose (labeled as the virtual arm). As the coordinate system of the AR headset changes due to the wearer's movement, the images of the real arm and the virtual arm maintain their relative pose. In one implementation, if the arm is to be robotically moved from its current draped pose to a ready pose, the AR headset receives information from the surgical robotic system regarding a trajectory generated by a robot control algorithm for moving the arm. As the arm is guided by the robot control algorithm along the trajectory from the draped pose to the ready pose, the AR headset generates a series of virtual images of the arm. The bedside person wearing the AR headset can confirm that the arm can move along the trajectory without being obstructed by potential obstacles before instructing the surgical robotic system to execute the robot control algorithm to move the arm.

[0041] In one implementation, if the arm is to be manually moved to a ready pose, an AR headset receives information from the surgical robotic system about a recommended trajectory for moving the arm to avoid collisions with other obstacles, the patient, bedside personnel, tables, other arms, etc. The AR headset can generate a series of waypoints to guide the bedside personnel on how to move the arm. As the arm is moved robotically, the waypoints can be rendered as a series of virtual images of the arm. The bedside personnel can manipulate the arm to align it with the virtual images of the arm along the recommended trajectory and ultimately into the target pose.

[0042] Figure 4 The actual and target poses of a robotic arm according to various aspects of the subject matter are illustrated. These poses can be used to generate waypoints, such as those rendered on an augmented reality headset, to guide a healthcare professional to move the robotic arm from its actual pose to the target pose. The arm in its current pose 309 can be rendered as a virtual image or projected as a real-world object captured by an RGBD sensor. The arm is also rendered as a virtual arm 311 in its target pose. The arm will move along trajectory 310 to the target pose of the virtual arm 311. The AR headset can render the virtual arm 311 at the same anchor / mount point as the arm in its current pose 309, thus providing a common anchoring reference point for both the real and virtual robotic arms. In one embodiment, the AR headset can render trajectory 310 as a series of waypoints or a series of virtual images of the arm. A bedside person wearing the AR headset can manipulate the arm from its current pose 309 along waypoints or with the virtual images of trajectory 310 until the arm is finally aligned with the virtual arm 311 in its target pose. When the arm is in the target pose or within the tolerance of the target pose, the AR headset can respond to the user using cues such as highlighting the arm. The AR headset can also render visual cues or generate audio cues to help the user manipulate the arm along the trajectory 310.

[0043] In one implementation, the AR headset can provide context-sensitive information to guide a user in the path of moving one arm away from the other arms. For example, before or during surgery, when a surgeon is using only three arms to perform surgery, the AR headset can highlight a “Release Arm Move” button on the fourth arm to allow a bedside person to disengage the arm from its ready or pre-docking position and safely move the arm away from the three other arms and the bed.

[0044] In one implementation, the AR headset can provide information about whether surgical instruments have been correctly attached to arm 4, as well as other context-sensitive troubleshooting information. For example, the AR headset can receive system status information from the surgical robot system 1 regarding the type of tool attached to arm 4 and the identification of arm 4 with the tool attached. When the tool is properly attached to the arm, the AR headset can highlight the arm or tool in blue to visually indicate to the user that the tool is operable. When the arm or tool attached to the arm is not highlighted in blue or is highlighted in red, the user is warned to troubleshoot the incorrectly installed tool. In one implementation, the AR headset can display troubleshooting codes or generate audio guidance to assist the user in diagnosing the problem.

[0045] In one implementation, the AR headset can provide bedside personnel with guidance on how to handle emergencies. For example, if a patient's vital signs deteriorate rapidly and the robotic arm 4 must be moved quickly away from the patient to allow for intervention, the AR headset can provide emergency guidance to the bedside personnel. The AR headset can receive instructions from the surgical robotic system 1 regarding the sequence of actions to release tools, disengage arm 4 from its docking position, and maneuver arm 4 away from the patient. The AR headset can use context-sensitive real-time information (such as highlighting buttons on arm 4, displaying text, or providing other types of visual or audio instructions on how to move arm 4 quickly without causing harm to the patient, other equipment, or bedside personnel) to guide the bedside personnel through the sequence of actions. Therefore, the AR headset can provide guidance on emergency procedures that may have been forgotten by the bedside personnel, thereby eliminating accidents that may result from user confusion or error in emergency situations.

[0046] In one embodiment, the AR headset can display the effects of the current pose, such as providing information about the workspace of arm 4 or the tool attached to arm 4. While arm 4 or the tool is currently positioned and oriented, the AR headset can receive information from the surgical robot system 1 about the maximum and minimum reachable distances of arm 4 or the tool. The AR headset can render an image of the workspace volume to help bedside personnel position arm 4 or the tool with greater accuracy. In one embodiment, the AR headset can receive a 3D scan of the patient from the surgical robot system 1. The AR headset can render an image of the cannula based on the 3D scan to guide bedside personnel in positioning the arm for engagement with the cannula.

[0047] Figure 5 A hologram rendered by an AR headset, illustrating various aspects of the technology described herein, is shown. The hologram highlights the workspace of the robotic arm and the attached endoscope to aid medical personnel in positioning the robotic arm. The hologram can guide the user in positioning the arm and endoscope to provide camera visualization when setting the starting position for surgical procedures. The hologram can display the viewpoint relative to the endoscope's position and orientation, as well as a video stream from the camera.

[0048] Figure 6 This paper illustrates information exchange between an augmented reality headset and a surgical robot system 1 according to various aspects of the subject matter, enabling the augmented reality headset to display spatial, systemic, and temporal information of the components of the surgical robot system 1 in the headset's coordinate system based on establishing a common coordinate system between the augmented reality headset and the surgical robot system 1 using fixed reference markers. Compared to... Figure 2 Surgical robotic system 1, Figure 6 The surgical robot system 1 lacks a set of RGBD sensors to capture images of the surgical robot system 1. These images can be analyzed using point clouds from the RGBD sensors of the headset to establish a common coordinate system between the AR headset and the surgical robot system 1.

[0049] Instead, to establish the position and orientation information of arm 4 and table 5 in the coordinate system of the AR headset, the AR headset can capture one or more predetermined and fixed reference markers on table 5. In one embodiment, the reference markers may be QR codes. The position of the reference markers can be known based on the coordinate system of surgical robot system 1. By analyzing the RGBD image of the reference markers in the coordinate system of the AR headset based on the knowledge of the fixed position of the reference markers in the coordinate system of surgical robot system 1, the AR headset can establish a common coordinate system between the AR headset and surgical robot system 1. In one embodiment, the AR headset can transmit the point cloud of the reference markers to surgical robot system 1 for the surgical robot system 1 to establish a common coordinate system. In one embodiment, the AR headset can run an object recognition algorithm on the RGBD image of surgical robot system 1 to identify arm 4 and table 5. In one embodiment, surgical robot system 1 may have Figure 2 The system includes both RGBD sensors and fixed reference markers. Images of the reference markers captured by the RGBD sensors of the AR headset, as well as images of arm 4 and platform 5, can be analyzed in series with images captured by the RGBD sensors of the surgical robot system 1 to establish a common coordinate system between the AR headset and the surgical robot system 1.

[0050] Once a common coordinate system is established, the surgical robot system 1 can transmit information to the AR headset so that the AR headset can create 3D virtual models of arms 4 and platform 5 that match their actual positions and orientations, or create 3D virtual models of arms 4 and platform 5 that can be integrated with their real-time real-world scene positions and orientations. In one implementation, the surgical robot system 1 can transmit to the AR headset real-time information on the positions and orientations of arms 4 and platform 5, 3D mesh models of components of the surgical robot system 1 or the operating room, real-time system status information of the surgical robot system 1, etc.

[0051] As the coordinate system of the AR headset changes relative to the surgical robot system 1, the SLAM algorithm running on the AR headset can locate the virtual position and orientation of the arms 4, platform 5, and other components of the surgical robot system 1. The AR headset can render context-sensitive information about the arms 4 and platform 5 based on real-time information received from the surgical robot system 1. This context-sensitive information can be positioned relative to the arms 4 and platform 5 to guide the wearer in configuring, operating, or diagnosing faults in the surgical robot system 1 before, during, or after surgery.

[0052] The AR headset can offer additional features such as a web portal (e.g., a browser) and can display information such as case settings, surgeon preference cards, instrument lifespan, archives, snapshots from the endoscope for archiving, photos of patients for archiving, patient data, etc. Other features may include: teleconferencing using a microphone, speaker, and rear-facing webcam; user authentication using a fingerprint reader and / or NFC card reader; and range detection from the AR headset to control tower 3 using Bluetooth and WiFi to monitor the signal strength of the connection and trigger an alarm when the interface device is too far away.

[0053] Figure 7 This is a block diagram illustrating exemplary hardware components of an AR headset 60 and a surgical robot system according to various aspects of the subject matter. The surgical robot system may include a surgical robot 80 and a control tower 70. The surgical robot system may include other or additional hardware components; therefore, this diagram is provided by way of example and not as a limitation on the system architecture.

[0054] The AR headset 60 includes a camera 51, a sensor 52, a display 53, a user command interface 54, a processor 55, a memory 56, and a network interface 57. The camera 51 and sensor 52 can be configured as RGBD sensors to capture color and depth image information of a surgical robotic system. Images captured by the camera 51 and sensor 52, or virtual images rendered by the AR headset 60, can be projected onto the display 53.

[0055] Processor 55 may be configured to run image processing algorithms to process images captured by camera 51 and sensor 52, thereby automatically identifying components of surgical robot 80, such as arm 4 or bed 5. In one embodiment, based on real-time system status information received from the surgical robot system (e.g., control tower 70) and a common coordinate system between AR headset 60 and surgical robot 80, processor 55 may be configured to create 3D virtual models of surgical robot 80 components that match their actual positions and orientations, or create 3D virtual models of surgical robot 80 components that can be fused with their real-time real-world scene positions and orientations. In one embodiment, processor 55 may run SLAM algorithms to locate the virtual positions and orientations of surgical robot 80 components as the coordinate system of AR headset 60 changes relative to the coordinate system of surgical robot 80. Processor 55 may be configured to run an operating system to control the operation of interface device 50. Memory 56 may store image processing algorithms, virtual image rendering algorithms, SLAM algorithms, operating system, program code, and other data used by processor 55.

[0056] User command interface 54 may include interfaces for other features such as a web portal. Hardware components may communicate via a bus. The interface device may communicate with the surgical robot system via external interface 58 using network interface 57. External interface 58 may be a wireless or wired interface.

[0057] The control tower 70 may include a mobile field care cart housing a touchscreen display, a computer controlling the surgeon's robotic-assisted manipulation of instruments, a computer for establishing a common coordinate system between the surgical robot 80 and the AR headset 60, a security system, a graphical user interface (GUI), light sources, and a video and graphics computer. The control tower 70 may include a central computer 71 (which may include at least a visualization computer, a control computer, and an auxiliary computer), various displays 73 (which may include team displays and nurse displays), and a network interface 78 coupling the control tower 70 to both the AR headset 60 and the surgical robot 80. The control tower 70 may also accommodate third-party devices such as an advanced light engine 72, an electrosurgical unit (ESU) 74, and inhalers and CO2 canisters 75. The control tower 70 may provide additional user-friendly features such as a nurse-display touchscreen, soft power and E-hold buttons, a user-facing USB port for video and still images, and an electronic caster control interface. The auxiliary computer may also run real-time Linux, providing logging / monitoring and interaction with cloud-based web services.

[0058] The surgical robot 80 includes an articulated operating table 84 with multiple integrated arms 82 positioned above the target patient anatomy. A set of compatible tools 83 can be attached to / detached from the distal ends of the arms 82, enabling the surgeon to perform various surgical procedures. The surgical robot 80 may also include a control interface 85 for manually controlling the arms 82, the operating table 84, and the tools 83. The control interface 85 may include items such as, but not limited to, remote controls, buttons, panels, and touchscreens. Other accessories such as cannulas (cannulas, sealing cartridges, and tamponades) and drapes can also be manipulated to perform surgery using the system. In one embodiment, the multiple arms 82 may include four arms mounted on both sides of the operating table 84, with two arms on each side. For a particular surgical procedure, an arm mounted on one side of the operating table 84 can be positioned on the other side of the operating table 84 by stretching and crossing under the operating table 84 and the arm mounted on the other side, resulting in a total of three arms positioned on the same side of the operating table 84. The surgical instrument may also include a computer 81 and a network interface 88, which allows the surgical robot 80 to be positioned to communicate with the control tower 70.

[0059] Figure 8This is a flowchart 800 illustrating a method for exchanging information between a surgical robotic system and an AR headset according to various aspects of the subject matter, to establish a common coordinate system between the AR headset and the surgical robotic system, and to transmit spatial, system, and temporal information of the components of the surgical robotic system to the AR headset for rendering the component and system information into a virtual image. The surgical robotic system can be... Figure 2 , Figure 6 or Figure 7 Surgical robotic systems.

[0060] In box 801, the surgical robot system receives image data captured by the AR headset. In one embodiment, the image data may be RGBD image data of the arm and stage of the surgical robot system captured by the RGBD sensor of the AR headset. This image data is based on the coordinate system of the AR headset.

[0061] In block 803, the surgical robot system receives image data captured by the surgical robot system. In one embodiment, the image data may be RGBD image data of the arm and stage of the surgical robot system captured by the RGBD sensors of the surgical robot system. This image data is based on the coordinate system of the surgical robot system.

[0062] In block 805, the surgical robot system establishes a common coordinate system between the AR headset and the surgical robot system based on image data from the AR headset and image data captured by the surgical robot system. In one embodiment, the surgical robot system can process image data of the arm and stage based on the AR headset's coordinate system, image data of the arm and stage based on the surgical robot system's coordinate system captured by the surgical robot system, and a real-time data stream describing the position and orientation information of the arm and stage, to identify and virtually establish the 3D position and orientation of the arm and stage in the AR headset's coordinate system.

[0063] In box 807, the surgical robot system transmits real-time information about itself based on a common coordinate system to the AR headset, enabling the AR headset to create 3D virtual models of the surgical robot system's components within its own coordinate system. In one embodiment, the real-time information of the surgical robot system may include: real-time information on the position and orientation of the arm and stage, analysis results of image data received from the AR headset, 3D mesh models of the surgical robot system's components, and real-time system status information of the surgical robot system. In one embodiment, the real-time system status information of the surgical robot system may include: joint angles and rotation degrees of the arm, type of tool attached to the arm, tool gripping state, active tool energy, arm mode, fault diagnosis error codes, etc. The AR headset can create 3D virtual models of the arm and stage that match their actual position and orientation, 3D virtual models of the arm and stage that can be integrated with the real-time scene position and orientation of the arm and stage, or other context-sensitive information about the arm and stage.

[0064] Real-time information from the surgical robotic system, transmitted to an AR headset based on a shared coordinate system, can be used by the AR headset to maintain the position and orientation of components of the surgical robotic system, such as arms and tables, as the AR headset's coordinate system changes relative to the surgical robotic system. In one implementation, a SLAM algorithm can be run on the AR headset to locate its position and orientation, such that virtual images of the surgical robotic system components are spatially fixed to maintain their virtual position and orientation as the wearer moves around the operating room. For example, when the AR headset's coordinate system changes relative to the surgical robotic system's coordinate system, the virtual reconstruction of the arms and tables can be matched to their real-time, real-world position and orientation.

[0065] Figure 9 This is a flowchart illustrating a method 900 for establishing a common coordinate system between an AR headset and a surgical robotic system, and receiving spatial, system, and temporal information of components of the surgical robotic system for rendering the component and system information into a virtual image, according to various aspects of the subject matter. The AR headset can be... Figure 2 , Figure 6 or Figure 7 AR headset.

[0066] In box 901, the AR headset captures image data of the surgical robotic system. In one embodiment, the image data may be RGBD image data of the arm and stage of the surgical robotic system captured by the RGBD sensor of the AR headset. In another embodiment, the image data may be RGBD image data of predetermined and fixed reference markers (e.g., QR codes) displayed on the surgical robotic system. The image data is based on the coordinate system of the AR headset.

[0067] In box 903, the AR headset establishes the 3D position and orientation of the surgical robot system in the AR headset's coordinate system based on captured image data. A global coordinate system can be established for both the AR headset and the surgical robot system to create or render virtual components of the surgical robot system within the AR headset's coordinate system. In one implementation, the AR headset can transmit RGBD image data of the arm and stage of the surgical robot system to the surgical robot system. The surgical robot system can establish a common coordinate system between the AR headset and the surgical robot system based on image data of the arm and stage in the AR headset's coordinate system, image data of the arm and stage in the surgical robot system's coordinate system captured by the surgical robot system, and a real-time data stream describing the position and orientation information of the arm and stage.

[0068] In one implementation, the AR headset can analyze the RGBD image of the reference marker in the coordinate system of the AR headset based on the knowledge of the fixed position of the reference marker in the coordinate system of the surgical robot system, in order to establish a common coordinate system between the AR headset and the surgical robot system.

[0069] In box 905, the AR headset receives real-time information from the surgical robot system. In one embodiment, the real-time information may include: real-time information on the position and orientation of the arm and table, analysis results of image data received from the AR headset, 3D mesh models of the components of the surgical robot system, and real-time system status information of the surgical robot system. In one embodiment, the real-time system status information may include: joint angles and rotation degrees of the arm, type of tool attached to the arm, tool gripping state, active tool energy, arm mode, fault diagnosis error codes, etc. In one embodiment, if the surgical robot system establishes a common coordinate system between the AR headset and the surgical robot system, the information received from the surgical robot system may have been transformed from the surgical robot system to the AR headset's coordinate system. In one embodiment, if the AR headset establishes a common coordinate system between the AR headset and the surgical robot system using reference markers, the information received from the surgical robot system may be located in the surgical robot system's coordinate system. The AR headset can transform information from the surgical robot system's coordinate system to its own coordinate system based on the common coordinate system.

[0070] In box 907, the AR headset can create 3D virtual models of the components of the surgical robotic system in the coordinate system of the AR headset based on the received information. In one embodiment, the AR headset can create 3D virtual models of the arms and platforms that match their actual positions and orientations, 3D virtual models of the arms and platforms that can be fused with their real-time real-scene positions and orientations, or other context-sensitive information about the arms and platforms. In one embodiment, the context-sensitive information can be positioned relative to the arms and platforms to guide the wearer in configuring, operating, or diagnosing faults in the surgical robotic system before, during, or after surgery.

[0071] In box 909, as the coordinate system of the AR headset changes relative to the surgical robot system, the AR headset maintains the position and orientation of components of the surgical robot system, such as the arms and table. In one implementation, a SLAM algorithm can be run on the AR headset to locate its position and orientation, such that virtual images of the components of the surgical robot system are spatially fixed to maintain their virtual position and orientation as the wearer moves around in the operating room. For example, as the coordinate system of the AR headset changes relative to the coordinate system of the surgical robot system, the virtual reconstruction of the arms and table can be matched to the real-time position and orientation of the arms and table in the real-world scene.

[0072] For purposes of explanation, the foregoing description uses specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that specific details are not required to practice the invention. The foregoing description of specific embodiments of the invention has been provided for illustrative and descriptive purposes. These are not intended to be exhaustive or to limit the invention to the specific forms disclosed; various modifications and alterations can be made to this disclosure in light of the foregoing teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application. Therefore, these embodiments enable others skilled in the art to best utilize the invention, as well as various embodiments with modifications suitable for the contemplated particular uses. The following claims and their equivalents are intended to define the scope of the invention.

[0073] The methods, devices, processes, and logic components described above can be implemented in a variety of different ways and in a variety of different combinations of hardware and software. Controllers and estimators may include electronic circuitry. For example, all or part of an implementation may be a circuit including an instruction processor, such as a central processing unit (CPU), microcontroller, or microprocessor; an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA); or a circuit including discrete logic components or other circuit components (including analog circuit components, digital circuit components, or both); or any combination thereof. As an example, the circuit may include discretely interconnected hardware components and / or may be combined on a single integrated circuit die, distributed among multiple integrated circuit dies, or implemented in a multi-chip module (MCM) of multiple integrated circuit dies in a co-package.

[0074] The circuit may also include or access instructions that are executed by the circuit. These instructions may be stored in a tangible storage medium other than transient signals, such as flash memory, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM); or on a magnetic disk or optical disk, such as an optical disc read-only memory (CDROM), a hard disk drive (HDD), or other magnetic disk or optical disk; or in or on another machine-readable medium. A product (such as a computer program product) may include a storage medium and instructions stored in or on that medium, and these instructions, when executed by circuitry in the device, may cause the device to perform any of the processes described above or shown in the accompanying drawings.

[0075] These implementations can be distributed as circuits among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems. Parameters, databases, and other data structures can be stored and managed separately, or combined into a single memory or database. They can be organized logically and physically in a variety of different ways and implemented in a variety of different ways, including as data structures such as linked lists, hash tables, arrays, records, objects, or implicit storage mechanisms. Programs can be parts of a single program (e.g., subroutines), stand-alone programs, distributed across multiple memories and processors, or implemented in a variety of different ways, such as in libraries, such as shared libraries (e.g., dynamic link libraries (DLLs)). For example, when executed by the circuit, the DLL can store instructions for performing any of the processes described above or shown in the figures.

[0076] Furthermore, the various controllers discussed herein may take the form of, for example, processing circuitry, microprocessors or processors, and computer-readable media storing computer-readable program code (e.g., firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Controllers may be configured with hardware and / or firmware to perform the various functions described below and shown in the flowcharts. Additionally, some components shown as being internal to the controller may also be stored externally, and other components may be used.

Claims

1. An augmented reality (AR) device for a surgical robotic system, comprising: Sensors configured to capture image data of a first component of the surgical robotic system; Processor, the processor being configured to: A global coordinate system is established for the AR device and the surgical robot system based on the image data, so as to create virtual components of the surgical robot system in the coordinate system of the AR device; Receive spatial information and real-time system status information of the surgical robot system; as well as Based on the spatial information and real-time system status information of the surgical robot system and based on the global coordinate system, a 3D virtual model of the second component of the surgical robot system is created in the coordinate system of the AR device. as well as A display configured to present the 3D virtual model of the second component of the surgical robot system; The spatial information of the surgical robot system includes the spatial information of the second component, and the processor is configured to create the 3D virtual model of the second component, including: Based on the spatial information of the second component, the position and orientation of the second component of the surgical robot system are created in the coordinate system of the AR device, wherein the position and orientation of the second component in the coordinate system of the AR device are matched with the actual position and orientation of the second component or the target position and orientation of the second component; The real-time system status information of the surgical robot system includes the real-time system status information of the second component, and the processor configured to create the 3D virtual model of the second component further includes: The context-sensitive information of the second component of the surgical robot system is created based on the real-time system state information of the second component.

2. The AR device of claim 1, wherein the processor is configured to establish the global coordinate system for the AR device and the surgical robot system, comprising: The AR device transmits the image data of the first component to the surgical robot system, so that the surgical robot system can establish the global coordinate system with respect to the coordinate system of the AR device and the coordinate system of the surgical robot system.

3. The AR device according to claim 2, wherein the spatial information of the surgical robot system is received in the coordinate system of the AR device.

4. The AR device of claim 1, wherein the first component of the surgical robot system is a reference marker of the surgical robot system, and wherein the processor is configured to establish the global coordinate system for the AR device and the surgical robot system, comprising: The image data of the reference marker is analyzed based on the fixed position of the reference marker in the coordinate system of the surgical robot system, so as to establish the global coordinate system relative to the coordinate system of the AR device.

5. The AR device of claim 4, wherein the spatial information of the surgical robot system includes the spatial information of the second component in the coordinate system of the surgical robot system, and wherein the processor is configured to create the 3D virtual model of the second component of the surgical robot system in the coordinate system of the AR device, comprising: The spatial information of the second component in the coordinate system of the surgical robot system is transformed to the coordinate system of the AR device based on the global coordinate system.

6. The AR device according to claim 1, further comprising: The processor is configured to maintain the position and orientation of the 3D virtual model of the second component of the surgical robot system when the coordinate system of the AR device changes relative to the coordinate system of the surgical robot system.

7. The AR device of claim 1, wherein the second component of the surgical robot system comprises the robotic arm or operating table of the surgical robot system.

8. The AR device of claim 7, wherein the 3D virtual model of the second component comprises: 3D virtual rendering of the position and orientation of the robotic arm or the operating table; as well as Visual or auditory communication of the real-time system status information of the robotic arm or the operating table.

9. A method for interacting with a surgical robotic system, comprising: Augmented reality (AR) headsets are used to capture image data of the first component of the surgical robotic system; Based on the image data and based on the global coordinate system for the AR headset and for the surgical robot system, establish the three-dimensional (3D) position and orientation of the surgical robot system in the coordinate system of the AR headset; The AR headset receives spatial information and real-time system status information of the surgical robot system; The AR headset creates a 3D virtual model of the second component of the surgical robot system in the coordinate system of the AR headset based on the spatial information and real-time system state information of the surgical robot system, the 3D position and orientation of the surgical robot system in the coordinate system of the AR headset, and the global coordinate system; and The 3D virtual model of the second component of the surgical robot system is maintained when the coordinate system of the AR headset changes relative to the coordinate system of the surgical robot system. The spatial information of the surgical robot system includes the spatial information of the second component, and the creation of the 3D virtual model of the second component includes: The second component of the surgical robot system is rendered in the coordinate system of the AR headset based on the spatial information of the second component, wherein the position and orientation of the second component rendered in the coordinate system of the AR headset match the actual position and orientation of the second component or the target position and orientation of the second component. The real-time system status information of the surgical robot system includes the real-time system status information of the second component, and the creation of the 3D virtual model of the second component further includes: The context-sensitive information of the second component of the surgical robot system is rendered based on the real-time system state information of the second component.

10. The method of claim 9, wherein establishing the 3D position and orientation of the surgical robot system in the coordinate system of the AR headset comprises: The image data of the first component is transmitted to the surgical robot system so that the surgical robot system can establish the global coordinate system with respect to the coordinate system of the AR headset and the coordinate system of the surgical robot system.

11. The method of claim 9, wherein the first component of the surgical robot system is a reference marker of the surgical robot system, and wherein establishing the 3D position and orientation of the surgical robot system in the coordinate system of the AR headset comprises: The image data of the reference marker is analyzed based on the fixed position of the reference marker in the coordinate system of the surgical robot system to establish the global coordinate system for the coordinate system of the AR headset and the coordinate system of the surgical robot system.

12. The method of claim 9, wherein the second component of the surgical robot system comprises the robotic arm or operating table of the surgical robot system.

13. The method of claim 12, wherein the 3D virtual model of the second component of the surgical robot system provides real-time guidance on manual or automatic manipulation of the arm of the surgical robot system.

14. The method of claim 12, wherein the 3D virtual model of the second component of the surgical robot system provides a real-time rendering of the workspace of the arm of the surgical robot system.

15. A surgical robotic system, comprising: Operating table; Multiple robotic arms are coupled to the operating table; A processor, which is communicatively coupled to the operating table, the plurality of robotic arms, and the augmented reality (AR) headset; as well as The AR headset is configured as follows: Capture image data of the operating table or one or more of the plurality of robotic arms; Collaborate with the processor to establish a global coordinate system based on the image data for the AR headset and for the operating table or the multiple robotic arms; The processor receives spatial information of the operating table or the multiple robotic arms and real-time system status information. Based on the spatial information and the real-time system state information, and based on the global coordinate system, create a 3D virtual model of the operating table or one or more robotic arms among the multiple robotic arms in the coordinate system of the AR headset; and Display the 3D virtual model of the operating table or one or more of the plurality of robotic arms; The 3D virtual model of the operating table or one or more of the plurality of robotic arms includes context-sensitive real-time information about the operating table or the robotic arm to assist the user of the surgical robot system in manipulating the operating table or the robotic arm or diagnosing faults.

Images