Remote surgical guidance method and system using augmented reality

Abstract

A virtual representation of the OR is generated based on robotic information and sensing of the operating room by depth cameras. One of the depth cameras is integrated with a portable electronic device operated by a local user in the operating room. The virtual representation of the OR is communicated to a virtual reality headset along with three-dimensional point cloud data. A virtual reality environment is presented to a display of the virtual reality headset operated by a remote user. A virtual representation of the remote user is presented in augmented reality to a display of the portable electronic device.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of surgical robotics, and more particularly to remote surgical guidance. BACKGROUND

[0002] Minimally invasive surgery (MIS), such as laparoscopic surgery, involves techniques intended to reduce tissue damage during a surgical procedure. For example, a laparoscopic procedure typically involves making a plurality of small incisions in a patient (e.g., in the abdomen), and introducing one or more tools and at least one endoscopic camera into the patient’s body through the incisions. The surgical procedure is then performed by using the introduced tools, with visualization assistance provided by the camera.

[0003] Generally, MIS provides multiple benefits, such as reduced patient scarring, reduced patient pain, shortened patient recovery period, and reduced medical expenses associated with patient recovery. In some embodiments, MIS can be performed with a surgical robotic system that includes one or more robotic arms for manipulating surgical instruments based on commands from an operator. For example, the operator can provide commands for manipulating the surgical instruments while viewing images provided by the camera and displayed to the user on a display.

[0004] Performance of a surgical procedure can require in-depth knowledge and experience regarding different aspects of the surgical procedure, including setup, workflow, etc. Use of a surgical robotic system potentially increases the complexity of the surgical procedure. It can be helpful for a surgeon or assistant to collaborate with medical professionals at a remote location pre-, intra-, and post-operatively. Through remote collaboration, procedures can be improved and risks can be identified and reduced. SUMMARY

[0005] Generally, a surgical robotic system has a remote guidance feature that allows for pre-, intra-, and post-operative collaboration. The remote guidance feature can provide an intuitive and robust method for guiding intra-operative procedures via a remote guide. A remote user (“guide”) wears a virtual reality (VR) headset to choreograph an avatar, which is a digital representation of the guide, that is virtually introduced into the operating room and is visible to a local user controlling or wearing an augmented reality (AR) device, such as a computer tablet, a mobile smartphone, or a headset (e.g., augmented reality or mixed reality glasses). AR refers to superimposing a virtual object (e.g., the digital representation of the guide) on a video feed showing the AR user’s physical environment (in this case, the operating room). The virtual object is typically integrated into the user’s physical environment, e.g., by interacting with the physical environment. It should be understood that for the purposes of the present disclosure, the terms augmented reality and mixed reality are interchangeable.

[0006] The remote VR user and the local intra-operative AR user are both defined in a shared coordinate system relative to the Verb robotic system. In other words, they have a common coordinate system so that they are co-located in the same virtual space in a realistic way. The remote virtual reality user is not only visible through an avatar, but can also create annotations and markers located within the 3D space, provide recommendations and examples of robotic arm movements / positions, and recommend surgical table orientation / position. Audio streams of the remote user and the local user can be transmitted back and forth between the remote user and the local user to facilitate verbal communication. Thus, input from the remote user (e.g., annotations, markers, recommendations, and / or audio) can provide helpful insights to the local user about different aspects of the surgical procedure.

[0007] In turn, point cloud information of the physical operating room environment can be captured by depth cameras and transmitted back to the virtual reality environment for the remote user to view and manipulate. By using an augmented reality headset / tablet as a sensor source and data generator, the remote VR user can construct a 3D mesh of the patient in VR that will indicate and enable intra-operative guidance.

[0008] In some embodiments, the system can include multiple depth cameras (e.g., RGBD scanners or equivalent technology). At least one of the depth cameras is integrated with a portable electronic device operated by a local user in the operating room (OR). Another of the depth cameras is arranged in the OR, such as mounted on equipment and / or a wall. The system can include a surgical robot, such as a robotic arm and platform for supporting a patient during a surgical procedure, and a controller for commanding movements of the surgical robot.

[0009] A virtual representation of the OR is generated based on a) robot information received from the surgical robot or the controller and b) sensing of the OR by the multiple depth cameras. The virtual representation of the OR and three-dimensional point cloud data are transmitted to a virtual reality headset for rendering a virtual reality environment to a display of the virtual reality headset worn by the remote user. In this way, the remote user can“see” what is happening in the OR and provide insights.

[0010] A virtual representation of a remote user is presented in augmented reality at a portable electronic device. The virtual representation can be a graphical object superimposed on the“live” video feed of the OR. The virtual representation of the remote user can be generated based on the location of the remote user such that the virtual representation of the remote user shares a common coordinate system that is common to the virtual representation of the OR, the surgical robot, and the portable electronic device of the local user. In other words, if the remote user moves forward toward the patient in the virtual representation of the OR (as shown to the remote user in virtual reality headgear), the avatar of the remote user will simultaneously move forward toward the patient as shown in the augmented reality to the local user. Similarly, if the remote user“points” to a portion of the patient in the virtual reality environment shown to the remote user, the avatar of the remote user will point to the corresponding portion of the patient in the augmented reality shown to the local user. This creates a“real” interaction between the local user and the remote user and merges the augmented reality of the local user with the virtual reality of the remote user.

[0011] Inputs from the remote user such as audio data (e.g., the voice of the remote user) and virtual annotations generated by a digital tool (e.g., a handheld user input device held by the remote user) can be indicated through the virtual representation of the remote user. The virtual representation can be a simplified, realistic, or symbolic representation of the person (e.g., head and hands) that conveys the gestures, movements, location, and / or intent of the remote user to the user. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 An example of a surgical robot system in an operating room is shown in accordance with some embodiments.

[0013] Figure 2 A surgical robot system with remote tele-guidance features is shown in accordance with some embodiments.

[0014] Figure 3 A remote user in an operating room is virtually introduced in an augmented reality experience in accordance with some embodiments.

[0015] Figure 4 An operating room and a local user are virtually introduced in a virtual reality experience of a remote user in accordance with some embodiments.

[0016] Figure 5 A method for remote guidance with a surgical robot system is shown in accordance with some embodiments. DETAILED DESCRIPTION

[0017] Non-limiting examples of various aspects and variations of the present invention are described herein and shown in the accompanying drawings.

[0018] REFERENCE Figure 1This is a pictorial view of an example surgical robotic system 1 in an operating arena. The system 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 at a surgical robotic platform 5 (e.g., a table, bed, etc.). The arms 4 can be mounted to a table or bed on which a patient 6 rests, as shown by the example Figure 1 , or they can be mounted to a cart that is separate from the table or bed. The system 1 can incorporate any number of devices, tools, or accessories used to perform a surgical procedure on the patient 6. For example, the system 1 can include one or more surgical tools 7 used to perform a surgical procedure. The surgical tools 7 can be end effectors attached to distal ends of the surgical arms 4 for performing a surgical procedure.

[0019] Each of the surgical tools 7 can be manually manipulated, robotically manipulated, or both during a surgical procedure. For example, the surgical tools 7 can be tools used to access, view, or manipulate the internal anatomy of the patient 6. In one aspect, the surgical tools 7 are graspers that can grasp tissue of the patient. The surgical tools 7 can be configured to be manually controlled by a bedside operator 8, robotically controlled via actuated movement of the surgical robotic arms 4 to which they are attached, or both. The robotic arms 4 are shown as being table-mounted, but in other configurations, the arms 4 can be mounted to a cart, a ceiling, or a sidewall, or to another suitable structural support.

[0020] A remote operator 9, such as a surgeon or other human operator, can use the user console 2 to remotely manipulate the arms 4 and their attached surgical tools 7, e.g., in what is referred to herein as teleoperation. The user console 2 can be located in the same operating room as the rest of the system 1, as shown in Figure 1 . However, in other environments, the user console 2 can be located in an adjacent or nearby room, or it can be located at a remote location, e.g., in a different building, city, or country. The user console 2 can include a seat 10, foot controls 13, one or more handheld user input devices (UIDs) 14, and at least one user display 15 configured to display, e.g., a view of a surgical procedure site within the patient 6. In the example user console 2, the remote operator 9 sits in the seat 10 and views the user display 15 while manipulating the foot controls 13 and the handheld UIDs 14 in order to remotely control the arms 4 and the surgical tools 7 mounted on distal ends of the arms 4.

[0021] In some variations, the bedside operator 8 can operate the system 1 in an "over-bed" mode, where the bedside operator 8 (user) is positioned at the side of the patient 6 and simultaneously manipulates both the robotically-driven tools (end effectors attached to the arms 4) and the manual laparoscopic tools. For example, the bedside operator's left hand can manipulate the handheld UID to control the robotically-driven tools, while the bedside operator's right hand can manipulate the manual laparoscopic tools. In this particular variation of the system 1, the bedside operator 8 can perform both robotically-assisted minimally invasive surgical procedures and manual laparoscopic surgical procedures on the patient 6.

[0022] During an example procedure (surgical procedure), the patient 6 is prepared for surgery and draped in a sterile manner with a sterile drape to achieve anesthesia. Initial access to the surgical site can be performed manually (to facilitate access to the surgical site) while the arms of the robotic system 1 are in a stowed configuration or a retracted configuration. Once access is complete, initial positioning or preparation of the robotic system 1, including its arms 4, can be performed. The surgical procedure then continues with the remote operator 9 at the user console 2 manipulating various end effectors, and possibly an imaging system, with the foot-operated controls 13 and the UID 14 to perform the surgical procedure. Human assistance can also be provided at the operating bed or table by a bedside staff member (e.g., the bedside operator 8) who can be dressed in sterile surgical gowns, who can perform tasks such as retracting tissue, performing manual repositioning, and tool changes on one or more of the robotic arms 4. There can also be non-sterile personnel to assist the remote operator 9 at the user console 2. When the procedure or surgical procedure is complete, the system 1 and the user console 2 can be configured or set into a state to facilitate completion of post-operative procedures, such as cleaning or sterilization, and inputting or printing of health records via the user console 2.

[0023] In one embodiment, the remote operator 9 holds and moves the UID 14 to provide input commands to move the robot arm actuators 17 in the robotic system 1. The UID 14 can be communicatively coupled to the rest of the robotic system 1, e.g., via the console computer system 16. The UID 14 can generate spatial state signals corresponding to the movement of the UID 14, e.g., the position and orientation of the handheld housing of the UID, and the spatial state signals can be input signals that control the motion of the robot arm actuators 17. The robotic system 1 can use control signals derived from the spatial state signals to control the proportional motion of the actuators 17. In one embodiment, a console processor of the console computer system 16 receives the spatial state signals and generates corresponding control signals. Based on these control signals that control how the actuators 17 are energized to move segments or links of the arm 4, the movement of the corresponding surgical tool attached to the arm can mimic the movement of the UID 14. Similarly, interactions between the remote operator 9 and the UID 14 can generate, e.g., a clamp control signal that causes the jaws of a grasper of the surgical tool 7 to close and clamp tissue of the patient 6.

[0024] The surgical robotic system 1 can include several UIDs 14, with each UID generating a corresponding control signal for controlling the actuators and surgical tools (end effectors) of the respective arm 4. For example, the remote operator 9 can move a first UID 14 to control the motion of an actuator 17 located in a left robot arm, with the actuator responding by moving links, gears, etc. in the arm 4. Similarly, movement of a second UID 14 by the remote operator 9 controls the motion of another actuator 17, which in turn moves other links, gears, etc. of the robotic system 1. The robotic system 1 can include a right arm 4 that is fixed to a bed or table on the right side of the patient, and a left arm 4 that is located on the left side of the patient. The actuators 17 can include one or more motors that are controlled so that they drive the articulations of the arm 4 to rotate, e.g., to change the orientation of an endoscope or grasper of a surgical tool 7 attached to the arm relative to the patient. The motion of several actuators 17 in the same arm 4 can be controlled by spatial state signals generated from a particular UID 14. The UIDs 14 can also control the motion of the respective surgical tool graspers. For example, each UID 14 can generate a corresponding grasp signal to control the motion of an actuator (e.g., a linear actuator) that opens or closes the jaws of a grasper at the distal end of the surgical tool 7 to grasp tissue within the patient 6.

[0025] In some respects, communication between platform 5 and user console 2 can be achieved via control tower 3, which translates user commands received from user console 2 (and more specifically from console computer system 16) into robot control commands transmitted to arm 4 on robot platform 5. Control tower 3 can also transmit status and feedback from platform 5 back to user console 2. The communication connection between robot platform 5, user console 2, and control tower 3 can be via wired and / or wireless links, using any suitable data communication protocol from a variety of data communication protocols. Any wired connection can optionally be integrated into the floor and / or walls or ceiling of the operating room. Robot system 1 can provide video output to one or more displays, including displays within the operating room and remote displays accessible via the Internet or other networks. Video output (video feed) can also be encrypted to ensure privacy, and all or part of the video output can be stored on a server or electronic healthcare record system.

[0026] Figure 2 A surgical robotic system 1 with remote features is illustrated, which allows surgeons to collaborate preoperatively, intraoperatively, and postoperatively. The development of real-time virtual reality and augmented reality collaborative platforms enables an interactive paradigm where remote surgical professionals can conveniently observe and advise on surgical procedures from different locations.

[0027] Processor 24 can generate a virtual representation of the operating room based on a) robot information and b) sensing of the operating room by one or more depth cameras (e.g., cameras 21, 23). The depth cameras may include at least one depth camera 23 integrated with a portable electronic device 22 operated by a local user in the operating room, and at least one stationary depth camera 21 positioned in a fixed location within the operating room (e.g., on a wall and / or on equipment). The depth cameras may be RGBD sensors or other equivalent technologies for sensing color and depth.

[0028] Robot information by Figure 1 The diagram illustrates the generation of components in a surgical robotic system. For example, the surgical robotic platform may provide telemetry generated by servo systems or sensors, indicating the platform's current position and orientation (e.g., platform height and angle). Similarly, sensors or controllers for the surgical robotic arm and surgical tools may provide telemetry describing the position of the surgical robotic arm (e.g., joint angles), the type of tool currently attached to the robotic arm, the tool gripping state (e.g., 90% closure), and the energy of the active tool. This data can be collected at a control tower or user console. Furthermore, system information such as system status (on, off, idle) or error codes may be collected and shared with remote users.

[0029] The processor can use robot information to generate virtual representations of corresponding robot components at locations indicated by the robot information, based on, for example, robot joint angles and platform height and angle. This avoids the need for extrapolating the precise positioning of the platform, robot arm, and tools from image data sensed by a depth camera, while also improving the accuracy of virtual representations of these components included in a virtual representation within the operating room.

[0030] In some embodiments, the portable electronic device 22 may be a tablet computer or a mobile smartphone 28. Augmented reality (e.g., a representation of a remote user and / or other virtual objects) is presented on the device's display, superimposed on a stream of images captured by the device. Alternatively, the portable electronic device may be a head-mounted device such as augmented reality or mixed reality glasses 39, having a transparent display worn in front of the eyes on which the augmented reality is presented. This is done in a "real-time" or "live" manner, meaning that the processing and display of the augmented reality are performed simultaneously with the capture of the image stream, despite unavoidable delays such as buffering, processing, and communication latency.

[0031] The virtual representation and 3D point cloud data of the operating room are transmitted to a virtual reality headset 26 (operated by a remote user), which is used to present the virtual reality environment to the headset's display. Here, the virtual reality environment is fully immersive, meaning that the content shown to the remote user is not integrated with the remote user's environment; rather, the remote user is virtually communicated to the operating room.

[0032] A virtual representation of the operating room can include a 3D rendering of the operating room and its components. In some cases, a virtualized operating room can include metadata such as equipment type, location, and orientation, allowing virtual reality headsets to render those objects into the virtual reality environment, thereby reducing the amount of data that must be transmitted to the headset, for example, if raw 3D point cloud data of all objects in the environment were to be transmitted. The virtual representation of the operating room can include a mesh representation of objects sensed in the operating room environment, walls, ceilings, and personnel (e.g., surgeons, assistants, etc.). The virtual representation of the operating room can be constructed based on 3D point cloud data captured by multiple depth cameras.

[0033] The virtual representation can be generated by processor 24. The processor can be integrated with a standalone computer, or with other computing systems such as controllers, laptop computers, portable electronic devices operated by a local user, or surgical robotic systems.

[0034] However, it should be recognized that for some areas, it is beneficial to transmit raw 3D point cloud data to a remote user, providing greater detail and resolution, and allowing greater ability for the remote user to interact with the sensed environment of the operating room. This 3D point cloud data can be generated by a depth camera of the portable electronic device 22, or by one or more depth cameras mounted on equipment or walls in the OR that are pointed at areas of interest. For example, the local user can aim the depth camera 23 of the portable electronic device at an area of interest on the patient (e.g., the patient’s abdomen). This raw 3D point cloud data can be transmitted to the remote VR headset to allow the remote user to analyze and manipulate this raw image data.

[0035] For example, a remote surgeon can perform segmentation of an organ of a patient while another surgeon is performing a surgical procedure, so that the intraoperative staff can receive real-time visual guidance on physiological landmarks. Segmentation refers to the process of dividing an image into regions of similar attributes, such as gray levels, colors, textures, brightness, and contrast. Thus, the remote user can subdivide objects in the image to help identify areas of interest (e.g., organs, tumors, lesions, other abnormal parts). Based on the raw point cloud data, the remote user can also identify the location of ports on the patient that serve as entry points for surgical robotic instruments. Thus, having a depth camera that can be directed to areas of interest will allow the local user to send detailed scans of the patient to the remote user.

[0036] The processor uses the robotic information and sensor data sensed by the depth camera to identify and virtually establish the 3D position and orientation of the surgical robotic system and its components in the coordinate system of the AR device. This can be performed by known computer vision algorithms, such as image classification and object localization, as well as performing transformations from one coordinate system to another to determine a common coordinate system.

[0037] Through this process, the depth camera, disposed across the surgical robotic system and the portable electronic device, establishes a common coordinate system and creates a virtual reconstruction of the robotic arms / table that matches the real-time position / orientation of the robotic arms / table in the room. After the components of the surgical robotic system are positioned in 3D space (e.g., the control tower, the surgeon bridge, and the table and arms), the remote participant can interact with the virtual reconstruction of the OR. The virtual representation of the remote user (e.g., the orientation / position of the remote user’s hands, body, and / or head) is transferred to the augmented reality user in the same relative coordinate system.

[0038] A simultaneous localization and mapping (SLAM) algorithm is performed on the data generated by the at least one depth camera integrated with the portable electronic device to localize the position and orientation of the portable electronic device 3D space in a common coordinate system.

[0039] The local user's portable electronic device and / or processor 24 can have a data connection with the surgical robotic system so that it can retrieve real-time robotic information (e.g., robotic arm joint angles, tool type, tool grip state, active tool energy, arm mode, table position and orientation, case surgeon, and troubleshooting error codes). This can be used to generate a virtual OR and provide notifications to the local user and the remote user.

[0040] In addition to positioning components of the surgical robotic system (e.g., the platform, the user console, the control tower, the robotic arms) in 3D space, a live "window" into the 3D intraoperative space is sent to the remote user by a stream of 3D point clouds generated by one or more depth cameras (e.g., 23 and / or 21). This point cloud provides a real-time stream of actual depth and RGB data from the OR, rather than a simplified stream of component position / orientation data, which adjusts the pre-rendered components of the surgical robotic system and the OR components in the virtual reality environment of the remote user. This point cloud stream, displayed in the correct coordinate space relative to the rest of the virtual reality environment, unlocks additional capabilities for the remote user.

[0041] For example, the remote user can see the patient's body in more detail through the 3D point cloud data (or mesh generated based on the 3D point cloud data) and create digital waypoints and virtual guides for where trocars should be placed in the patient (see, e.g., FIG. 6). Figure 4 Additionally or alternatively, the remote user can create digital waypoints and virtual guides for how the robotic arms should be oriented relative to the patient's body. Additionally or alternatively, the remote user can give educational advice about where personnel should stand in the operating room or where equipment should be arranged in the operating room. The local user interacts with the remote user through the portable device 22 as shown in Figure 3 .

[0042] Figure 3 The operating room and a virtual representation of the remote user placed in the operating room are shown, creating an augmented reality experience for the local user. The virtual representation of the remote user is presented in the augmented reality to the display of the portable electronic device 22 based on the remote user's location and input from the remote user. The remote user's location shares a common coordinate system, i.e., it is common to the virtual representation of the OR and the local user's portable electronic device.

[0043] The location of the remote user relative to the OR can be determined using one or more reference points at the remote location of the user. For example, at the remote location of the remote user, sensors can be disposed in the surrounding environment of the remote user and / or on the VR headset. An origin point can be selected (e.g., arbitrarily) in the location of the remote user to match the origin point in the space of the operating room. Transformations can be performed to map the movements and locations of the remote user to the virtual representation of the OR. Thus, the location and orientation of the remote user can be calculated relative to the coordinate system of the operating room, allowing the remote user to virtually explore and interact with the operating room.

[0044] As discussed in other sections, the local user can aim a depth camera, which can be integrated with the portable device 22, at a location of interest, such as the patient’s abdomen. Raw point cloud data generated by the depth camera can be transmitted in real-time to the remote user. Virtual markers and objects can be presented on the display of the portable device based on inputs generated by the remote user, such as: virtual annotations remotely generated by the digital tool, anatomical segmentation data, indications of trocar placement, and prior surgical setup.

[0045] For example, the remote user can use the handheld digital tool 27 as shown in Figure 2 to circle an area to be cut or removed, mark where the entry port of a trocar should be placed, or mark where a suture is needed. These “markers” can be virtually represented in the portable device of the local user by virtual objects, such as lines, highlighted areas, or shapes.

[0046] As discussed, inputs from the remote user can include audio data captured at the VR headset. Audio from the local user can be simultaneously transmitted to the VR headset, providing two-way communication. Voice over IP (VOIP) or other audio communication technology can be used.

[0047] Referring back to Figure 3 , when the remote user verbally indicates and marks where a trocar should be inserted, the local user can “see” and “hear” the remote user. The virtual presence of the remote user creates a natural tele-guidance platform. Spatially contextual 3D markers and annotations are communicated from the remote user to the local user. This can be done simultaneously with the performance of the surgical procedure. The remote user can provide visual guidance not only in the endoscope feed but also around the robot, e.g., how to orient the arms, locations for placement of endoscope and tools, where the first assistant should stand. The virtual representation of the remote user can include one or more of: a) a virtual representation of the virtual reality headset, b) a virtual hand indicating the remote user’s hand movements or hand position, and c) a virtual avatar resembling a portion of a whole person or a whole person.

[0048] Figure 4The virtual reality experience of a remote user is illustrated. The remote user is communicated to the operating room where the surgical robotic system is located. The remote user, wearing VR headset 26, can explore the virtual space and verbally and / or use remote digital tools such as a digital pen or a handheld UID (e.g., Figure 2 The remote digital tool 27 shown is used to provide input.

[0049] Virtualized OR data and point cloud data are transmitted to a VR headset, where they are used to present a virtual representation of the OR and detailed areas of points of interest (e.g., the patient) on the display of a remote user's VR headset. Furthermore, the position and orientation of the local user's portable electronic device 22 can be transmitted to the virtual reality headset. This can be used to present a virtual representation of the portable electronic device (e.g., floating in the virtual environment) and / or a virtual representation of the local user (e.g., an avatar) within the virtual reality environment. In this way, the remote user can feel as if she is interacting with the local user within the virtual OR.

[0050] As discussed in other sections, remote users can use medical imaging segmentation to highlight, segment, and visually annotate operation-related tissues and organs as the procedure is in progress. Remote users can load and overlay their own prior surgical setups onto the augmented reality user's existing intraoperative setup as guidance on how they can organize their own OR / robot.

[0051] According to some implementation plans, in Figure 5 The document illustrates method 40 for providing interactive and immersive remote guidance. The method can be provided by... Figure 1 and Figure 2 The surgical robot system shown performs the procedure. For example, the method can be executed by processor 24, which can be integrated with a stand-alone computer and the surgical robot system 1. Figure 1 or Figure 2 Components integrated with (as shown) or with Figure 2 The portable electronic device 22 is integrated.

[0052] At operation 41, the method includes generating a virtual representation of the OR based on a) robot information and b) sensing of the OR by one or more depth cameras. The depth cameras include at least one depth camera integrated with a portable electronic device operated by a local user in the OR, and at least one stationary depth camera arranged in the OR. The robot information can be used to more accurately generate the positions of robot components such as robotic arms, tools / instruments attached to them, and surgical platforms.

[0053] At operation 42, the method includes transmitting the virtual representation of the OR and the three-dimensional point cloud data to a virtual reality headset, where this is used to render the virtual reality environment to a display of the virtual reality headset operated by a remote user. The virtual representation of the OR can include mesh data describing the geometry of the operating room and objects in the operating room, and / or position and orientation data of identified objects in the operating room. The remote user is virtually teleported into the surgical operating room so that the remote user can analyze the setup, workflow, or provide other helpful insights to perform the surgical procedure.

[0054] At operation 43, the method includes rendering a virtual representation of the remote user in augmented reality to a display of the portable electronic device based on the position of the remote user and input from the remote user. The virtual representation of the remote user is rendered in a common coordinate system that is common to the virtual representation of the OR and the portable electronic device of the local user. The common coordinate system can be determined based on a transformation between image data captured by a depth camera attached to or integrated with the portable electronic device and image data captured by a depth camera installed in the operating room.

[0055] The method can be performed continuously and in real-time. For example, operations 42 and 43 are performed continuously and simultaneously to provide coherent interaction between the local user and the remote user.

[0056] The foregoing description, for purposes of explanation, uses specific nomenclature to provide a thorough understanding of the application. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the application. The foregoing descriptions of specific embodiments of the application have been presented for the purpose of illustration and description. They are not intended to be exhaustive or to limit the application to the precise forms disclosed; various modifications and changes can be made without departing from the scope and spirit of the application. The embodiments were chosen and described in order to best explain the principles of the application and its practical application, thereby enabling others skilled in the art to best utilize the application and various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. A method for remote guidance using a surgical robotic system, comprising: A virtual representation of the operating room OR is generated based on a) robot information and b) sensing of the operating room OR by one or more depth cameras, the one or more depth cameras including at least one depth camera integrated with a portable electronic device operated by a local user in the OR and at least one stationary depth camera arranged in the OR, the one or more depth cameras being configured to generate three-dimensional point cloud data. Transmit the virtual representation of the OR and the three-dimensional point cloud data of the virtual reality headset for presenting the virtual reality environment to the display of the virtual reality headset operated by a remote user; as well as Based on the remote user's location and input from the remote user, a virtual representation of the remote user is presented to the display of the portable electronic device in augmented reality. The location of the remote user is determined using one or more reference points at the location of the remote user.

2. The method according to claim 1, wherein, The three-dimensional point cloud data includes patients in the OR.

3. The method according to claim 1, wherein, The input from the remote user includes audio data captured at the virtual reality headset.

4. The method according to claim 1, wherein, The input from the remote user includes at least one of the following to be presented in the augmented reality on the portable electronic device: virtual annotations remotely generated by digital tools, anatomical segmentation data, cannula placement, and previous surgical settings.

5. The method according to claim 1, wherein, The portable electronic device of the local user is a tablet computer, and the augmented reality is presented on the display on top of the image stream captured by the tablet computer.

6. The method according to claim 1, wherein, The portable electronic device of the local user is a head-mounted device with a transparent display worn in front of the eyes, on which the augmented reality is displayed.

7. The method according to claim 1, wherein, The robot information includes at least one of the following: joint angles of one or more surgical robot arms, tool type, tool gripping state, active tool energy, position and orientation of the surgical table, and error codes of the surgical robot system.

8. The method according to claim 1, wherein, Simultaneous localization and mapping SLAM is performed on data generated by the at least one depth camera integrated with the portable electronic device to locate the position and orientation of the portable electronic device in a common coordinate system.

9. The method according to claim 1, wherein, The position and orientation of the portable electronic device are transmitted to the virtual reality headset for presenting a virtual representation of the portable electronic device in the virtual reality environment of the remote user or a virtual representation of the local user.

10. The method according to claim 1, wherein, The one or more depth cameras include RGBD cameras.

11. The method according to claim 1, wherein, The virtual representation of the remote user includes at least one of the following: a) a virtual representation of the virtual reality headset, b) a virtual hand indicating the movement or position of the remote user's hand, or c) a virtual avatar resembling a whole person or a part of a whole person.

12. A surgical robot system, comprising: Multiple depth cameras, including at least one depth camera integrated with a portable electronic device operated by a local user in the operating room OR and at least one stationary depth camera arranged in the OR; Surgical robots and controllers; as well as Processor, the processor being configured to perform the following operations: A virtual representation of the OR is generated based on a) robot information received from the surgical robot or controller and b) sensing of the OR by the plurality of depth cameras, which are configured to generate three-dimensional point cloud data. The virtual representation of the OR and the three-dimensional point cloud data are transmitted to a virtual reality headset, thereby causing the virtual reality headset to present a virtual reality environment to the display of the virtual reality headset worn by a remote user; as well as The location of the remote user and input from the remote user are transmitted to the portable electronic device of the local user for presenting a virtual representation of the remote user in augmented reality at the portable electronic device. The location of the remote user is determined by a sensor at the location of the remote user.

13. The surgical robot system according to claim 12, wherein, The three-dimensional point cloud data includes the patient's body.

14. The surgical robot system of claim 12, wherein, The input from the remote user includes audio data captured at the virtual reality headset.

15. The surgical robot system according to claim 12, wherein, The input from the remote user includes at least one of the following to be presented in the augmented reality on the portable electronic device: virtual annotations remotely generated by digital tools, anatomical segmentation data, cannula placement, and previous surgical settings.

16. The surgical robot system of claim 12, wherein, The portable electronic device of the local user is a tablet computer, and the augmented reality is presented on top of the image stream captured by the tablet computer and displayed on the screen of the tablet computer.

17. The surgical robot system of claim 12, wherein, The portable electronic device of the local user is a head-mounted display with a transparent display area on which the augmented reality is presented.

18. The surgical robot system of claim 12, wherein, The robot information includes at least one of the following: joint angles of one or more surgical robot arms, tool type, tool gripping state, active tool energy, position and orientation of the surgical table, and error codes of the surgical robot system.

19. The surgical robot system according to claim 12, wherein, Simultaneous localization and mapping SLAM is performed on data generated by the at least one depth camera integrated with the portable electronic device to locate the position and orientation of the portable electronic device in a common coordinate system.

20. The surgical robot system of claim 12, wherein, The position and orientation of the portable electronic device are transmitted to the virtual reality headset for presenting a virtual representation of the portable electronic device in the virtual reality environment of the remote user or a virtual representation of the local user.

Images