Systems and methods for mixed reality remote monitoring in minimally invasive surgery

Abstract

The present invention relates to a mixed reality remote supervision system (500) for use in surgical procedures, comprising a surgeon module (200), a WebRTC-based network architecture (400) for low-latency communication, and a supervision module (300). The system (500) enables real-time audiovisual communication and annotation between the surgeon (202) and the supervisor (302). The surgeon module (200) is adapted to process and transmit 2D and 3D stereoscopic images from a wide-range endoscopic camera (214). The network architecture (400) employs a signaling server (402) for handshaking, exchanging encoding formats, and bitrate information, creating a peer-to-peer encrypted communication link between the supervisor (302) and the surgeon (202). The supervision module (300) allows for adjustment of screen size and interpupillary distance, and allows the supervisor (302) to provide suggestions via voice and annotations in a virtual annotation panel (310). Multiple directors can connect to the same operating room, facilitating a method of live surgery streaming for educational purposes.

Description

[Technical Field]

[0001] The present disclosure relates generally to the field of robotic surgical systems for minimally invasive surgery, and more particularly, the present disclosure relates to a mixed reality remote monitoring module for use in a multi-arm robotic surgical environment in medical applications. [Background technology]

[0002] This section is intended to introduce the reader to various aspects of technology that may be related to various aspects of the present disclosure, as described below. This disclosure is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not merely as admissions of prior art.

[0003] Robotic-assisted surgery systems are being adopted worldwide to gradually replace traditional surgical procedures, such as open and laparoscopic procedures. Robotic-assisted surgery provides various benefits to patients during surgery and post-operative recovery. Robotic-assisted surgery also provides many advantages to surgeons in terms of improved ability to perform surgery accurately, reduced fatigue, and enhanced, clear, three-dimensional (3D) views of the surgical site. Furthermore, in robotic-assisted surgery, surgeons typically operate with hand controllers / master controllers / surgeon input devices / joysticks at a surgeon console system that seamlessly receives and communicates complex movements performed by the surgeon, giving the surgeon the perception that they are directly articulating surgical tools / instruments to perform the surgery. A surgeon operating on a surgeon console system may be located remotely from the surgical site or inside the operating room where the patient is being operated on. A robotic-assisted surgery system may include multiple robotic arms to assist in the performance of robotic-assisted surgery.

[0004] The robotic-assisted surgery system utilizes a sterile adapter / barrier to separate non-sterile portions of the multiple robotic arms from the essential sterile surgical tools / instruments attached to one end of the multiple robotic arms. The sterile adapter / barrier may include a sterile plastic drape that encases the multiple robotic arms and a sterile adapter / barrier that operably engages the sterile surgical tools / instruments in a sterile field.

[0005] One of the main challenges is the lack of real-time guidance and supervision from an experienced supervisor to the medical professional / surgeon while performing the robotic surgical procedure. Another challenge is that existing methods for remote supervision have limitations such as higher latency, complexity, and lack of immersive interaction. Furthermore, another challenge is that the supervisor often does not receive a real-time 3D reconstruction of the endoscopic view of the surgical site, leading to incorrect decisions.

[0006] In light of the above challenges, there is a need to provide an improved method of remote supervision for surgeons in a multi-arm robotic surgical system that solves the above-mentioned problems associated with robotic-assisted surgery. Summary of the Invention

[0007] It is proposed that some or all of the above-mentioned problems associated with providing training to surgeons and OT staff are addressed by certain embodiments of the present disclosure.

[0008] According to an aspect of the present invention, a mixed reality remote supervision system is disclosed in a multi-arm robotic surgery system comprising an operating table around which one or more robotic arms are arranged and a surgeon console, the mixed reality remote supervision system comprising: a surgeon module for use by the surgeon, the surgeon module comprising: a processor coupled to an encryption device, the processor configured to receive a video stream from an endoscopic camera; the encryption device configured to encrypt the received video stream; an input device; an output device; and a two-dimensional (2D) touchscreen monitor coupled to the processor, the 2D touchscreen monitor configured to be used as a graphical user interface for capturing input from the surgeon; a supervision module for use by a director, the head-mounted device including a processor operably coupled to a webRTC architecture, a 3D stereoscopic display, and a speaker, the head-mounted device including a processor configured to process the endoscopic feed received via the webRTC architecture and render it as a stereoscopic 3D image for the left and right eyes of the 3D stereoscopic display; and a virtual annotation panel for adding annotations by the director; and a webRTC architecture configured to exchange signals between the surgeon module and the supervision module.

[0009] According to an embodiment of the present invention, the input device may be any of a microphone or a web-enabled input device for inputting a 3D video stream, such as a camera.

[0010] According to another embodiment of the present invention, the virtual annotation panel is configured to capture the hand gestures of the director.

[0011] According to yet another embodiment of the present invention, the director can provide annotations and voice instructions to the surgeon.

[0012] According to yet another embodiment of the present invention, the director can control screen size adjustments and interpupillary distance (IPD) adjustments, providing custom control and adjustments.

[0013] According to yet another embodiment of the present invention, the supervision module supports ultra-low latency audio communication and advanced mixed reality headsets for annotation and synchronization with the console used by the surgeon.

[0014] According to yet another embodiment of the present invention, the director can select to connect to the correct operating room from a list of all live cases under his / her supervision.

[0015] According to yet another embodiment of the present invention, the supervision module allows for adjustment of screen size and interpupillary distance.

[0016] According to yet another embodiment of the present invention, the webRTC architecture creates a peer-to-peer connection between the surgeon module and the supervisor module.

[0017] According to yet another embodiment of the present invention, creating a peer-to-peer connection between the surgeon module and the supervisor module includes collecting network addresses of the surgeon module and the supervisor module via a STUN (Session Traversal Utility for NAT) server, exchanging the collected addresses, and establishing a peer-to-peer encrypted direct connection between the surgeon module and the supervisor module.

[0018] According to yet another embodiment of the present invention, the webRTC architecture utilizes a signaling server to establish a direct connection between the surgeon and the director.

[0019] According to yet another embodiment of the present invention, multiple directors can connect to a feed from the operating room received via the webRTC architecture.

[0020] According to yet another embodiment of the present invention, a 360° camera can be installed in the operating room to provide a real-time view of the operating room.

[0021] According to yet another embodiment of the present invention, a director can monitor a real-time operating room view.

[0022] Other embodiments, systems, methods, apparatus aspects, and features of the present invention will become apparent to those skilled in the art from the following detailed description, the accompanying drawings, and the appended claims.

[0023] The foregoing summary, as well as the following detailed description of the present disclosure, will be better understood when read in conjunction with the accompanying drawings. For the purpose of illustrating the present disclosure, there are shown in the drawings exemplary configurations of the present disclosure. However, the disclosure is not limited to the particular methods and instrumentalities disclosed herein. Moreover, those skilled in the art will appreciate that the drawings are not to scale. Wherever possible, like elements will be designated by like numerals. Embodiments of the present disclosure will now be described, by way of example only, with reference to the following figures: [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 illustrates an exemplary implementation of a multi-arm teleoperated surgical system that may be used with one or more features according to an embodiment of the present disclosure. [Figure 2] FIG. 2 illustrates a surgeon's side module according to an embodiment of the present disclosure. [Figure 3] FIG. 3 illustrates a supervisory module according to an embodiment of the present disclosure. [Figure 4] FIG. 4 illustrates a webRTC architecture according to an embodiment of the present disclosure. [Figure 5] FIG. 5 illustrates a remote supervision module according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0025] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same. Nonetheless, it will be understood that no limitation of the scope of the disclosure is thereby intended, and that such changes and further modifications in the illustrated systems, and such further applications of the principles of the present disclosure as set forth therein, as would normally occur to one skilled in the art to which the present disclosure pertains, are contemplated.

[0026] It will be understood by those skilled in the art that the foregoing general description and the following detailed description are exemplary and explanatory of the present disclosure and are not intended to be limiting of the present disclosure. A common convention throughout this patent specification is that like reference numerals refer to like elements in the accompanying drawings.

[0027] Reference throughout this specification to "an embodiment," "another embodiment," "an implementation," "another implementation," or similar language means that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the present disclosure. Thus, throughout this specification, the appearances of "in an embodiment," "in another embodiment," "in one embodiment," "in another embodiment," and similar language may, but do not necessarily, all refer to the same embodiment.

[0028] The terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process or method comprising a list of steps does not include only those steps, but may also include other steps not expressly listed or inherent in such process or method. Similarly, the term "comprises...a" preceding one or more devices or subsystems or elements or structures does not, without further constraints, exclude the presence of other devices or subsystems or elements or structures or additional devices or subsystems or elements or structures.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The devices, systems, and examples provided herein are illustrative only and are not intended to be limiting.

[0030] The use of the terms "a" and "an" herein does not denote a limitation of quantity, but rather denote the presence of at least one of the referenced item. Additionally, the terms sterile barrier and sterile adapter are synonymous and may be used interchangeably throughout the description.

[0031] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0032] Figure 1 illustrates an exemplary implementation of a multi-arm teleoperated surgical system that may be used with one or more features according to embodiments of the present disclosure. Specifically, Figure 1 illustrates a multi-arm teleoperated surgical system 100 having five robotic arms 102a, 102b, 102c, 102d, 102d, and 102e mounted on five robotic arm carts around a surgical table 104. The five robotic arms 102a, 102b, 102c, 102d, and 102e depicted in Figure 1 are for illustrative purposes only; the number of robotic arms may vary depending on the type of surgery. The five exemplary robotic arms (102a), (102b), (102c), (102d), and (102e) are positioned along the operating table (104) and may be positioned in different ways, including but not limited to, robotic arms (102a), (102b), (102c), (102d), and (102e) positioned along the operating table (104). The robotic arms 102a, 102b, 102c, 102d, and 102e may be separately mounted on five robotic arm carts, or the robotic arms 102a, 102b, 102c, 102d, and 102e may be mechanically and / or operatively connected to one another, or the robotic arms 102a, 102b, 102c, 102d, and 102e may be connected to a central body (not shown) such that the robotic arms 102a, 102b, 102c, 102d, and 102e branch off from the central body (not shown). Additionally, the multi-arm teleoperated surgical system 100 may include a surgeon console 106, a vision cart 108, and a table for surgical instruments and accessories.

[0033] During surgical procedures, especially in environments such as operating rooms (OTs), it is essential to provide medical professionals, especially surgeons, with real-time guidance and supervision from an experienced supervisor. Traditional methods for remote supervision have limitations, including latency, complexity, and a lack of immersive interaction; the supervisor often does not receive a 3D reconstruction of the endoscopic view, which can lead to incorrect decisions.

[0034] Remote monitoring and supervision have the potential to transform the medical field, but it is essential to recognize the complexities involved. Expertise in telecommunications, data security, technological adaptability, and network engineering is crucial to overcoming challenges and ensuring the successful operation of such systems. As the demand for remote surgical guidance grows, so does the need for experts who can navigate the complexities of this innovative technology. Their contributions are critical to advancing patient care and medical practice.

[0035] A mixed reality remote supervision module for use in surgical procedures includes a surgeon-side module, a network architecture based on WebRTC for low-latency communication, and a supervision-side mixed reality headset, where the system enables real-time audiovisual communication and annotation between the surgeon and the supervision.

[0036] FIG. 2 illustrates a surgeon module according to an embodiment of the present disclosure. The surgeon module (200) serves as an interface for a surgeon (202) in an operating room. The surgeon module (200) includes a processor (204), an input device (206), output devices (208, 210), an encryptor (212), and an endoscopic camera (214). The input device (206) may be either a microphone (206) or a web-enabled input device for inputting a 3D video stream, such as a camera (214). The surgeon module (200) can receive endoscopic signals from the endoscopic camera (214), which offers unique advantages, including a 3D stereoscopic display (216) for a mono endoscopic feed and a 2D display (218) for 360° operating room capture. The endoscopic camera (214) continuously transmits a video stream to the processor (204). The output device may be a speaker 210, a 2D display 218, or the like. The processor 204 includes an encryptor 212 configured to encrypt the received video stream and transmit the encrypted video to the 2D display 218. The surgeon module 200 is adapted to process and transmit 2D and 3D stereoscopic images from a wide range endoscopic camera 214.

[0037] FIG. 3 illustrates a supervision module according to an embodiment of the present disclosure. The supervision module (300) is designed for use by a director (302). The supervision module (300) includes a head-mounted device (304) including a processor (306), a 3D stereoscopic display (308), and a virtual annotation panel (310) for capturing the hand gestures of the director (302). The supervision module (300) processes the received endoscopic feed and renders it as a stereoscopic 3D image for the left and right eyes on the 3D stereoscopic display (308). The head-mounted device (304) is a headset that generates stereoscopic depth vision for the director (302). The director (302) wears the head-mounted device (304) and views the 3D video stream on the 3D stereoscopic display (308) of the head-mounted device (304). The director (302) can add annotations simply by writing something virtually using hand gestures. Annotations and voice commands from an experienced surgeon / director can be provided to the surgeon (202) performing surgery in the operating room for better surgical outcomes. The director (302) can control screen size and interpupillary distance (IPD) adjustments to provide custom control and adjustments. The director module (302) supports ultra-low latency audio communication and advanced mixed reality headsets (304) for annotation and synchronization with the console used by the surgeon (202). The director (302) can select to connect to the correct operating room from a list of all live cases at that time. The director module (300) allows adjustment of screen size and interpupillary distance. Additionally, the director module (300) allows the director (302) to provide suggestions via voice using the speaker (312) and annotations.

[0038] Figure 4 shows the webRTC architecture (400) for the exchange of signals between the surgeon module (200) and the supervisor module (300). Creating a peer-to-peer connection in WebRTC involves several steps, including collecting network addresses via a Session Traversal Utility for NAT (STUN) server, signaling to exchange these addresses, and finally establishing a direct connection between the peers.

[0039] An overview of this process is as follows: 1. Peer Initialization: a. Both the sender (Peer A) and receiver (Peer B) initialize the WebRTC application and set up the components required for real-time communication. 2. Network Address Collection (STUN): a. Peer A and Peer B use a STUN server to collect network addresses. STUN helps discover the public IP address and port where each peer can be reached on the Internet. b. For example, Peer A sends a request to the STUN server, which responds with the public IP address and port number Peer A is using to communicate with the STUN server. Similarly, Peer B does the same. 3. Signaling: a. Peer A and Peer B exchange information about network addresses using a signaling server. Signaling does not transmit media, but is responsible for coordinating negotiations between peers. b. Peer A generates an "offer" message containing its network address information (collected in step 2) and the types of media and codecs it can handle. This offer is sent to the signaling server. c. The signaling server forwards Peer A's offer to Peer B. d. Peer B receives the offer from the signaling server. It also generates an "Answer" message in response. The answer includes Peer B's network address information and its preferred media settings. e. The answer is sent back to the signaling server, which then forwards it to Peer A.

[0040] 4. ICE (Interactive Connectivity Establishment): a. Before establishing a direct connection, both Peer A and Peer B use the ICE framework to determine the best possible network path to reach each other. b. ICE assembles a list of network candidates, which may include public IP addresses and ports, local IP addresses, and relay candidates (if a direct P2P connection is not possible due to NAT or firewall restrictions). c. Local candidates are used for communication within the same local network, and public candidates are used for communication over the Internet. 5. Direct Peer-to-Peer Connection: a. Using the network address information obtained through STUN and the negotiated settings from the signaling process, both Peer A and Peer B attempt to connect directly to each other. b. They use the network addresses (public IP and port) provided during the ICE process to initiate the direct connection. This connection can traverse NATs and firewalls because ICE selects the most appropriate network path. c. Once a direct connection is successfully established, Peer A and Peer B can exchange media and data directly without the need for a relay server. The combination of STUN for network address lookup, signaling for offer / answer exchange, and ICE for network route discovery allows WebRTC to set up secure and efficient peer-to-peer connections, enabling real-time communication between users over the Internet.

[0041] Figure 5 illustrates a system for a mixed reality remote supervision module in minimally invasive surgery, according to an embodiment of the present disclosure. The mixed reality remote supervision module (500) serves as an interface for the surgeon (202) in the operating room. The mixed reality remote supervision module (500) can receive endoscopic signals, including 3D stereoscopic images and 2D mono endoscopic feeds, along with a 360° capture of the operating room, which is a unique advantage. The mixed reality remote supervision module (500) connects to a signaling server (402) and streams video feeds. It manages the transmission of two-way audio and the reception of annotations, enabling clear communication between the surgeon (202) and the supervision (302).

[0042] The present invention utilizes a WebRTC-based, ultra-low latency peer-to-peer architecture (400), as shown in Figure 4, in which the role of a central server (402) is minimized, reducing latency and network overhead. A server (402) acting as a signaling server (e.g., Amazon Azure Virtual Desktop) facilitates the handshake between the surgeon (202) and the director (302). The server (402) exchanges important information between the two nodes (404, 406) for communication, such as IP addresses, encoder format, and bit rate. This information is called the ICE candidate. A STUN (408) / TURN (410) server takes over after the initial handshake and handles seamless peer-to-peer communication. The architecture (400) employs the signaling server (402) for the handshake, exchange of encoding format, and bit rate information, creating a peer-to-peer encrypted communication link between the director (302) and the surgeon (202). Multiple directors can connect to the same operating room, facilitating live surgical streaming for educational purposes.

[0043] The present disclosure has the following advantages: The surgeon module supports a very wide range of endoscopic devices and cameras. It allows the use of endoscopic views in mono and stereo, and supports line-by-line and side-by-side stereo formats. The use of an ultra-low latency architecture (WebRTC) 400 ensures real-time communication and feedback. The director 302 can provide guidance using a mixed reality headset, improving the quality of support. The director 302 can annotate findings and communicate via voice. The director 302 can also monitor the real-time operating room view using a 360 camera feed. The director 302 can view the 3D endoscopic feed on a monitor instead of a headset. There can be multiple directors or viewers who can subscribe to the feed from the operating room, allowing multiple directors to review at once. The system enables real-time audiovisual communication and annotation between the surgeon and the director.

[0044] The foregoing description of exemplary embodiments of the present disclosure has been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the disclosure to the precise forms disclosed, as obviously many modifications and variations are possible in light of the above teachings. The exemplary embodiments were chosen and described to best explain the principles of the present disclosure and its practical application, thereby enabling others skilled in the art to best utilize the present disclosure and various embodiments with various modifications suited to the particular uses contemplated. It is understood that various omissions and substitutions of equivalents are contemplated as circumstances may suggest or render expedient, but are intended to cover applications or implementations without departing from the spirit or scope of the claims of the present disclosure.

[0045] Benefits, other advantages, and solutions to problems have been described above with respect to particular embodiments. However, the benefits, advantages, solutions to problems, and any elements that may cause or make more pronounced any benefit, advantage, or solution should not be construed as critical, necessary, or essential features or elements of any or all of the claims.

[0046] Although specific language has been used to describe this disclosure, no limitations are intended to arise therefrom. As will be apparent to those skilled in the art, various functional modifications can be made to the device to implement the inventive concepts taught herein.

Claims

1. A mixed reality remote supervision system (500) for a multi-arm robotic surgery system (100) comprising an operating table (104) around which one or more robotic arms (102a, 102b, 102c, 102d, 102e) are arranged, and a surgeon console (106), comprising: A surgeon module (200) for use by a surgeon (202), comprising: a processor (204) coupled to an encryptor (212), the processor (204) configured to receive a video stream from an endoscopic camera (214), the encryptor (212) configured to encrypt the received video stream; an input device (206, 214); an output device (208, 210); a two-dimensional (2D) touchscreen monitor (218) coupled to the processor (204), the 2D touchscreen monitor (218) configured for use as a graphical user interface for capturing input from the surgeon (202); a surgeon module (200) comprising: A supervisory module (300) for use by a supervisor (302), comprising: a head-mounted device (304) including a processor (306) operably coupled to a webRTC architecture (400), a 3D stereoscopic display (308), and a speaker (312), the processor (306) configured to process an endoscopic feed received via the webRTC architecture and render it as a stereoscopic 3D image for a left eye and a right eye of the 3D stereoscopic display (308); a virtual annotation panel (310) for adding annotations by the director (302); a supervisory module (300) comprising: the webRTC architecture (400) configured to exchange signals between the surgeon module (200) and the supervisor module (300); A mixed reality remote supervision system (500) comprising:

2. 2. The mixed reality remote supervision system of claim 1, wherein the input device may be either a microphone or a web-enabled input device, such as a camera, for inputting a 3D video stream.

3. The mixed reality remote supervision system of claim 1 , wherein the virtual annotation panel is configured to capture hand gestures of the director.

4. The mixed reality remote supervision system (500) of claim 1, wherein the supervision (302) is capable of providing annotations and voice instructions to the surgeon (202).

5. 10. The mixed reality remote supervision system of claim 1, wherein the supervision can control screen size and interpupillary distance (IPD) adjustments to provide custom control and adjustments.

6. 10. The mixed reality remote supervision system of claim 1, wherein the supervision module supports ultra-low latency audio communication and advanced mixed reality headsets for annotation and synchronization with the console used by the surgeon.

7. 10. The mixed reality remote supervision system of claim 1, wherein the supervising director can select to connect to the correct operating room from a list of all live cases being supervised.

8. The mixed reality remote supervision system (500) of claim 1 , wherein the supervision module (300) allows for adjustment of screen size and interpupillary distance.

9. The mixed reality remote supervision system (500) of claim 1, wherein the webRTC architecture (400) creates a peer-to-peer connection between the surgeon module (200) and the supervision module (300).

10. The creation of the peer-to-peer connection between the surgeon module (200) and the supervisor module (300) comprises: collecting the network addresses of the surgeon module (200) and the supervisor module (300) via a Session Traversal Utilities for NAT (STUN) server; exchanging the collected addresses; and establishing a peer-to-peer encrypted direct connection between the surgeon module (200) and the supervisor module (300); The mixed reality remote supervision system (500) of claim 9, comprising:

11. 10. The mixed reality remote supervision system of claim 9, wherein the webRTC architecture utilizes a signaling server to facilitate a handshake between the surgeon and the director.

12. 10. The mixed reality remote supervision system of claim 1, wherein multiple supervisors can connect to the feed from an operating room received via the webRTC architecture.

13. 10. The mixed reality remote supervision system (500) of claim 1, wherein a 360° camera can be installed in the operating room to provide a real-time view of the operating room.

14. 14. The mixed reality remote supervision system (500) of claim 13, wherein the supervising person (302) can monitor the real-time operating room view.

Images