Robotic remote surgery portal system and method

Abstract

The robotic remote surgery portal system has a first robotic remote surgery portal at a first location, which is connectable to a surgical robot and is configured to receive surgical video and robot control feedback data from the surgical robot, synchronize the video and robot control feedback data, transmit surgical video and robot control feedback data, receive robot control information, and communicate robot control information to the surgical robot. The system also has a second robotic remote surgery portal at a second location, which is connectable to a surgical robot control console and is configured to receive surgical video and robot control feedback data from the first robotic remote surgery portal, communicate the received surgical video and robot control feedback data to the surgical robot control console, receive robot control data, and transmit the received robot control data to the first robotic remote surgery portal.

Description

Technical Field

[0001] The present disclosure relates to devices used in surgery, and more particularly, to a system for enabling robotic telesurgery.

Background Art

[0002] Robot-assisted surgery has enabled very significant surgical improvements. Typically, a surgical robot includes a camera and mechanical arms. Surgical tools are disposed within the mechanical arms and are controlled by a surgeon using a surgical robot control console. The surgical robot control console provides the surgeon with a view of the surgical site and enables the surgeon to control the mechanical arms very precisely. This has enabled minimally invasive surgery, resulting in surgeries with fewer complications and less scarring.

[0003] However, the surgical robot control console is typically placed very close to the operating table with the patient. Thus, robot-assisted surgery has mainly been limited to situations where the surgeon is located with the surgical robot and the patient. This limits the applicability of robot-assisted surgery to situations where a surgeon with the specific skills required for the surgery can be located with the patient.

[0004] Therefore, there is a need for a system and method that enables a surgeon to perform robot-assisted surgery from a location remote from the surgical robot and the patient. This would enable the most skilled surgeons for a particular surgical procedure to treat patients from a remote location.

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] This application relates to a robot telesurgery portal system that provides safe and effective visualization of surgical video, low-latency remote surgical robot control, feedback, and reliable routing of video signals. The robot telesurgery portal system also provides telepresence between the patient-side portal (PsP) and the surgeon-side portal (SsP). [Means for solving the problem]

[0006] In one implementation configuration, the robotic remote surgery portal system has a first robotic remote surgery portal at a first location, which is connectable to a surgical robot and is configured to receive surgical video and robot control feedback data from the surgical robot, synchronize the video and robot control data from the surgical robot, transmit surgical video and robot control feedback data, receive robot control information, and communicate the robot control information to the surgical robot. The robotic remote surgery portal system also has a second robotic remote surgery portal at a second location away from the first location, which is connectable to a surgical robot control console and is configured to receive surgical video and robot control feedback data from the first robotic remote surgery portal, communicate the received surgical video and robot control feedback data to the surgical robot control console, receive robot control data from the surgical robot control console, and transmit the received robot control data to the first robotic remote surgery portal.

[0007] The first and second robotic remote surgery portals are configured for secure and redundant communication in multiple low-latency modes. In one implementation, at least one of the low-latency mode communications is conducted over a 5G cellular communication network. Optionally, the first and second robotic remote surgery portals are configured to prevent the initiation of surgery until secure and redundant communication in multiple modes has been verified.

[0008] In one implementation, the first robotic remote surgery portal is further configured to synchronize surgical video and robotic control feedback data before transmitting them. The first robotic remote surgery portal may also be configured to encode surgical video and robotic control feedback data before transmitting them. The first robotic remote surgery portal may also be configured to variably encode the surgical video and robotic control feedback data according to the perceived communication bandwidth in order to ensure that sufficient surgical video and robotic control feedback data is communicated to the second robotic remote surgery portal.

[0009] In one implementation, each of the first and second robotic remote surgery portals further includes a microphone, a speaker, a camera, and a display, and each of the first and second robotic remote surgery portals is further configured to communicate telepresence data received from the microphone and camera to the speaker and display. A single robotic remote surgery portal may be configured as both the first and second robotic remote surgery portals.

[0010] These and other features are described below. [Brief explanation of the drawing]

[0011] The features, aspects, and advantages of the present invention will be better understood with reference to the following description, the appended claims, and the appended drawings:

[0012] [Figure 1] Figure 1 is a schematic diagram of a robotic remote surgery portal system placed at the patient site along with a surgical robot according to one embodiment;

[0013] [Figure 2] Figure 2 is a schematic diagram of a robotic remote surgery portal system located at the surgeon's site, away from the patient's site, using a surgical robot control console according to one embodiment.

[0014] [Figure 3] Figure 3 is a schematic diagram of a system having a robotic remote surgery portal system according to one embodiment. [Modes for carrying out the invention]

[0015] <Detailed explanation> The following description of preferred embodiments refers to the accompanying drawings illustrating specific embodiments in which the present invention may be carried out. In the drawings, the same reference numerals are used for the same or similar components whenever possible. It should be understood that other implementations may be used and structural and functional modifications may be made without departing from the scope of this disclosure.

[0016] Referring to Figures 1 and 2, various embodiments of robotic remote surgery portal systems are described below. In one embodiment, as shown in Figure 1, the first robotic remote surgery portal 10A is located in the patient's operating room together with the surgical robot 12. The first robotic remote surgery portal 10A may be referred to herein as the “Patient-Side Portal (PsP)”. As shown in Figure 2, the second robotic remote surgery portal 10B is located at the surgeon's site, away from the patient's site, together with the surgical robot control console 14. The second robotic remote surgery portal 10B may be referred to herein as the “Surgeon-Side Portal (SsP)”. In one implementation, the first robotic remote surgery portal 10A and the second robotic remote surgery portal 10B are connectable to the surgical robot and robotic remote surgery portal, respectively, and are substantially interchangeable. In additional implementations, the first robotic remote surgery portal 10A is configured for use with a specific surgical robot, and the second robotic remote surgery portal 10B is configured for use with a specific surgical robot control console.

[0017] Each robotic remote surgery portal 10A and 10B has a transceiver 16 for communicating with other robotic remote surgery portals. The transceiver 16 is configured for secure communication in multiple modes (or multiple modes). For example, but not limited to, the transceiver 16 may be configured for 5G cellular communication. Furthermore, the transceiver 16 may be configured for wired and wireless wideband wide area network (WAN) communication. In one implementation, the transceiver 16 is configured for wired communication over a private wide area network line to ensure low latency transmission. The transceiver is further configured for redundant communication so that the same data is transmitted in multiple modes. Cellular interfaces and redundant communication are particularly important in the context of robotic remote surgery to ensure the communication of critical information in emergencies if the wide area network connection fails. In one implementation, robotic remote surgery portals 10A and 10B include two transceivers 16 to increase bandwidth and in case of hardware failure.

[0018] The transceiver 16 is coupled to the image processing module (IPM) 18 and the processor 20. The IPM 18 may include a field-programmable gate array (FPGA). The IPM 18 is coupled to the processor 20. The IPM 18 is also coupled to the remote surgical video and control input / output module 22. In one implementation, the remote surgical video and control input / output module 22 can be connected to the surgical robot 12 or the surgical robot control console 14. As described below, the IPM 18 is responsible for video encoding and decoding.

[0019] In one implementation, the remote surgical video and control module 22 is multi-standard and can be connected to multiple different types of surgical robots 12. In an additional implementation, the video and control module 22 is configured to connect to a specific surgical robot 12. In another implementation, the video and control module 22 is multi-standard and can be connected to multiple different types of surgical robot control consoles 14. In an additional implementation, the video and control module 22 is configured to connect to a specific robot control console 14.

[0020] In the implementation configuration, the video and control module 22 receives and transmits video (1080p) via a 3G-Serial Digital Interface (3G-SDI), Digital Visual Interface (DVI), or High Definition Multimedia Interface (HDMI®). In the implementation configuration, the video and control module 22 receives ultra-high-definition video or 4K video via a High Definition Multimedia Interface (HDMI®), 12G-Serial Digital Interface (12G-SDI), or DisplayPort Interface. In the implementation configuration, the video and control module 22 receives and transmits robot control and feedback using a Controller Area Network (CAN) bus, Ethernet for Control Automation Technology (EtherCat), Universal Serial Bus (USB) 3.0, or Transmission Control Protocol / Internet Protocol (TCP / IP) interface.

[0021] Processor 20 can be, for example, the Nvidia Jetson Orin NX. A video signal can be passed from IPM18 to processor 20. Processor 20 can enhance the video signal and can be used to return the video signal to IPM18. For example, processor 20 can add overlays and prompts to the video. Although IPM18 and processor 20 are shown as separate units, they can be implemented as a single module.

[0022] Processor 20 is coupled to an audio subsystem 30 that includes at least one microphone 32 and a visualization subsystem 40 that includes at least one camera 42, such as for capturing an image of an operating room. Audio subsystem 30 and visualization subsystem 40 are used for telepresence, which, as will be further described below, enables a surgeon to feel connected to the patient's operating room and enables people in the patient's operating room to feel connected to the surgeon.

[0023] ? Audio subsystem 30 can include an array of microphones 32. In one implementation, audio subsystem 30 includes two microphones 32 configured for stereo. In one implementation, audio subsystem 30 includes five microphones 32. Microphones 32 can be placed at various points along the outer edges of the robotic telesurgery portals 10A, 10B. Microphones 32 can be configured to perform beam steer and cross noise cancel.

[0024] Furthermore, the audio subsystem 30 may include at least one speaker 34. Additionally, the audio subsystem 30 may include a digital signal processor (DSP) 36 for managing audio (or sound / audio) detected by at least one microphone 32 and playing the audio through at least one speaker 34. Although the digital signal processor 36 and the processor 20 are shown as separate units, they may be implemented in a single module. In one implementation, the audio subsystem 30 includes two speakers 34 and a subwoofer for typical Dolby 2.1 audio 34. The robot remote surgery portals 10A, 10B may also have indicator lights (or display lights) 38. The indicator lights 38 may display statuses such as, for example, the status of an Internet connection. The audio subsystem 30 is used in conjunction with the processor 20 to perform various functions such as, for example, but not limited to, voice control of the robot remote surgery portals 10A, 10B, audio prompts to the user, feedback to the user, alarm notifications, alarm detection, telemedicine, communication, and music playback.

[0025] The visualization subsystem 40 may have multiple cameras 42. In one implementation, the visualization subsystem 40 has a stereo camera 42 for assisting in room situation recognition or face recognition. In one implementation, at least one camera 44 is a time of flight camera. In one implementation, the visualization subsystem 40 has two cameras 42 for stereo imaging. In additional implementations, the visualization subsystem 40 has two cameras for three-dimensional image and video capture. The visualization subsystem 40 can be used in conjunction with the processor 20 to view the surroundings of the robot remote surgery portals 10A, 10B, such as in an operating room. Furthermore, the visualization subsystem 40 is used in conjunction with the processor 20 to perform various functions such as, for example, but not limited to, situation recognition, learning, training, telemedicine, and distance measurement.

[0026] The robotic remote surgery portals 10A and 10B may have a wireless network interface such as Wi-Fi or Bluetooth to connect to compatible external devices, including, for example, input devices such as mice and keyboards, display devices, and portable computing devices such as telephones, tablets, and laptops for exchanging information.

[0027] In addition, the robotic remote surgery portals 10A and 10B may have a secondary display interface for connecting to an additional display. The additional display may be a touchscreen, may enable annotation of images, recordings, or reports, or may be used for controlling the robotic remote surgery portals 10A and 10B. The secondary display interface may be, but is not limited to, an HDMI®, DisplayPort connector, or Serial Digital Interface (SDI) connector. Furthermore, the secondary display interface may include a Universal Serial Bus (USB) interface for touchscreen control.

[0028] The robotic remote surgery portals 10A and 10B have numerous features. The robotic remote surgery portals 10A and 10B may be configured to authenticate users using at least one form of biometric authentication. In one implementation, the authentication system utilizes speech recognition perceived by at least one microphone 32 and at least partial facial recognition perceived by at least one camera 42. Upon authentication, the user may be provided with access to hospital records and the ability to interact with other robotic remote surgery portals.

[0029] Referring to Figure 3, the robotic remote surgery portals 10A and 10B may be linked to the hospital network, databases, and external data processing. Once a user is authenticated, information specific to that user and the patient in the operating room may be automatically collected. For example, the robotic remote surgery portals 10A and 10B may be linked to a medical image management system (PACS) 60, either internal or external to the hospital, to retrieve or store patient information. Furthermore, the robotic remote surgery portals 10A and 10B may be linked to an electronic medical record (EMR) / electronic health record (HER) server 62, either internal or external to the hospital, to retrieve or store patient information. In addition, the robotic remote surgery portals 10A and 10B may be linked to a cloud-based storage system 63.

[0030] Furthermore, the robotic remote surgery portals 10A and 10B can be linked to external data processing systems and applications. The ability to link to external data storage and processing systems and applications enables deeper artificial intelligence and machine learning capabilities. For example, if a voice command requesting "turn the lights up to maximum" is received by the robotic remote surgery portals 10A and 10B, the command can be processed by an external application and the command can be sent back to the robotic remote surgery portal to turn the lights up to 100%. The robotic remote surgery portals 10A and 10B may also be linked to cloud or local services via applications (apps) residing on the robotic remote surgery portals. Based on events in the operating room, the robotic remote surgery portals 10A and 10B can learn locally or through machine learning cloud services and provide guidance to users and administrators about events in the operating room via artificial intelligence.

[0031] Referring to Figure 3, in one implementation, the robotic remote surgery portals 10A and 10B can be linked to machine learning and / or artificial intelligence applications and services 64. In addition, the robotic remote surgery portals 10A and 10B can be linked to telemedicine services 66. Furthermore, the robotic remote surgery portals 10A and 10B can be linked to external authentication services 68, as well as external transcription services 70 and external fleet management services 72. Furthermore, the robotic remote surgery portals 10A and 10B can be linked to external communication services 74 such as Voice over Internet Protocol (VOIP) and cellular communication.

[0032] Furthermore, the user can also control several different aspects of the operating room environment. This control may be performed using one or more of the following: voice commands, gestures embedded in a user input device, gestures sensed by at least a camera, and user input devices coupled to robotic remote surgery portals 10A and 10B. In one implementation, a touchscreen surgical display 24 is coupled to the processor 20 by a universal serial bus (USB) interface 28, for example. In another implementation, the surgical display 24 is also coupled to an IPM 18 to display a secure picture on the surgical display, regardless of whether the processor is operational.

[0033] Using at least one camera, the robotic remote surgery portals 10A and 10B may be able to identify and track each person in the operating room, as well as when each person enters and leaves. This may include the entry and exit of patients. The robotic remote surgery portals 10A and 10B may also record video and audio of surgical procedures as records of the procedures, for creating reports, and for training purposes.

[0034] Next, the operation of the robotic remote surgery portals 10A and 10B will be further described. Surgical video and robot control feedback data from the surgical robot 12 enter the patient-side robotic remote surgery portal 10A via the video and control module 22. In one implementation, the surgical video is encoded by the IPM 18 to reduce the required data bandwidth. In another implementation, the surgical video is encoded using an H.264 encoder. In a preferred embodiment, the surgical video is encoded using an H.265 encoder. In one implementation, the encoder and decoder are implemented within the IPM 18 to minimize latency.

[0035] The surgical video is synchronized with the received robot control feedback data by the IPM18 and / or processor 20. In one implementation, the video and control data are synchronized using one or more of the following: a) time stamping, b) synchronization words associated with packetized data before and after transmission, and c) a phase-locked loop at the receiving end.

[0036] Synchronized surgical video and robot control feedback data are transmitted by transceiver 16 over a secure, redundant, low-latency network. For example, synchronized surgical video and robot control feedback data may be transmitted over a 5G cellular data network and an Ethernet® connected wide-area network. Furthermore, the patient-side robotic remote surgery portal 10A device provides telepresence functionality by transmitting audio and video received from the audio subsystem 30 and video subsystem 40.

[0037] The surgeon-side robotic remote surgery portal 10B receives synchronized surgical video and robot control feedback data from the patient-side robotic remote surgery portal 10A via the transceiver 16. The received synchronized surgical video and robot control feedback data are routed and decoded through the IPM 18 and then output to the surgical robot control console 14 via the video and control module 22 for visualization and observation by the surgeon. Furthermore, telepresence information received from the patient-side robotic remote surgery portal 10A is communicated through the display 24 and audio subsystem 30 of the surgeon-side robotic remote surgery portal 10B.

[0038] Robot control signals generated by the surgeon operating the robot control console 14 are sent to the surgeon-side robotic remote surgery portal 10B via the video and control module 22. In addition, the surgeon-side robotic remote surgery portal 10B device provides telepresence functionality by transmitting audio and video received from the audio subsystem 30 and the video subsystem 40.

[0039] Robot control signals generated by the surgeon are encoded by the IPM 18 and transmitted to the patient-side robotic remote surgery portal 10A via a secure, redundant, low-latency network. The robot control signals generated by the surgeon are received within the transceiver 16 of the patient-side robotic remote surgery portal 10A. The received robot control signals are routed through the IPM 18, decoded, and then output to the surgical robot 12 for execution by the surgical robot via the video and control module 22. In addition, telepresence information received from the surgeon-side robotic remote surgery portal 10B is communicated through the display 24 and audio subsystem 30 of the patient-side robotic remote surgery portal 10A. The provision of telepresence information enables natural communication between the operating room where the patient and robot 12 are located and the location of the surgeon and robot control panel 14.

[0040] To ensure patient safety, the communication of surgical video, robot control feedback data, and robot control data is prioritized and guaranteed. For example, in one implementation, the robotic remote surgery portals 10A and 10B will not allow the start of surgery until the presence of two redundant low-latency communication networks is verified. In addition, in one implementation, if communication bandwidth is limited, the communication of telepresence information may be restricted to ensure the communication of surgical video, robot control feedback data, and robot control data. Furthermore, in one implementation, variable encoding / decoding quality (dynamic compression level) is used to vary the data rate and ensure the communication of essential surgical video, robot control feedback data, and robot control data as required by network conditions.

[0041] The above description and drawings disclose a robotic remote surgical portal system that completely and effectively overcomes the shortcomings associated with the prior art. However, it will be apparent that modifications and alterations of the disclosed implementations can be made without departing from the principles of the present invention. The implementations presented herein are provided only as examples and not as limitations, and the true scope and spirit of the invention are shown by the following claims.

Claims

1. A robotic remote surgery portal system: A first robotic remote surgery portal located at a first location, the first robotic remote surgery portal being connectable to a surgical robot, and: The surgical robot receives surgical video and robot control feedback data. The video and robot control feedback data from the surgical robot are synchronized, The surgical video and robot control feedback data are transmitted. The surgical robot receives robot control information, The robot control information is communicated to the surgical robot. The first robotic remote surgery portal is configured as follows: A second robotic remote surgery portal located at a second location, separate from the first location, wherein the second robotic remote surgery portal is connectable to a surgical robot control console, and: The first robotic remote surgery portal receives surgical video and robot control feedback data. The received surgical video and robot control feedback data are communicated to the surgical robot control console. The robot control data is received from the surgical robot control console. The received robot control data is transmitted to the first robot remote surgery portal. The second robotic remote surgery portal is configured as follows: A robotic remote surgery portal system, including...

2. The robotic remote surgery portal system according to claim 1, wherein the first robotic remote surgery portal and the second robotic remote surgery portal are configured for secure and redundant communication in multiple low-latency modes.

3. The robotic remote surgery portal system according to claim 2, wherein at least one of the low-latency mode communications is via a 5G cellular communication network.

4. The robotic remote surgery portal system according to claim 2, wherein the first robotic remote surgery portal and the second robotic remote surgery portal are configured to prevent the commencement of surgery until secure and redundant communication in multiple modes has been verified.

5. The robotic remote surgery portal system according to claim 1, further configured to synchronize surgical video and robot control feedback data before transmitting the surgical video and robot control feedback data.

6. The robotic remote surgery portal system according to claim 1, wherein the first robotic remote surgery portal is further configured to encode the surgical video before transmitting the surgical video and robot control feedback data.

7. The robotic remote surgery portal system according to claim 6, further configured to variably encode the surgical video according to the sensed communication bandwidth in order to ensure that sufficient surgical video and robot control feedback data are communicated to the second robotic remote surgery portal.

8. Each of the first robotic remote surgery portal and the second robotic remote surgery portal further, Microphone and, Speakers and, Camera and, The display and Includes, The robotic remote surgery portal system according to claim 1, wherein each of the first robotic remote surgery portal and the second robotic remote surgery portal is further configured to communicate telepresence data received from the microphone and camera to the speaker and display.

9. The robotic remote surgery portal system according to claim 1, wherein a single robotic remote surgery portal can be configured as both the first robotic remote surgery portal and the second robotic remote surgery portal.

Images