Systems and methods for provisioning remote operation of medical equipment

Abstract

Disclosed herein are systems and method for provisioning remote operation of medical equipment. A keyboard, video, and mouse (KVM) component may be configured to capture a video output comprising medical data as displayed on a first computing device connected to the medical equipment, transmit the captured video output to a second computing device remotely connected to the KVM component, receive, from the second computing device, a selection of a virtual button of the one or more virtual buttons, and transmit, to medical interfacing equipment, a command to execute a function associated with a button, of the one or more buttons, that corresponds to the virtual button.

Description

SYSTEMS AND METHODS FOR PROVISIONING REMOTE OPERATION OF MEDICAL EQUIPMENTFIELD OF TECHNOLOGY[1] The present disclosure relates to the field of biomedical technology, and, more specifically, to systems and methods for provisioning remote operation of medical equipment.BACKGROUND[2] Due to advancements in communication technology, the medical field has experienced improved accessibility. Medical professionals performing checkups no longer need to be physically present with their patients. For example, patients are now able to video call their doctors during an appointment instead of going into a hospital or clinic. Likewise, medical specialists that are unable to be physically present in a hospital can remotely monitor operations and provide guidance to technicians, nurses, and other doctors present at the hospital.[3] However, conventional remote assistance fails to meet the requirements of performing complex remote tasks such as taking control of different types of medical equipment from a remote setting and executing commands in near real-time. As medical equipment continues to advance exponentially in sophistication and functionality, it has becoming increasingly difficult to perform remote tasks, which further exposes the weaknesses of conventional remote assistance.SUMMARY[4] To overcome the shortcomings of conventional remote assistance technology in the medical field, the present disclosure describes systems and methods that enable the remote operation of medical equipment (e.g., magnetic resonance imaging (MRI) & computerized tomography (CT) equipment) through a platform that combines robotics,hardware, and software in a non-invasive solution. Through the platform, users are also able to stream medical sessions (e.g., MRI, CT, PET-CT, etc), which improves collaboration, consulting, support, professional training, and quality control.[5] In one exemplary aspect, the techniques described herein relate to a system for provisioning remote operation of medical equipment, including: medical equipment that produces medical data; medical interfacing equipment including one or more buttons each controlling a respective function of the medical equipment; and a keyboard, video, and mouse (KVM) component configured to: capture a video output including the medical data as displayed on a first computing device connected to the medical equipment; transmit the captured video output to a second computing device remotely connected to the KVM component, wherein the second computing device outputs a user interface that includes the captured video output and one or more virtual buttons corresponding to the one or more buttons on the medical interfacing equipment; receive, from the second computing device, a selection of a virtual button of the one or more virtual buttons; and transmit, to the medical interfacing equipment, a command to execute a function associated with a button, of the one or more buttons, that corresponds to the virtual button.[6] In some aspects, the techniques described herein relate to a system, wherein the KVM component is a software application installed on a third computing device connected to the first computing device.[7] In some aspects, the techniques described herein relate to a system, wherein the KVM component is a standalone device connected to the first computing device.[8] In some aspects, the techniques described herein relate to a system, wherein the KVM component includes an embedded video splitter.[9] In some aspects, the techniques described herein relate to a system, wherein the KVM component connects to the medical equipment using a USB cable.

[0010] In some aspects, the techniques described herein relate to a system, wherein the user interface further enables interaction with the medical data using one or more options selectable from the captured video output, wherein the KVM component is further configured to: receive, from the second computing device, a selection of the one or more options; and transmit, to the first computing device, the selection of the one or more options for execution.[111 In some aspects, the techniques described herein relate to a system, wherein the KVM component is further configured to: capture user inputs received at the first computing device; and transmit the captured user inputs to the second computing device.

[0012] In some aspects, the techniques described herein relate to a system, wherein the KVM component is configured to use Web Real-Time Communication (WebRTC) over a connection with the second computing device.

[0013] In some aspects, the techniques described herein relate to a system, wherein prior to transmitting the command, an interface controller device is configured to determine that the virtual button in the selection corresponds to the button on the medical interfacing equipment by: searching a layout database that maps different virtual buttons to physical buttons on a plurality of medical interfacing equipment.

[0014] In some aspects, the techniques described herein relate to a method for provisioning remote operation of medical equipment, including: capturing, by a keyboard, video, and mouse (KVM) component, a video output including medical data as displayed on a first computing device connected to the medical equipment, wherein the medical data is produced by the medical equipment; transmitting, by the KVM component, the captured video output to a second computing device remotely connected to the KVM component, wherein the second computing device outputs a user interface that includes the captured video output and one or more virtual buttons corresponding to one or more buttons on medical interfacing equipment, wherein the one or more buttons each control a respective function of the medical equipment; receiving, from the second computing device, a selection of a virtual button of the one or more virtual buttons; and transmitting, to the medical interfacing equipment, a command to execute a function corresponding to a button, of the one or more buttons, that corresponds to the virtual button.

[0015] In some aspects, the techniques described herein relate to a method, wherein the KVM component is a software application installed on a third computing device connected to the first computing device.

[0016] In some aspects, the techniques described herein relate to a method, wherein the KVM component is a standalone device connected to the first computing device.

[0017] In some aspects, the techniques described herein relate to a method, wherein the KVM component includes an embedded video splitter.

[0018] In some aspects, the techniques described herein relate to a method, wherein the KVM component connects to the medical equipment using a USB cable.

[0019] In some aspects, the techniques described herein relate to a method, wherein the user interface further enables interaction with the medical data using one or more options selectable from the captured video output, wherein the KVM component is further configured to: receive, from the second computing device, a selection of the one or more options; and transmit, to the first computing device, the selection of the one or more options for execution.

[0020] In some aspects, the techniques described herein relate to a method, wherein the KVM component is further configured to: capture user inputs received at the first computing device; and transmit the captured user inputs to the second computing device.

[0021] In some aspects, the techniques described herein relate to a method, wherein the KVM component is configured to use Web Real-Time Communication (WebRTC) over a connection with the second computing device.

[0022] In some aspects, the techniques described herein relate to a method, wherein prior to transmitting the command, an interface controller device is configured to determine that the virtual button in the selection corresponds to the button on the medical interfacing equipment by: searching a layout database that maps different virtual buttons to physical buttons on a plurality of medical interfacing equipment models that it can be configured.

[0023] It should be noted that the methods described above may be implemented in a system comprising a hardware processor. Alternatively, the methods may be implemented using computer executable instructions of a non-transitory computer readable medium.

[0024] The above simplified summary of example aspects serves to provide a basic understanding of the present disclosure. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects of the present disclosure. Its sole purpose is to present one or more aspects in a simplified form as a prelude to the more detailed description of the disclosure that follows. To the accomplishment of the foregoing, the one or more aspects of the present disclosure include the features described and exemplarily pointed out in the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate one or more example aspects of the present disclosure and, together with the detailed description, serve to explain their principles and implementations.

[0026] FIG. 1 is a diagram illustrating a remote operation system.

[0027] FIG. 2 is a diagram illustrating network connections in the remote operation system using a software-based keyboard, video, and mouse (KVM) component.

[0028] FIG. 3 is a diagram illustrating an exemplary user interface for accessing medical equipment remotely.

[0029] FIG. 4 is a diagram illustrating physical connections in the remote operation system using a software-based KVM component.

[0030] FIG. 5 is a diagram illustrating security features of the remote operation system.

[0031] FIG. 6 is a diagram illustrating a plurality of channels used to exchange data in the remote operation system.

[0032] FIG. 7 is a diagram illustrating physical connections in the remote operation system using a hardware KVM device.

[0033] FIG. 8 illustrates a flow diagram of a method for provisioning remote operation of medical equipment.

[0034] FIG. 9 presents an example of a general-purpose computer system on which aspects of the present disclosure can be implemented.DETAILED DESCRIPTION

[0035] Exemplary aspects are described herein in the context of a system, method, and computer program product for provisioning remote operation of medical equipment. Those of ordinary skill in the art will realize that the following description is illustrative only and is not intended to be in any way limiting. Other aspects will readily suggest themselves to those skilled in the art having the benefit of this disclosure. Reference will now be made in detail to implementations of the example aspects as illustrated in the accompanying drawings. The same reference indicators will be used to the extent possible throughout the drawings and the following description to refer to the same or like items.

[0036] FIG. 1 is a diagram illustrating a remote operation system 100. Remote operation system 100 includes medical equipment 102, medical interfacing equipment 104, computing device 106 (e.g., the medical device), camera 112, and interface controller device 114. Medical interfacing equipment 104 may be a controller for adjusting parameters of medical equipment 102. Computing device 106 may be any computer device capable of receiving and displaying results from medical equipment 102. Camera 112 may be used to monitor the medical environment (e.g., a clinic, hospital room, laboratory room, exam room, workstation, etc.).

[0037] In a typical scenario, user 108 is physically present in the environment and uses medical interfacing equipment 104 to control medical equipment 102 and view results on a monitor of computing device 106. For example, computing device 106 may be a desktop computer, medical equipment 102 may be an MRI machine, and medical interfacing equipment 104 may be a keyboard that controls scanning, adjusts movements of the components in the MRI machine, and enables communication between user 108 and the patient in the MRI machine.

[0038] Remote operation system 100 enables remote user 110 to remotely control medical interfacing equipment 104 using interface controller device 114 and remotely view outputs of medical equipment 102 displayed on computing device 106. Interface controller device 114 receives a remote command from remote user 110, parses the remote command into a physical action (e.g., pressing a button, turning on a switch / toggle, rotating a dial, etc.) to perform on medical interfacing equipment 104. As a result, even though there is no direct connection between the device of remote user 110 and medical interfacing equipment 104, remote user 110 can control medical equipment 102 as though user 108 is physically pressing the physical inputs of medical interfacing equipment 104.

[0039] In an exemplary aspect, the remote operation system 100 utilizes a Keyboard, Video, and Mouse (KVM) component that enables the mentioned remote viewing and control. In one aspect, the KVM component is software-based and is installed on an independent computing device that employs a video capture card and USB cable connected to computing device 106. The video capture card is responsible for capturing the video from the computing device 106 through a video cable. The USB cable is responsible for transmitting mouse movements and keyboard inputs to the computing device 106. In another aspect, theKVM component is hardware-based and is connected to the computing device 106 as an independent KVM device. Both aspects are thoroughly described in this present disclosure.

[0040] FIG. 2 is a diagram 200 illustrating network connections in the remote operation system 100 using KVM component 212 that can be hardware-based or software-based. Diagram 200 depicts two networks: customer network 202 and public network 204. In some aspects, customer network 202 is a local area network (LAN) and public network 204 is a wide area network (WAN) (e.g., the Internet). In other aspects, both customer network 202 and public network 204 are WANs.

[0041] Customer network 202 features remote communication between devices in site A (e.g., a medical facility) and site B (e.g., a command center). Remote computing device 216 establishes a secure connection with KVM component 212 connected to computing device 206, the secure connection including one or more of virtual private network (VPN) capabilities, multi-protocol label switching (MPLS) capabilities, and LAN to LAN connection capabilities. In some aspects, remote computing device 216 may communicate with camera 208, interface controller device 228, and / or computing device 206 using any combination of Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Internet Control Message Protocol (ICMP).

[0042] Suppose that the display output of medical equipment 214 (e.g., an MRI machine) is presented on computing device 206 (e.g., a desktop computer). For example, MRI images may be displayed on computing device 206, which can be viewed, modified, and interacted with using peripheral devices (e.g., a mouse and keyboard). Using the secure connection, remote computing device 216 is able to receive and replicate the display output of medical equipment 214, as well as transmit commands to control the mouse and keyboard of computing device 206 and / or interface controller device 228 (to control medical interfacing equipment 210). This ultimately allows a remote user to program and perform a medical task (e.g., an examination process using medical equipment 214) directly as if he / she were actually sitting at the clinic or hospital workstation.

[0043] In some aspects, a remote user may also be assisted by one or more of cameras 208 that generate real-time images of the patient's position. For example, camera 208 may capture a video of the workstation in site A and transmit a stream overthe secure connection to remote computing device 216.

[0044] In some aspects, the remote user of remote computing device 216 may communicate with a local user of computing device 207 (e.g., a device dedicated to local support) using Voice over Internet Protocol (VoIP). This is enabled using communication platform 222, which receives and transmits audio information between the respective computing devices. In an exemplary aspect, communication platform includes VoIP server(s) 226 that perform communication exchanges.

[0045] In some aspects, cloud computation platform 220 may include a collection of resources and services provided over the Internet to enable users to access, store, manage, and run remote medical software (e.g., nCommand and / or nVirtual software). These platforms are hosted by service providers in their data centers, which includes application server(s) 224, may be used by remote computing device 216 to perform various computational tasks. For example, the software (e.g., nCommand) utilized by remote computing device 216 to perform remote access and control of device(s) in site A may be an application featuring a thin client and thick client. The thin client application may be locally installed on remote computing device 216 to perform non-intensive processing tasks such as displaying information. The thick client application may be installed on application server(s) 224 and may perform intensive processing tasks such as translating keyboard inputs received from remote computing device 216 to inputs for medical interfacing equipment 210 based on layouts. Application server(s) 224 may additionally store device configuration information, user registrations, and other information. It should be noted that no patient information or exam images are stored on public network 204. In some aspects, the software may be a web application.

[0046] In some aspects, both remote computing device 216 and KVM component 212 may communicate with servers in cloud computation platform 220 and communication platform 222 using TCP and Hypertext Transfer Protocol Secure (HTTPS) (with Secure Sockets Layer(SSL) / Transport Layer Security (TLS) 1.2) (Firewall or Proxy).

[0047] In some aspects, both computing device 207 and remote computing device 216 may contact technical support 218 to report, troubleshoot, and resolve remote access issues.

[0048] As mentioned before, a KVM may be implemented as a standalone hardware device (e.g., described in FIG. 7), or may be implemented as a software installed on an independent computing device. In terms of the software implementation, a physicalconnection (e.g., via a USB ca le and / or video) between the device on which KVM component 212 is installed and computing device 206. The KVM component 212 may be programmed to capture the display output on computing device 206 and transmit a real-time video stream of the display output to remote computing device 216. A remote control software (e.g., nCommand or nVirtual) installed on remote computing device 216 may receive and generate the video stream for display.

[0049] FIG. 3 is a diagram illustrating an exemplary user interface 300 for accessing medical equipment remotely. For example, user interface 300 may be displayed on remote computing device 216 during an MRI scan. Output window 302 may include the video stream captured from computing device 206. As shown in FIG. 3, output window 302 includes MRI images and various toolbar(s) / menu(s) 304 with imaging options such as for rotating, cropping, zooming into, and modifying the MRI images. A remote user can access these options using their interface hardware (e.g., mouse, touchscreen, keyboard, etc.) connected to remote computing device 216. For example, the remote user may select option 306, which adjusts a contrast / brightness level of an MRI image. The remote control software may transmit the selection of option 306 to KVM component 212, which executes the command on computing device 206.

[0050] In some aspects, the remote control software further receives and displays camera stream 308 on user interface 300. Camera stream 308 may be a video stream captured by camera(s) 208. In some aspects, camera(s) 208 transmit their video stream directly to remote computing device 216. In other aspects, camera(s) 208 transmit their video stream to KVM component 212. The KVM component 212 then forwards the video stream from computing device 206 to remote computing device 216.

[0051] User interface 300 may further display command window 310, which generates virtual options corresponding to physical input options on medical interfacing equipment 210. For example, a physical input option may include, but it is not limited to, a button, a switch, a toggle, a touch-sensitive surface, etc. It should be noted that while the mouse and keyboard of computing device 206 may be used to interact with the medical equipment 214, medical interfacing equipment 210 is used to control functionality of medical equipment 214. Each medical interfacing equipment 210 may have a particular device type. Accordingly, when beginning a remote session, the remote control software may transmit a query to applicationserver(s) 224 that asks for the device type. This query may specifically ask for a product identifier such as a model number. In response to receiving the device type (e.g., a Philips intercom, a GE intercom, a Siemens intercom, etc.) and / or the product identifier (e.g., part number 4757261), the remote control software may perform a lookup in an interfacing device layout database. The interfacing device layout database may indicate, for each entry (of medical interfacing equipment), a plurality of options found physically on the equipment. In response to finding the entry for the received device type and product identifier, the remote control software may generate the corresponding plurality of options in command window 310. In some aspects, interfacing device layout database is stored on application server(s) 224 and is periodically updated to include layouts of new medical equipment.

[0052] Suppose that the remote control software receives a user selection of option 312 in command window 310. In one aspect, the remote control software converts the user selection on remote computing device 216 to a selection of the button corresponding to option 312 on medical interfacing equipment 210. For example, command window 310 displays five options, which correspond to five buttons on medical interfacing equipment 210. In response to receiving option 312, the remote control software determines, based on a mapping in the interfacing device layout database, that button 1 on medical interfacing equipment 210 should be selected. The remote control software transmits a command to interface controller device 228 to select button 1. More specifically, the command may be to activate a solenoid that corresponds to button 1.

[0053] In an alternative aspect, interface controller device 228 locally converts the user selection on remote computing device 216 to a selection of the button corresponding to option 312 on medical interfacing equipment 210. For example, command window 310 displays five options, which correspond to five buttons on medical interfacing equipment 210. In response to receiving option 312, the remote control software transmits an indication that option 312 has been selected to interface controller device 228. Interface controller device 228 then determines, based on a mapping in the interfacing device layout database, that button 1 on medical interfacing equipment 210 should be selected.

[0054] In some aspects, computing device 206 is connected to medical interfacing equipment 210. In this case, remote computing device 216 transmits, via the network, the selection of the button 1 to interface controller device 228, which executes the buttonselection and performs the action on medical interfacing equipment 210 accordingly. For example, option 312 may involve activating a speaker of medical interfacing equipment 210 in order to allow the local user and / or the remote user to speak with a patient inside the medical equipment 214. This option is unavailable on the software that displays the MRI scans on computing device 206 and requires the use of medical interfacing equipment 210. In a traditional setting, the local user selects the button manually. However, using the remote control software and interface controller device 228, the remote user is able to access the functionality of medical interfacing equipment 210 as described above.

[0055] In some aspects, the medical interfacing equipment 210 cannot be connected directly to computing device 206. For example, medical interfacing equipment 210 may be connected to medical equipment 214, which may further be connected to computing device 206. The indirect connection prevents computing device 206 from transmitting commands to medical interfacing equipment 210. In this case, a button pressing device may be connected to remote computing device 216. The button pressing device may receive (via a wired connection) inputs from remote computing device 216 (e.g., to select button 1 on medical interfacing equipment 210) and activate a solenoid built in the button pressing device that executes the inputs (e.g., a manual press of button 1).

[0056] It should be noted that the remote control software is not comparable to the screensharing and remote control capabilities in videotelephony software. This is because videotelephony software requires being installed on both devices to enable screensharing and remote control. However, installing videotelephony software on computing device (Medical Device) 206 or on medical equipment 214 is not readily possible. The fact that the remote control software of the present disclosure does not need to be installed on computing device (Medical Device) 206 or on medical equipment 102 provides an additional benefit that computing resources are not taken from either computing device (Medical Device) 206 or on medical equipment 214.

[0057] In some aspects, KVM component 212 utilizes Web Real-Time Communication (WebRTC) to perform keyboard actions, mouse movements and clicks for mouse / touch inputs received on remote computing device 216 and execution on the inputs on computing device 206. Because time is a crucial component in certain medical procedures (e.g., delays even by a few seconds can null an examination or may be harmful to the patient), KVM component212 is configured to minimize the time latency between input receipt and command execution. In conventional remote assistance systems, WebRTC is used for transmitting audio, video and any other data streaming as mouse / keyboard actions.

[0058] There are various benefits to using a software-based KVM component. In practice, the medical equipment 214 is connected to the computing device 206, which in turn recognizes the KVM component 212 only as if a mouse, keyboard, and monitor were attached (plug and play). Thus, there is no additional memory or processor consumption, allowing the use of the KVM mechanism based on the software-based KVM without compromising the efficiency and effectiveness of the machine that is being remotely controlled. Additionally, maintaining the characteristics of medical equipment 214 and the computing device 206, which undergoes no alteration, enables the preservation of the manufacturer's machine warranty, an asset of significant importance to owners.

[0059] Moreover, the absence of software installation on the computing device 206 being remotely controlled is essential for remote operation activities in high-security contexts (where the medical equipment must be isolated from network connections and software installations that may compromise its security) or low physical access to the medical equipment, compromising its on-site operation. These are conditions found, for example, in computers used in mining, power plants, industrial plants, health, among other market segments. Therefore, whenever there is a need for remote operation of medical equipment, but there is an impediment to access or a restriction on the installation of remote access software on medical equipment that is being accessed, the solution in the present disclosure can be applied.

[0060] In addition to using the software-based KVM, it is possible to stream video from devices with USB / HDMI output, as well as the medical equipment screen, to the remote user through the installed software. No additional software installation is necessary to enable this support.

[0061] The video quality of the solution is defined according to the requirements of computing device 206, specifically the capture card, which can reach a resolution of 4K (3840 x 2160 pixels) at 60 frames per second.

[0062] In some aspects, the communication of mouse and keyboard events is done through a proprietary protocol, reducing the amount of network traffic and providing greatersecurity for performing any action on the computing device 206.

[0063] In addition to performing mouse and keyboard inputs, KVM component 212 allows a user of remote computing device 216 to check the state of active keys on computing device 206. This functionality allows the keyboard of remote computing device 216 to be synchronized with the physical keyboard of the computing device 206.

[0064] To ensure the security of access and control of devices used by the KVM component 212, all access events to the product are saved and can be requested at any time. In one aspect, the systems and methods utilize the WebRTC protocol (RFC 8835) for controlling computing device 206 by remote computing device 216 using a secure connection..

[0065] Regarding the availability of the solution in the market, its scalability is a key feature considering the following set of factors. First, there is an independence from standalone KVM hardware for the solution to work. In addition to the computing device 206 and remote computing device 216, which are always present in any operation intermediated by KVM solutions already available in the market, only a third computer (e.g., computing device), the USB cable (Connect Cable), and the video cable are required. Third, there is interoperability of systems, as the computing device 206 recognizes the KVM component 212 as if a monitor, a mouse, and a keyboard were attached (plug and play), without requiring specific adaptations for each type of system.

[0066] FIG. 4 is a diagram 400 illustrating physical connections in the remote operation system 100 using a software-based KVM component. In diagram 400, computing device 403 may be a part of medical equipment 214. Accordingly, the software-based KVM component is installed on computing device 404, which is connected to computing device 403.

[0067] For example, monitor 402 may be connected to the video source of computing device 403 via cable 412. A user may adjust the views outputted by medical device 214 using mouse 410 and keyboard 408 (connected to computing device 403 via cables 416 and 418, respectively). In order to receive the video output from medical equipment 214, cable 414 is connected between the respective video sources of computing device 403 and computing device 404 (the device with KVM component 212 installed). In order to receive mouse and keyboard inputs, cable 406, which may be a USB cable, is connected between computing device 403 and 404.

[0068] FIG. 5 is a diagram 500 illustrating security features of the remote operation system. The state changes of Ionic Connect 502 (used interchangeably with the remote operating system 100) and its interactions with devices are carried out through requests to REST Application Programming Interface (API) 504. This interface for accessing the KVM component software classifies users (e.g., client 501), ensuring that only authorized individuals can access sensitive resources (such as network configuration and access to video / mouse / keyboard) of the computing devices. In some aspects, authentication and authorization management are provided by the OAuth 2.0 security standard (RFC 6749). Information about authorized users may be stored in database 508. User status verification is based on access and refresh tokens. Information regarding available transmission channels is obtained through requests to the API.

[0069] In FIG. 5, the flow of information and data security mechanisms present in the software-based KVM component is illustrated. In some aspects, all input and output data in the software is encrypted. Sensitive information contained in Ionic Connect 502 is only available to authorized users. Therefore, the solution does not offer methods of control or access to the environment in which it is installed (i.e., host) without control. The user's connection to WebRTC 506 is established through a session, and this context is fully encrypted, preventing scenarios of data capture such as man-in-the-middle.

[0070] FIG. 5 further shows Datagram Transport Layer Security (DTLS) being used in the transmittal of video 520, audio 522, data 524 (e.g., keyboard / mouse inputs), and data 526 (e.g., quality of service (QoS) information) between client 501 (e.g., remote computing device 216) and devices 510 and 512 (e.g., computing devices 403 and 404). In some aspects, data 524 passes through control channel 516 and data 526 passes through QoS channel 518.

[0071] FIG. 6 is a diagram 600 illustrating a plurality of channels used to exchange data in the remote operation system. In some aspects, Ionic Connect 502 uses the network infrastructure to communicate with end-users. For example, this data may be passed through the HTTPS Over TLS protocol (RFC 2818) or WebRTC technology (RFC 8835). Transport of data through WebRTC may be securely done via Datagram Transport Layer Security (DTLS) defined in RFC 6347. Therefore, all communication in the Ionic software contains end-to-end data protection.

[0072] The WebRTC 506 enables web and mobile applications to make real-time audio and video calls directly between users' devices without the need for plugins or additional software. To implement WebRTC 506 in an application, it's necessary to use WebRTC's communication APIs, which allow communication between users' browsers and devices. This technology defines three types of channels for data traffic: data, video, and audio.

[0073] In this context, WebRTC is used to perform video, mouse / keyboard, and general data traffic, as detailed below:- Mouse / Keyboard Channel 604: The mouse and keyboard event communication channel is bidirectional (i.e., sends events and receives returns from the Ionic Connect).- Quality of Service Channel 606: Every WebRTC connection in Ionic Connect 502 has a functional QoS channel used to validate the user experience.- Video / Audio Track (608 and 612 / 610 and 614, respectively): Video / audio transmission data is carried by the video track in the WebRTC standard. In Ionic Connect 502, this stream can be either a data send or receive, depending on the perspective.

[0074] As illustrated in FIG. 6, the communication channels have variations, Table 1 below describes the WebRTC channels in Ionic Connect 502.Table 1

[0075] To ensure reliability and security in the use of KVM component 212, interactions with devices and the host machine (e.g., computing device 404), as well as changes and accesses, are recorded to allow for future analysis. Table 2 describes the levels of logs that may be present in the system:

[0076] In some aspects, the Ionic Connect 502 allows for video processing using video compression standards such as h.264 and h.265. H.265, a newer codec also known as HEVC, provides greater compression efficiency, resulting in smaller file sizes while maintaining the same or even superior video quality compared to h.264. In other words, h.265 is capable of compressing videos with improved quality, occupying less storage space or consuming less bandwidth for transmission. The traffic of this data can be carried out through a video capture card or through the USB interface.

[0077] In some aspects, Ionic Connect 502 further includes a service responsible for managing and communicating between the end client and the target machine (i.e., the computer that is being controlled). Mouse and keyboard events are stored in a buffer until they are processed and sent to the Ionic Health Connect Cable (e.g., cable 406). This buffer has another service, which is responsible for serially writing the packet to the mouse / keyboard cable.

[0078] In some aspects, the architecture implemented in the mouse and keyboard service allows cable 406 to receive data (i.e., mouse and keyboard events) from multiple clients, thus enabling remote machine control by multiple users simultaneously. As previously explained, the cable 406 provides key state feedback to the remote user through a broadcast topology. Each mouse and keyboard cable (e.g., cable 406) has its own instance responsible for its management within Ionic Connect 502.

[0079] In addition to connection establishments that come from the API, there are scenarios on the current internet where the end-user cannot access KVM component 212 directly. As a solution, Ionic Connect 502 has a service where the connection is made to some visible service (i.e., accessible via the internet) through the Remote Procedure Call (RPC) protocol. This input data inhibits the need to use the API directly.

[0080] Because the software uses WebRTC to transmit video, mouse, and keyboard data, this transmission occurs completely with the help of a common STUN / TURN server. In this case, with the help of a signaling service (Signaling Server) that functions as middleware, the connection is established end-to-end through NAT traversal. The servers described above may be part of cloud computation platform 220.

[0081] FIG. 7 is a diagram 700 illustrating physical connections in the remote operation system 100 using a hardware KVM device 702. In diagram 700, medical equipment 214 is connected to computing device 206 via cable 704 (e.g., an HDMI, DVI, DisplayPort, USB-C, VGA, etc.). Computing device 206 is connected to mouse 714 and keyboard 716 via USB cables 718. In the software-implementation of the KVM described in FIG. 2, computing device 206 is directly connected to monitor 708 using a display cable. In contrast, the hardwareimplementation of the KVM involves connecting computing device 206 to KVM device 702 using display cable 710. KVM device 702 is further connected to monitor 708 using display cable 712. In an exemplary aspect, KVM device 702 has an embedded video splitter that enables one video stream from computing device 206 to be presented on monitor 708 and a duplicative video stream to be presented on a monitor of remote computing device 216. KVM device 702 further receives peripheral device inputs (e.g., from mouse 714 and keyboard 716) via one or more USB cable(s) 706. Other peripherical device inputs include touchscreen inputs, camera inputs, and microphone inputs.

[0082] In an exemplary aspect, KVM device 702 utilizes WebRTC when communicating with remote computing device 216. This offers the advantage of not needing a VPN connection between KVM device 702 and remote computing device 216, increasing the product scalability and installation process.

[0083] FIG. 8 illustrates a flow diagram of method 800 for provisioning remote operation of medical equipment.

[0084] At 802, a KVM component captures a video output comprising medical data as displayed on a first computing device connected to the medical equipment, wherein the medical data is produced by the medical equipment.

[0085] At 804, the KVM component transmits the captured video output to a second computing device remotely connected to the KVM component, wherein the second computing device outputs a user interface that includes the captured video output and one or more virtual buttons corresponding to one or more buttons on medical interfacing equipment, wherein the one or more buttons each control a respective function of the medical equipment.

[0086] At 806, the KVM component receives, from the second computing device, a selection of a virtual button of the one or more virtual buttons.

[0087] At 808, the KVM component transmits, to the medical interfacing equipment via an interface controller device, a command to execute a function corresponding to a button, of the one or more buttons, that corresponds to the virtual button.

[0088] FIG. 9 is a block diagram illustrating a computer system 20 on which aspects of systems and methods for provisioning remote operation of medical equipment may be implemented in accordance with an exemplary aspect. The computer system 20 can be in the form of multiple computing devices, or in the form of a single computing device (e.g., computing device 206, remote computing device 216, etc.), for example, a desktop computer, a notebook computer, a laptop computer, a mobile computing device, a smart phone, a tablet computer, a server, a mainframe, an embedded device, and other forms of computing devices.

[0089] As shown, the computer system 20 includes a central processing unit (CPU) 21, a system memory 22, and a system bus 23 connecting the various system components, including the memory associated with the central processing unit 21. The system bus 23 may comprise a bus memory or bus memory controller, a peripheral bus, and a local bus that is able to interact with any other bus architecture. Examples of the buses may include PCI, ISA, PCI-Express, HyperTransport™, InfiniBand™, Serial ATA, l2C, and other suitable interconnects. The central processing unit 21 (also referred to as a processor) can include a single or multiple sets of processors having single or multiple cores. The processor 21 may execute one or more computer-executable code implementing the techniques of the present disclosure. For example, any of commands / steps discussed in FIGS. 1-8 may be performed by processor 21. The system memory 22 may be any memory for storing data used herein and / or computer programs that are executable by the processor 21. The system memory 22 may include volatile memory such as a random access memory (RAM) 25 and non-volatile memory such as a read only memory (ROM) 24, flash memory, etc., or any combination thereof. The basic input / output system (BIOS) 26 may store the basic procedures for transfer of information between elements of the computer system 20, such as those at the time of loading the operating system with the use of the ROM 24.

[0090] The computer system 20 may include one or more storage devices such as one or more removable storage devices 27, one or more non-removable storage devices 28, or a combination thereof. The one or more removable storage devices 27 and non-removablestorage devices 28 are connected to the system bus 23 via a storage interface 32. In an aspect, the storage devices and the corresponding computer-readable storage media are powerindependent modules for the storage of computer instructions, data structures, program modules, and other data of the computer system 20. The system memory 22, removable storage devices 27, and non-removable storage devices 28 may use a variety of computer- readable storage media. Examples of computer-readable storage media include machine memory such as cache, SRAM, DRAM, zero capacitor RAM, twin transistor RAM, eDRAM, EDO RAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM; flash memory or other memory technology such as in solid state drives (SSDs) or flash drives; magnetic cassettes, magnetic tape, and magnetic disk storage such as in hard disk drives or floppy disks; optical storage such as in compact disks (CD-ROM) or digital versatile disks (DVDs); and any other medium which may be used to store the desired data and which can be accessed by the computer system 20.

[0091] The system memory 22, removable storage devices 27, and non-removable storage devices 28 of the computer system 20 may be used to store an operating system 35, additional program applications 37, other program modules 38, and program data 39. The computer system 20 may include a peripheral interface 46 for communicating data from input devices 40, such as a keyboard, mouse, stylus, game controller, voice input device, touch input device, or other peripheral devices, such as a printer or scanner via one or more I / O ports, such as a serial port, a parallel port, a universal serial bus (USB), or other peripheral interface. A display device 47 such as one or more monitors, projectors, or integrated display, may also be connected to the system bus 23 across an output interface 48, such as a video adapter. In addition to the display devices 47, the computer system 20 may be equipped with other peripheral output devices (not shown), such as loudspeakers and other audiovisual devices.

[0092] The computer system 20 may operate in a network environment, using a network connection to one or more remote computers 49. The remote computer (or computers) 49 may be local computer workstations or servers comprising most or all of the aforementioned elements in describing the nature of a computer system 20. Other devices may also be present in the computer network, such as, but not limited to, routers, network stations, peer devices or other network nodes. The computer system 20 may include one or more network interfaces 51 or network adapters for communicating with the remote computers 49 via oneor more networks such as a local-area computer network (LAN) 50, a wide-area computer network (WAN), an intranet, and the Internet. Examples of the network interface 51 may include an Ethernet interface, a Frame Relay interface, SONET interface, and wireless interfaces.

[0093] Aspects of the present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0094] The computer readable storage medium can be a tangible device that can retain and store program code in the form of instructions or data structures that can be accessed by a processor of a computing device, such as the computing system 20. The computer readable storage medium may be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. By way of example, such computer-readable storage medium can comprise a random access memory (RAM), a read-only memory (ROM), EEPROM, a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), flash memory, a hard disk, a portable computer diskette, a memory stick, a floppy disk, or even a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon. As used herein, a computer readable storage medium is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or transmission media, or electrical signals transmitted through a wire.

[0095] Computer readable program instructions described herein can be downloaded to respective computing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network interface in each computing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing device.

[0096] Computer readable program instructions for carrying out operations of the present disclosure may be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language, and conventional procedural programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a LAN or WAN, or the connection may be made to an external computer (for example, through the Internet). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0097] In various aspects, the systems and methods described in the present disclosure can be addressed in terms of modules. The term "module" as used herein refers to a real- world device, component, or arrangement of components implemented using hardware, such as by an application specific integrated circuit (ASIC) or FPGA, for example, or as a combination of hardware and software, such as by a microprocessor system and a set of instructions to im lement the module's functionality, which (while being executed) transform the microprocessor system into a special-purpose device. A module may also be implemented as a combination of the two, with certain functions facilitated by hardware alone, and other functions facilitated by a combination of hardware and software. In certain implementations, at least a portion, and in some cases, all, of a module may be executed on the processor of a computer system. Accordingly, each module may be realized in a variety of suitable configurations, and should not be limited to any particular implementation exemplified herein.

[0098] In the interest of clarity, not all of the routine features of the aspects are disclosed herein. It would be appreciated that in the development of any actual implementation of thepresent disclosure, numerous implementation-specific decisions must be made in order to achieve the developer's specific goals, and these specific goals will vary for different implementations and different developers. It is understood that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skill in the art, having the benefit of this disclosure.

[0099] Furthermore, it is to be understood that the phraseology or terminology used herein is for the purpose of description and not of restriction, such that the terminology or phraseology of the present specification is to be interpreted by the skilled in the art in light of the teachings and guidance presented herein, in combination with the knowledge of those skilled in the relevant art(s). Moreover, it is not intended for any term in the specification or claims to be ascribed an uncommon or special meaning unless explicitly set forth as such.

[0100] The various aspects disclosed herein encompass present and future known equivalents to the known modules referred to herein by way of illustration. Moreover, while aspects and applications have been shown and described, it would be apparent to those skilled in the art having the benefit of this disclosure that many more modifications than mentioned above are possible without departing from the inventive concepts disclosed herein.

Claims

CLAIMS1. A system for provisioning remote operation of medical equipment; comprising: medical equipment that produces medical data; medical interfacing equipment comprising one or more buttons each controlling a respective function of the medical equipment; and a keyboard, video, and mouse (KVM) component configured to: capture a video output comprising the medical data as displayed on a first computing device connected to the medical equipment; transmit the captured video output to a second computing device remotely connected to the KVM component, wherein the second computing device outputs a user interface that includes the captured video output and one or more virtual buttons corresponding to the one or more buttons on the medical interfacing equipment; receive, from the second computing device, a selection of a virtual button of the one or more virtual buttons; and transmit, to the medical interfacing equipment, a command to execute a function associated with a button, of the one or more buttons, that corresponds to the virtual button.

2. The system of claim 1, wherein the KVM component is a software application installed on a third computing device connected to the first computing device.

3. The system of claim 1, wherein the KVM component is a standalone device connected to the first computing device.

4. The system of claim 3, wherein the KVM component comprises an embedded video splitter.

5. The system of claim 3, wherein the KVM component connects to the medical equipment using a USB cable.

6. The system of claim 1, wherein the user interface further enables interaction with the medical data using one or more options selectable from the captured video output, wherein the KVM component is further configured to: receive, from the second computing device, a selection of the one or more options; and transmit, to the first computing device, the selection of the one or more options for execution.

7. The system of claim 1, wherein the KVM component is further configured to: capture user inputs received at the first computing device; and transmit the captured user inputs to the second computing device.

8. The system of claim 1, wherein the KVM component is configured to use Web Real- Time Communication (WebRTC) over a connection with the second computing device.

9. The system of claim 1, wherein prior to transmitting the command, an interface controller device is configured to determine that the virtual button in the selection corresponds to the button on the medical interfacing equipment by: searching a layout database that maps different virtual buttons to physical buttons on a plurality of medical interfacing equipment.

10. A method for provisioning remote operation of medical equipment, comprising: capturing, by a keyboard, video, and mouse (KVM) component, a video output comprising medical data as displayed on a first computing device connected to the medical equipment, wherein the medical data is produced by the medical equipment; transmitting, by the KVM component, the captured video output to a second computing device remotely connected to the KVM component, wherein the second computing device outputs a user interface that includes the captured video output and one or more virtual buttons corresponding to one or more buttons on medical interfacing equipment, wherein the one or more buttons each control a respective function of the medical equipment;receiving; from the second computing device, a selection of a virtual button of the one or more virtual buttons; and transmitting, to the medical interfacing equipment, a command to execute a function corresponding to a button, of the one or more buttons, that corresponds to the virtual button.

11. The method of claim 10, wherein the KVM component is a software application installed on a third computing device connected to the first computing device.

12. The method of claim 10, wherein the KVM component is a standalone device connected to the first computing device.

13. The method of claim 12, wherein the KVM component comprises an embedded video splitter.

14. The method of claim 12, wherein the KVM component connects to the medical equipment using a USB cable.

15. The method of claim 10, wherein the user interface further enables interaction with the medical data using one or more options selectable from the captured video output, wherein the KVM component is further configured to: receive, from the second computing device, a selection of the one or more options; and transmit, to the first computing device, the selection of the one or more options for execution.

16. The method of claim 10, wherein the KVM component is further configured to: capture user inputs received at the first computing device; and transmit the captured user inputs to the second computing device.

17. The method of claim 10, wherein the KVM component is configured to use Web Real- Time Communication (WebRTC) over a connection with the second computing device.

18. The method of claim 10, wherein prior to transmitting the command, an interface controller device is configured to determine that the virtual button in the selection corresponds to the button on the medical interfacing equipment by: searching a layout database that maps different virtual buttons to physical buttons on a plurality of medical interfacing equipment models that it can be configured.

Images