Systems and methods for mouse and keyboard synchronization

Abstract

Disclosed herein are systems and method for keyboard and mouse synchronization. The method may include receiving, from a remote computing device, cursor position values over a period of time. The method may include determining vertical and horizontal difference values of the cursor position. Depending on the size of the vertical and horizontal difference values, the method may include dividing the at least one of the horizontal difference value and the vertical difference value into a plurality of movement values, applying a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the first position to the second position, and transmitting the plurality of movement values to a local computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

Description

044617-00026 SYSTEMS AND METHODS FOR MOUSE AND KEYBOARD SYNCHRONIZATION CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of United States Provisional Application No. 63 / 517,002, filed August 1, 2023, which is herein incorporated by reference. FIELD OF TECHNOLOGY

[0002] The present disclosure relates to the field of remote technology, and, more specifically, to systems and methods for mouse and keyboard synchronization. BACKGROUND

[0003] Due to advancements in communication technology, the medical field has experienced improved accessibility. Medical professionals performing checkups no longer need to be physically present with their patients. For example, patients are now able to video call their doctors during an appointment instead of going into a hospital or clinic. Likewise, medical specialists that are unable to be physically present in a hospital can remotely monitor operations and provide guidance to technicians, nurses, and other doctors present at the hospital.

[0004] However, conventional remote assistance fails to meet the requirements of performing complex remote tasks such as taking control of different types of medical equipment from a remote setting and executing commands in near real-time. As medical equipment continues to advance exponentially in sophistication and functionality, it has becoming increasingly difficult to perform remote tasks, which further exposes the weaknesses of conventional remote assistance.

[0005] Time is one factor that greatly affects the effectiveness of remote operation of medical equipment. For example, if there is a large amount of latency between the transmission of a command from a remote computer and execution of the command at a044617-00026 local computer, certain tasks requiring near-instant execution simply cannot be performed. Depending on the medical procedure being performed, latency may potentially cause great harm to a patient. For example, latency during remote surgeries can be extremely dangerous. SUMMARY

[0006] To overcome the shortcomings of conventional remote assistance technology in the medical field, the present disclosure describes systems and methods that enable the remote operation of medical equipment (e.g., magnetic resonance imaging (MRI) & computerized tomography (CT) equipment) through a platform that combines robotics, hardware, and software in a non-invasive solution. Through the platform, users are also able to stream medical sessions (e.g., MRI, CT, PET-CT, etc), which improves collaboration, consulting, support, professional training, and quality control.

[0007] In some aspects, the techniques described herein relate to a method for mouse and keyboard synchronization, including: receiving, from a remote computing device, a first indication of a first position of a cursor on a user interface, wherein the first position is a first ratio of a horizontal cursor position X1 over a width of the user interface and a second ratio of a vertical cursor position Y1 over a height of the user interface; subsequently receiving, from the remote computing device, a second indication of a second position of the cursor on the user interface, wherein the second position is a third ratio of a new horizontal cursor position Xn over a width of the user interface and a fourth ratio of a new vertical cursor position Yn over a height of the user interface; calculating a horizontal difference value between the first ratio and the third ratio and a vertical difference value between the second ratio and the fourth ratio, wherein the horizontal difference value is a horizontal relative movement of the cursor and the vertical difference value is a vertical relative movement of the cursor; determining whether the horizontal difference value and the vertical difference value are greater than a predetermined size; in response to determining that at least one of the horizontal difference value and the vertical difference value is greater than a predetermined size, dividing the at least one of the horizontal difference value and the vertical difference value into a plurality of movement values; applying a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the first position to the second position; and transmitting the plurality of044617-00026 movement values to a local computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

[0008] In some aspects, the techniques described herein relate to a system for mouse and keyboard synchronization, including: memory; and at least one hardware processor coupled with the memory, wherein the at least one hardware processor is configured to: receive, from a remote computing device, a first indication of a first position of a cursor on a user interface, wherein the first position is a first ratio of a horizontal cursor position X1 over a width of the user interface and a second ratio of a vertical cursor position Y1 over a height of the user interface; subsequently receive, from the remote computing device, a second indication of a second position of the cursor on the user interface, wherein the second position is a third ratio of a new horizontal cursor position Xn over a width of the user interface and a fourth ratio of a new vertical cursor position Yn over a height of the user interface; calculate a horizontal difference value between the first ratio and the third ratio and a vertical difference value between the second ratio and the fourth ratio, wherein the horizontal difference value is a horizontal relative movement of the cursor and the vertical difference value is a vertical relative movement of the cursor; determine whether the horizontal difference value and the vertical difference value are greater than a predetermined size; in response to determining that at least one of the horizontal difference value and the vertical difference value is greater than a predetermined size, divide the at least one of the horizontal difference value and the vertical difference value into a plurality of movement values; apply a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the first position to the second position; and transmit the plurality of movement values to a local computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

[0009] In some aspects, the techniques described herein relate to a method for mouse synchronization, including: establishing a connection between a first computing device and a second computing device that has a cable emulating a mouse device; executing a request that shifts a cursor within a desktop environment of the first computing device to an origin position; receiving, from a remote computing device by the second computing device, a first indication of a first position of a cursor on a user interface, wherein the first position is a first044617-00026 ratio of a horizontal cursor position X1 over a width of the user interface and a second ratio of a vertical cursor position Y1 over a height of the user interface; determining a projected first position by calculating a horizontal projection and a vertical projection of the first position using a desktop specification of the first computing device; determining a plurality of moves between the origin position and the projected first position; dividing at least one of a horizontal difference value and a vertical difference value into a plurality of movement values based on an amount of the plurality of moves, wherein the horizontal difference value is a horizontal relative movement of the cursor from the origin position and the projected first position and the vertical difference value is a vertical relative movement of the cursor from the origin position and the projected first position; applying a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the origin position to the projected first position; and transmitting the plurality of movement values to the first computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

[0010] The above simplified summary of example aspects serves to provide a basic understanding of the present disclosure. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects of the present disclosure. Its sole purpose is to present one or more aspects in a simplified form as a prelude to the more detailed description of the disclosure that follows. To the accomplishment of the foregoing, the one or more aspects of the present disclosure include the features described and exemplarily pointed out in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate one or more example aspects of the present disclosure and, together with the detailed description, serve to explain their principles and implementations.

[0012] FIG.1 is a diagram illustrating a remote operation system.

[0013] FIG. 2 is a diagram illustrating network connections in the remote operation system using a software-based keyboard, video, and mouse (KVM) component.044617-00026

[0014] FIG.3 is a diagram illustrating an exemplary user interface for accessing medical equipment remotely.

[0015] FIG. 4 is a diagram illustrating physical connections in the remote operation system using a software-based KVM component.

[0016] FIG.5 illustrates a flow diagram of a method for pan smoothing.

[0017] FIG. 6 illustrates a flow diagram of a method for mouse synchronization when provisioning remote operation of medical equipment.

[0018] FIG. 7 illustrates a flow diagram of a method for mouse synchronization when provisioning remote operation of medical equipment using an origin position.

[0019] FIG. 8 presents an example of a general-purpose computer system on which aspects of the present disclosure can be implemented. DETAILED DESCRIPTION

[0020] Exemplary aspects are described herein in the context of a system, method, and computer program product for provisioning remote operation of medical equipment. Those of ordinary skill in the art will realize that the following description is illustrative only and is not intended to be in any way limiting. Other aspects will readily suggest themselves to those skilled in the art having the benefit of this disclosure. Reference will now be made in detail to implementations of the example aspects as illustrated in the accompanying drawings. The same reference indicators will be used to the extent possible throughout the drawings and the following description to refer to the same or like items.

[0021] FIG.1 is a diagram illustrating a remote operation system 100. Remote operation system 100 includes medical equipment 102, medical interfacing equipment 104, computing device 106, and camera 112. Medical interfacing equipment 104 may be a controller for adjusting parameters of medical equipment 102. Computing device 106 may be any computer device capable of receiving and displaying results from medical equipment 102. Camera 112 may be used to monitor the medical environment (e.g., a clinic, hospital room, laboratory room, workstation, etc.).

[0022] In a typical scenario, user 108 is physically present in the environment and uses medical interfacing equipment 104 to control medical equipment 102 and view results on a044617-00026 monitor of computing device 106. For example, computing device 106 may be a desktop computer, medical equipment 102 may be an MRI machine, and medical interfacing equipment 104 may be an intercom / controller that controls scanning, adjusts movements of the components in the MRI machine, and enables communication between user 108 and the patient in the MRI machine.

[0023] Remote operation system 100 enables remote user 110 to remotely control medical interfacing equipment 104 and remotely view outputs of medical equipment 102 displayed on computing device 106. In an exemplary aspect, remote operation system 100 utilizes a keyboard, video, and mouse (KVM) component that enables said remote viewing and remote control. In one aspect, the KVM component is software-based and is installed on an independent computing device that employs a video capture card and USB cable connected to computing device 106. In another aspect, the KVM component is hardware- based and is connected to computing device 106 as a standalone KVM device. Both aspects are thoroughly described in the present disclosure.

[0024] In particular, the present disclosure describes the “remote control” aspect of the remote operation. In a medical setting, it may be possible that a specialist in a particular medical field cannot be physically present to perform a surgery, check-up, or medical procedure. Accordingly, remote operation is necessary in which the specialist becomes remote user 110. A successful remote operation involves enabling the remote user 110 to access medical equipment 102 almost as though he / she is physically present in the local environment. One aspect in achieving this level of access is minimizing any latencies between commands exchanged between remote user 110 and user 108. For example, there may be a threshold period of time (e.g., 100 milliseconds) of a Quality of Service (QoS) parameter within which a command executed by remote user 110 should appear on the screen of computing device 106 and vice versa. To meet this threshold and, the present disclosure describes systems and methods for mouse and keyboard synchronization.

[0025] FIG. 2 is a diagram 200 illustrating network connections in remote operation system 100. FIG.2 is a diagram 200 illustrating network connections in the remote operation system 100 using a software-based keyboard, video, and mouse (KVM) component 212. Diagram 200 depicts two networks: customer network 202 and public network 204. In some aspects, customer network 202 is a local area network (LAN) and public network 204 is a wide044617-00026 area network (WAN) (e.g., the Internet). In other aspects, both customer network 202 and public network 204 are WANs.

[0026] Customer network 202 features remote communication between devices in site A (e.g., a medical facility) and site B (e.g., a command center). Remote computing device 216 establishes a secure connection with KVM component 212 via computing device 206, the secure connection including one or more of virtual private network (VPN) capabilities, multi- protocol label switching (MPLS) capabilities, and LAN to LAN connection capabilities. For example, KVM component 212 may be installed on installed on an independent computing device that employs a video capture card and USB cable connected to computing device 206. In some aspects, remote computing device 216 may communicate with camera 208, medical interfacing equipment 210, and / or computing device 206 using any combination of Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Internet Control Message Protocol (ICMP).

[0027] Suppose that the display output of medical equipment 214 (e.g., an MRI machine) is presented on computing device 206 (e.g., a desktop computer). For example, MRI images may be displayed on computing device 206, which can be viewed, modified, and interacted with using peripheral devices (e.g., a mouse and keyboard). Using the secure connection, remote computing device 216 is able to receive and replicate the display output of medical equipment 214, as well as transmit commands to control the mouse and keyboard of computing device 206 and / or medical interfacing equipment 210. This ultimately allows a remote user to program and perform a medical task (e.g., an examination process using medical equipment 214) directly as if he / she were actually sitting at the clinic or hospital workstation.

[0028] In some aspects, a remote user may also be assisted by one or more of cameras 208 that generate real-time images of the patient’s position. For example, camera 208 may capture a video of the workstation in site A and transmit a stream over the secure connection to remote computing device 216.

[0029] In some aspects, the remote user of remote computing device 216 may communicate with a local user of computing device 206 using Voice over Internet Protocol (VoIP). This is enabled using communication platform 222, which receives and transmits audio information between the respective computing devices. In an exemplary aspect,044617-00026 communication platform includes VoIP server(s) 226 that perform communication exchanges.

[0030] In some aspects, cloud computation platform 220, which includes application server(s) 224, may be used by remote computing device 216 to perform various computational tasks. For example, the software (e.g., nCommand) utilized by remote computing device 216 to perform remote access and control of device(s) in site A may be an application featuring a thin client and thick client. The thin client application may be locally installed on remote computing device 216 to perform non-intensive processing tasks such as displaying information. The thick client application may be installed on application server(s) 224 and may perform intensive processing tasks such as translating keyboard inputs received from remote computing device 216 to inputs for medical interfacing equipment 210 based on layouts. Application server(s) 224 may additionally store device configuration information, user registrations, and other information. It should be noted that no patient information or exam images are stored on public network 204. In some aspects, the software may be a web application.

[0031] In some aspects, both remote computing device 216 and computing device 206 may communicate with servers in cloud computation platform 220 and communication platform 222 using TCP and Hypertext Transfer Protocol Secure (HTTPS) (with Secure Sockets Layer(SSL) / Transport Layer Security (TLS) 1.2) (Firewall or Proxy).

[0032] In some aspects, both computing device 206 and remote computing device 216 may contact technical support 218 to report, troubleshoot, and resolve remote access issues.

[0033] As mentioned before, a KVM may be implemented as a standalone hardware device (e.g., described in FIG.7), or may be implemented as a software installed on computing device 206. In terms of the software implementation, a physical connection (e.g., via a USB cable) may be made between computing device 206, on which KVM component 212 is installed, and medical equipment 214. KVM component 212 may be programmed to capture the display output on computing device 206 and transmit a real-time video stream of the display output to remote computing device 216. A remote control software (e.g., nCommand or nVirtual) installed on remote computing device 216 may receive and generate the video stream for display.

[0034] FIG. 3 is a diagram illustrating an exemplary user interface 300 for accessing medical equipment remotely. For example, user interface 300 may be displayed on remote044617-00026 computing device 216 during an MRI scan and a mirror of the display from computing device 206 may be displayed on output window 302 (e.g., a mirror user interface based on user interface 300 may be displayed on computing device 206). In some aspects, output window 302 may include the video stream captured from computing device 206. In an exemplary aspect, when a local user is moving a mouse or typing on a keyboard connected to computing device 206, the keyboard selections and the mouse movements of a cursor on the user interface of computing device 206 are shown on user interface 300. For example, cursor 314 may be moved by the local user from point A to point B. This movement is shown on user interface 300 as well. Due to the mouse synchronization algorithm of the present disclosure, the latency between the mouse being moved by the local user and the cursor 314 changing position is minimized to be no greater than a threshold period of time. Likewise, a remote user may change the position of the cursor 314 by moving his / her mouse connected to remote computing device 216 or finger (in a touchscreen setup), and the cursor shown on the user interface of computing device 206 will change position accordingly.

[0035] As shown in FIG. 3, output window 302 includes MRI images and various toolbar(s) / menu(s) 304 with imaging options such as for rotating, cropping, zooming into, and modifying the MRI images. A remote user can access these options using their interface hardware (e.g., mouse, touchscreen, keyboard, etc.) connected to remote computing device 216. For example, the remote user may select option 306, which adjusts a contrast / brightness level of an MRI image. The remote control software may transmit the selection of option 306 to KVM component 212, which executes the command on computing device 206.

[0036] In some aspects, the remote control software further receives and displays camera stream 308 on user interface 300. Camera stream 308 may be a video stream captured by camera(s) 208. In some aspects, camera(s) 208 transmit their video stream directly to remote computing device 216. In other aspects, camera(s) 208 transmit their video stream to KVM component 212. KVM component 212 then forwards the video stream to remote computing device 216.

[0037] User interface 300 may further display command window 310, which generates virtual options corresponding to physical input options on medical interfacing equipment 210. For example, a physical input option may include, but is not limited to, a button, a switch, a044617-00026 toggle, a touch-sensitive surface, etc. It should be noted that while the mouse and keyboard of computing device 206 may be used to interact with the output of medical equipment 214, medical interfacing equipment 210 is used to control functionality of medical equipment 214. Each medical interfacing equipment 210 may have a particular device type. Accordingly, when beginning a remote session, the remote control software may transmit a query to computing device 206 that asks for the device type. This query may specifically ask for a product identifier such as a model number. In response to receiving the device type (e.g., a Philips intercom, a GE intercom, a Siemens intercom, etc.) and / or the product identifier (e.g., part number 4757261), the remote control software may perform a lookup in an interfacing device layout database. The interfacing device layout database may indicate, for each entry (of medical interfacing equipment), a plurality of options found physically on the equipment. In response to finding the entry for the received device type and product identifier, the remote control software may generate the corresponding plurality of options in command window 310. In some aspects, interfacing device layout database is stored on application server(s) 224 and is periodically updated to include layouts of new medical equipment.

[0038] Suppose that the remote control software receives a user selection of option 312 in command window 310. In one aspect, the remote control software converts the user selection on remote computing device 216 to a selection of the button corresponding to option 312 on medical interfacing equipment 210. For example, command window 310 displays five options, which correspond to five buttons on medical interfacing equipment 210. In response to receiving option 312, the remote control software determines, based on a mapping in the interfacing device layout database, that button 1 on medical interfacing equipment 210 should be selected. The remote control software transmits a command to KVM component 212 to select button 1.

[0039] In an alternative aspect, KVM component 212 locally converts the user selection on remote computing device 216 to a selection of the button corresponding to option 312 on medical interfacing equipment 210. For example, command window 310 displays five options, which correspond to five buttons on medical interfacing equipment 210. In response to receiving option 312, the remote control software transmits an indication that option 312 has been selected to KVM component 212. KVM component 212 then determines, based on a mapping in the interfacing device layout database, that button 1 on medical interfacing044617-00026 equipment 210 should be selected.

[0040] In some aspects, computing device 206 is connected to medical interfacing equipment 210 via a USB cable. In this case, KVM component 212 transmits, via the USB cable, the selection of the button 1 to medical interfacing equipment 210, which executes the button selection and adjusts medical equipment 214 accordingly. For example, option 312 may involve activating a speaker of medical equipment 214 in order to allow the local user and / or the remote user to speak with a patient inside the MRI machine. This option is unavailable on the software that displays the MRI scans on computing device 206 and requires the use of medical interfacing equipment 210. In a traditional setting, the local user selects the button manually. However, using the remote control software and KVM component 212, the remote user is able to access the functionality of medical interfacing equipment 210 as described above.

[0041] In some aspects, the medical interfacing equipment 210 cannot be connected directly to computing device 206. For example, medical interfacing equipment 210 may be connected to medical equipment 214, which may further be connected to computing device 206. The indirect connection prevents computing device 206 from transmitting commands to medical interfacing equipment 210. In this case, a button pressing device may be connected to computing device 206. The button pressing device may receive (via a wired or wireless connection) inputs from KVM component 212 (e.g., to select button 1 on medical interfacing equipment 210) and activate a solenoid built in the button pressing device that executes the inputs (e.g., a manual press of button 1).

[0042] It should be noted that the remote control software is not comparable to the screensharing and remote control capabilities in videotelephony software. This is because videotelephony software requires being installed on both devices to enable screensharing and remote control. However, installing videotelephony software on medical interfacing equipment 210 is not readily possible. This prevents such software from allowing command selection on medical interfacing equipment 210. The fact that the remote control software of the present disclosure does not need to be installed on medical interfacing equipment 210 provides an additional benefit that computing resources are not taken from either medical equipment 214 or medical interfacing equipment 210.

[0043] In some aspects, KVM component 212 utilizes Web Real-Time Communication044617-00026 (WebRTC) to perform mouse and keyboard synchronization for mouse / touch inputs received on remote computing device 216 and execution on the inputs on computing device 206. Because time is a crucial component in certain medical procedures (e.g., delays even by a few seconds can null an examination or may be harmful to the patient), KVM component 212 is configured to minimize the time latency between input receipt and command execution. In conventional remote assistance systems, WebRTC is solely used for transmitting audio and video and is not used for mouse and keyboard synchronization.

[0044] There are various benefits to using a software-based KVM component. In practice, the medical equipment 214 recognizes the KVM component 212 only as if a mouse, keyboard, and monitor were attached (plug and play). Thus, there is no additional memory or processor consumption, allowing the use of the KVM mechanism based on the software-based KVM without compromising the efficiency and effectiveness of the machine that is being remotely controlled. Additionally, maintaining the characteristics of medical equipment 214, which undergoes no alteration, enables the preservation of the manufacturer's machine warranty, an asset of significant importance to owners.

[0045] Moreover, the absence of software installation on the medical equipment 214 being remotely controlled is essential for remote operation activities in high-security contexts (where the medical equipment must be isolated from network connections and software installations that may compromise its security) or low physical access to the medical equipment, compromising its on-site operation. These are conditions found, for example, in computers used in mining, power plants, industrial plants, health, among other market segments. Therefore, whenever there is a need for remote operation of medical equipment, but there is an impediment to access or a restriction on the installation of remote access software on medical equipment that is being accessed, the solution in the present disclosure can be applied.

[0046] In addition to using the software-based KVM, it is possible to stream video from devices with USB / HDMI output, as well as the medical equipment screen, to the remote user through the installed software. No additional software installation is necessary to enable this support.

[0047] The video quality of the solution is defined according to the requirements of computing device 206, specifically the capture card, which can reach a resolution of 4K (3840044617-00026 x 2160 pixels) at 60 frames per second.

[0048] In some aspects, the communication of mouse and keyboard events is done through a proprietary protocol, reducing the amount of network traffic and providing greater security for performing any action on the medical equipment.

[0049] In addition to performing mouse and keyboard inputs, KVM component 212 allows a user of remote computing device 216 to check the state of active keys on computing device 206. This functionality allows the keyboard of remote computing device 216 to be synchronized with the physical keyboard of the computing device 206.

[0050] To ensure the security of access and control of devices used by the KVM component 212, all access events to the product are saved and can be requested at any time. In one aspect, the systems and methods utilize the WebRTC protocol (RFC 8835) for controlling computing device 206 by remote computing device 216 without the need for a VPN. The traffic of streaming data (mouse, keyboard, and video) through application servers 224 is optional.

[0051] Regarding the availability of the solution in the market, its scalability is a key feature considering the following set of factors. First, there is an independence from standalone KVM hardware for the solution to work. In addition to the medical equipment 214 and remote computing device 216, which are always present in any operation intermediated by KVM solutions already available in the market, only a third computer (e.g., computing device represented by KVM component 212), the USB cable (Connect Cable), and the video cable are required. Third, there is interoperability of systems, as the computing device 206 recognizes the KVM component 212 as if a monitor, a mouse, and a keyboard were attached (plug and play), without requiring specific adaptations for each type of system.

[0052] FIG.4 is a diagram 400 illustrating physical connections in the remote operation system 100 using a software-based KVM component. In diagram 400, computing device 403 may be a part of medical equipment 214. This prevents installation of the software-based KVM component. Accordingly, the software-based KVM component is installed on computing device 404, which is connected to computing device 403.

[0053] For example, monitor 402 may be connected to the video source of computing device 403 via cable 412. A user may adjust the views outputted by medical device 214 using mouse 410 and keyboard 408 (connected to computing device 403 via cables 416 and 418,044617-00026 respectively). In order to receive the video output from medical equipment 214, cable 414 is connected between the respective video sources of computing device 403 and computing device 404 (the device with KVM component 212 installed). In order to receive mouse and keyboard inputs, cable 406, which may be a USB cable, is connected between computing device 403 and 404.

[0054] In an exemplary aspect, a remote user may transmit a command (e.g., a selection of option 306 on user interface 300) via remote computing device 216 to computing device 404. In particular, the command may be initiated by clicking option 306 with a mouse or a finger (in the case of touchscreen). Because the selection of option 306 is made on remote computing device 216, the selection is to be made on computing device 403 as though option 306 were selected via mouse 410. Likewise, if a selection is made by a local user on computing device 403 (e.g., via mouse 410), the selection is to be shown and made on remote computing device 216.

[0055] As mentioned before, minimizing the latency between making a selection and remotely seeing the selection be selected is crucial to allow for complicated tasks (e.g., advanced surgeries or checkups) to be performed. Thus, the present disclosure describes an advanced mouse and keyboard event algorithm that minimizes the latency relative to conventional mouse and keyboard synchronization algorithms and provides an excellent quality of experience for the end-user. To achieve this, an external device is necessary to emulate a relative mouse, an absolute mouse, and a keyboard. This external device is called the connect cable (e.g., cable 406) and functions as a cable that connects computing device 403 and computing device 404. In one aspect, cable 406 emulates a relative mouse, an absolute mouse, and a keyboard using a universal serial bus (USB) and human interface devices (HID). In one aspect, cable 406 receives and sends data packets.

[0056] Client sync application 424 and server sync application 426 are installed on remote computing device 420 and computing device 404, respectively. These are applications configured to generate and transmit mouse and keyboard events over one or more server(s) over a wireless connection. In some aspects, the server(s) are one or more application server(s) part of cloud computation platform 220. Remote computing device 420 may be connected to mouse 432 and keyboard 434, through which a remote user may make selections and provide inputs. Remote computing device 420 is connected to computing044617-00026 device 404 over a wireless connection (e.g., the Internet). In some aspects, information exchanged between remote computing device 420 and computing device 404 via the server(s) uses an "Ionic Connect Protocol" (ICP), which is a proprietary protocol designed to save bytes sent over the network by being lightweight and simple. In some aspects, client sync application 424 and server sync application 426 are modules with a software application such as nCommand.

[0057] Cable 406 is configured to listen to inputs from server sync application 426 (originating from remote computing device 420) and emulate a keyboard, absolute mouse, and relative mouse device. There is a thread responsible for listening to requests from client sync application 424 and validating the request format based on Connect Cable Protocol (CCP), which is the standard communication protocol between client sync application 424 and cable 406. When the information is fully validated, it is inserted into the queue for their respective devices. For each device, keyboard, absolute mouse, or relative mouse, there is a thread that reads its respective queue and, if there is a new request (e.g., a new object in the queue), it is sent through the USB HID device. Computing device 403 will recognize the HID input request and perform the requested action.

[0058] In an exemplary aspect, keyboard events are detected by client sync application 424 and undergo an internal encoding defined by the ICP. Then a data packet containing information about the key state (e.g., pressed or released), and the encoded key is sent to server sync application 426. When server sync application 426 receives a keyboard command, it also receives the key identifier (in the ICP-encoded format), a timestamp, and the key state (e.g., pressed or released), and processes it for computing device 404. Computing device 404 performs a series of checks to validate the information and ensure that the requested command is expected. For example, computing device 404 may keep track of the keys pressed in a buffer of length six, so if a request is made to release a key that is not in the buffer, it will simply ignore the request.

[0059] Computing device 404 implements a structure to simulate how the keyboard's "HID input report" should be, so whenever cable 406 receives an input from client sync application 424, it simply performs the USB "HID input report" without having to keep track of the keys. This structure, implemented by server sync application 426, has a six-byte buffer length and a one-byte modifier. The modifier byte may include the modifier keys: Shift Left,044617-00026 Control Left, Meta Left, Alt Left, Shift Right, Control Right, Meta Right, and Alt Right. In an exemplary aspect, the buffer stores other keys, such as Key “A,” Key “B,” etc. These general keys, combined with the modifier, change the final result of the key, but this is entirely a configuration of the operating system. In some aspects, server sync application 426 is configured to re-encode the keys from "W3C" to "HID Usage ID".

[0060] In an exemplary aspect, a mouse event captured by client sync application 424 includes information about the current position of a cursor on a user interface (e.g., cursor 314 on user interface 300). In an exemplary aspect, the cursor position within computing device 403 is affected by the movements of a client mouse (e.g., mouse 410 and / or mouse 432). The cursor may move in a specific area inside the client-side application (showing user interface 300 on computing device 403). This area is defined by a width and a height measured in pixels. For example, consider the following: - ℕ0 = {0, 1, 2, 3, 4, ...} – Width, Height ∈ ℕ0 and Width, Height > 0 - Area = { (x, y) | x, y ∈ ℕ0 and x < Width and y < Height}

[0061] In an exemplary aspect, client sync application 424 calculates a first ratio of x over the width and a second ratio of y over the height such that: - ClientX = x / Width - ClientY = y / Height - ClientX, ClientY ∈ ℝ and 0 ≤ ClientX, ClientY < 1

[0062] More specifically, x is the horizontal cursor position inside a given area, y is the vertical cursor position inside the given area. In some aspects, when sending these ratios to server sync application 426, client sync application 424 multiplies the ratios by a predetermined amount (e.g., 224) and rounds the products to a lower integer. This normalization step is defined in ICP and the values may be three bytes long and encoded in big-endian byte sequence. For example: - ⌊x⌋ = floor(x) = rounds x to lower integer - dClientX = floor(224· ClientX) is the horizontal mouse data that is sent via ICP to Ionic Connect - dClientY = floor(224· ClientY) is the vertical mouse data that is sent via ICP - dClientX and dClientY are encoded using 24 bits to keep the data as precise as044617-00026 needed

[0063] A mouse click is an event captured within the client-side application area (e.g., within user interface 300), representing a button pressed or released. In some aspects, mouse 410 and / or mouse 432 may have up to three buttons: Button 0 (primary), Button 1 (auxiliary), and Button 2 (secondary). Each button has a button identifier (ID) and state (e.g., pressed, released).

[0064] Suppose that one of a local user and remote user presses and immediately releases a button on mouse 432. Client sync application 424 may detect a mouse click event and transmit a timestamp, the button ID, and state associated with the pressed button to server sync application 426. For example, when pressed, the button information may be [5 / 1 / 2023 22:10:11, Button 1, pressed]. When released, the button information may be [5 / 1 / 202322:10:51, Button 1, released]. This information suggests that Button 1 was pressed between 22:10:11 and 22:10:51.

[0065] Suppose that the remote user moves mouse 432. Client sync application 424 may detect a mouse wheel event and transmit a timestamp, and movement information to server sync application 426.

[0066] In one aspect, server sync application 426 receives data from the client sync application 424 and is configured to parse the data coming in ICP format and process the data as it is defined in ICP. For example, in response to receiving information about an event from remote computing device 420, server sync application 426 translates the information to a Connect Cable Protocol (CCP), which transfers the information to computing device 403 via cable 406. More specifically, because cable 406 is recognized as a secondary mouse and keyboard for computing device 403, cable 406 performs the command (e.g., mouse movement, mouse click, keyboard selection, etc.) on computing device 403.

[0067] In some aspects, the events recognized by client sync application 424 may be divided into event types including: (A) keyboard, which is an event type indicating a press or release of a key on keyboard 434, (B) mouse wheel, which may be a scrolling event type indicative of movement of a vertical wheel for mouse 432 or a horizontal wheel generally implemented in touchpads, (C) mouse click, which is an event type used to indicate the primary, secondary, and / or tertiary button press and / or release, and (D) mouse move, which may be an event type indicative of either an absolute move or a relative move.044617-00026

[0068] In one aspect, for absolute movement, client sync application 424 transmits the exact position of the cursor to server sync application 426. More specifically, the cursor position data sent by client sync application 424 is transposed into a 216x 216matrix. Cable 406 emulates this “absolute mouse” and expects the data to come in 16 bits for x (i.e., horizontal direction), and 16 bits for y (i.e., vertical direction). Accordingly, the x and y represent the absolute coordinates of the cursor 314 on user interface 300.

[0069] More specifically: - TargetArea = {(x, y) | x, y ∈ ℕ0 and 0 ≤ x, y < 216}, where area is the set of cursor coordinates - dClientX and dClientY are the client-side data about the user mouse point coordinates. It is possible to retrieve the ClientX and ClientY information by undoing the normalization step used by ICP - TargetX = floor(dClientX 16 −8224· 2 ) = floor( dClientX · 2 ) - TargetY = floor(dClientY 16 −8224· 2 ) = floor(dClientY · 2 ), where TargetX and TargetY are, the horizontal and vertical directions in the targetcomputer (e.g., user interface 300) - (floor( dClientX · 2−8), floor(dClientY · 2−8)) ∈ TargetArea

[0070] In one aspect, for relative movement, client sync application 424 transmits the exact position of the cursor to server sync application 426. A set of variables influence the calculation of relative move, and the main goal is to calculate a cursor projection using the data sent by the client sync application 424 so that the cursors associated with the desktop environment within the computing device 403 and cursors associated with the desktop environment within the computing device 420 remain synchronized.

[0071] In one aspect, the cursor within the computing device 403 moves inside a desktop environment. The desktop environment of computing device 403 has a width and a height directly related to the monitor 402 and the operating system (OS) configuration.

[0072] Width, Height ∈ ℕ0 and Width, Height > 0, where Width and Height are the desktop dimensions measured in pixels

[0073] In a typical OS there may be a feature called scale, which affects the width and the height. For example:044617-00026 Scale ∈ ℝ and Scale > 0 and Scale ≤ Width and Scale ≤ Height The TargetArea in the relative mouse move context is the desktop resolution, and can be described as: TargetArea = floor(Width / Scale) x floor(Height / Scale) TargetArea = { (x, y) | x, y ∈ ℕ0 and x < floor(Width / Scale) and y < floor(Height / Scale)}, where the TargetArea is a set of points inside the desktop, where each point is a pixel.

[0074] In the relative movement case, server sync application 426 sets the initial mouse pointer position as (0,0), which is the top left corner of the screen. This setting is required because subsequent projections will take into account the last known position of the cursor. Consider the following:

[0075] dClientXn is the current horizontal data received by server sync application 426 from the client-side and dClientXn-1means the previous horizontal data received.

[0076] dClientYnis the current vertical data received by server sync application 426 from the client-side and dClientYn-1 means the previous vertical data received. In this context there is a variable called x that represents “how far” the target mouse pointer is, in the horizontal direction, from the desktop coordinate origin. A similar idea applies to y, it represents “how far” it is, in the vertical direction, from the desktop coordinate origin.

[0077] xn is the current calculation of x and xn-1 is the previous calculation of x

[0078] ynis the current calculation of y and yn-1is the previous calculation of y

[0079] Naming variables for this context:

[0080] - UserX and UserY are, respectively, the horizontal and the vertical coordinates of the user mouse pointer inside ClientArea

[0081] - ClientWidth and ClientHeight are the dimensiosn of the ClientArea. ^^lientWidth stands for the horizontal direction and ClientHeight stands for the vertical direction.

[0082] – TargetX and TargetY are respectively, the horizontal and the vertical coordinates of the target mouse pointer inside TargetArea

[0083] - TargetWidth and TargetHeight are, the dimensions of the TargetArea. TargetWidth stands for the horizontal direction and TargetHeight stands for the vertical direction.

[0084] It cannot be assumed that the client-side width and height are the same as the target computer width and height, therefore it is not correct to say that the same amount of044617-00026 pixels that was moved in the client-side application must be the same amount of pixels to be moved in the target computer. That said, a way to find the correct amount to move in the horizontal direction and in the vertical direction is to find the proportion between the client- side and the target computer. Given that server sync application 426 receives the following data: – dClientX = floor(224· ClientX) – dClientY = floor(224· ClientY)

[0085] It can be extrapolated that ClientX ≈ dClientX / 224and ClientY ≈ dClientY / 224where ClientX = UserX / ClientWidth and ClientY = UserY / ClientHeight and TargetX and TargetY that are proportional to UserX and UserY need to be determined. In order to fix some inaccuracies caused by the floor function and the normalization step, a rounding function that rounds to the nearest integer may be applied: - TargetX ≈ round (dClientX / 224· floor(TargetWidth · Scale-1)) - TargetY ≈ round (dClientY / 224· floor(TargetHeight · Scale-1)) - xn≈ round (dClientXn / 224· floor(TargetWidth · Scale-1)) - yn≈ round (dClientYn / 224· floor(TargetHeight · Scale-1))

[0086] xn and yn are, respectively, the current calculation of the TargetX and TargetY, and will, approximately, be a good projection for the target mouse pointer in the horizontal and in the vertical direction. The round function for the calculation of xnworks well for all the values in the interval of [0, floor(TargetWidth · Scale-1)], and the same is true for the calculation of ynfor all the values in the interval of [0, floor(TargetHeight · Scale-1)], but whenever the value of xn = floor(TargetWidth · Scale-1) or yn = floor(TargetHeight · Scale-1) due to the round function, xn or yn will not satisfy the restriction of xn , yn ∈ TargetArea, so in order to fix this round issue: - if xn = floor(TargetWidth · Scale-1), then xn = floor(TargetWidth · Scale-1) – 1 - if yn = floor(TargetHeight · Scale-1), then yn = floor(TargetWidth · Scale-1) - 1 The relative move will be the difference between the latest calculation of x and y and their previous calculation: - Δx = xn – xn-1 - Δy = yn – yn-1044617-00026 - Δx and Δy respectively represent the horizontal and vertical relative movements of the cursor It should be noted that whenever the client-side application sends a mouse move command, that is directly related to the user mouse moving around the ClientArea, a new Δx and Δy will be calculated, and each projection of the target mouse pointer position can be enumerated into a sequence: x1, x2, …, xn-1, xn y1, y2, …, yn-1, yn x0= 0 and y0= 0

[0087] When the user moves the mouse for the first time and the client-side sends two commands, the mouse sync command and the mouse move command containing the information about the position of the user mouse, the following calculation occurs: - x1≈ round (dClientX1 / 224· floor(TargetWidth · Scale-1)) - y1 ≈ round (dClientY1 / 224· floor(TargetHeight · Scale-1))

[0088] Note the notation dClientX1 and dClientY1 are indicating that this is the first arrived data. Furthermore x1 and y1 notations indicate that this is the first target mouse pointer projection. The relative move will be: Δx = x1 – x0 = x1 and Δy = y1 – y0 = y1

[0089] When the client-side sends the second mouse move command, the next values for x and y will be: - X2 ≈ round (dClientX2 / 224· floor(TargetWidth · Scale-1)) - Y2 ≈ round (dClientY2 / 224· floor(TargetHeight · Scale-1)) and the relative move will be : Δx = x2 – x1 and Δy = y2 – y1. By every interaction in the system, n in the “sub n“ notation is incremented, where n ∈ ℕ0. ∆x and ∆y respectively represent the horizontal and vertical relative movements that the target mouse pointer needs to perform in order to be synchronized with the user mouse pointer.

[0090] For most of the operating systems a feature to scale up or down the mouse move is implemented. This feature is called mouse velocity. A mouse relative move can be described as a vector m such that m ∈ {(x, y) | x, y ∈ ℤ}; - Velocity ∈ ℝ and Velocity > 0044617-00026 - ^^(m) = m ⊙ Velocity = floor(Velocity) · x), floor(Velocity) · y)) The ⊙ notation indicates a vector multiplication that is not usual, it caused by the floor function since the OS does not perform real numbers values, only integer numbers values. For example, move = (∆x, ∆y), ∆x = 23, ∆y =− 42 Velocity = 1.25 V(move) = Velocity ⊙ (∆x, ∆y) = 1.25 ⊙ (23, − 42) = (floor(23 · 1.25), floor(− 42 · 1.25)) V(move) = (28, − 52)

[0091] Here, V(m) is a general function for how the operating systems handles a relative move when the feature of velocity is active. m is a relative move that goes into the function, and Velocity a real number greater than zero.

[0092] Taking that into account, server sync application 426 redefines the equation for a relative move in a way to “cancel” the OS mouse velocity within computing device 403 and fix impressions due to this calculation. Consider two more variables to handle the implications caused by this new calculation: RemnantsX, RemnantsY ∈ ℝ

[0093] The RemnantsX is an “accumulator” that represents a variable storing the sum of the decimal part of the current and all the previous calculations of^^ ^^ – ^^ ^^−1^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ . For example: RemnantsX0= 0 ^^ RemnantsXn = RemnantsX^^– ^^^^−1n-1 +^^ ^^ – ^^ ^^−1^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ – floor ( ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ )RemnantsY with its respective calculation of^^ ^^ – ^^ ^^−1^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^. For example:RemnantsY0 = 0 ^^ RemnantsYn = RemnantsY^^– ^^^^−1n-1 +^^ ^^ – ^^ ^^−1^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ – floor ( ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ )

[0095] The process of calculating the relative move involves:

[0096] If abs(RemnantsXn) ≥ 1, then Δx = floor (^^ ^^ – ^^ ^^−1^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ) + floor(RemnantsXn), where abs(B) returns the absolute value of B044617-00026

[0097] If abs(RemnantsYn) ≥ 1, then Δy = floor (^^ ^^ – ^^ ^^−1^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ) + floor(RemnantsYn),

[0098] If abs(RemnantsXn) < 1, then Δx =^^ ^^ – ^^

[0099] If abs(RemnantsYn) < 1, then Δy =

[0100] whenever the RemnantsX sum up or equal to one, it mustsubtract from itself its integer part:

[0101] If RemnantsX ≥ 1, then newRemnantsX = RemnantsX – [RemnantsX].

[0102] It should be noted that whenever abs(RemnantsXn) ≥ 1 the information of its integer part will be in Δx, and the same is true for Δy with RemnantsYn. Due to this reason it is important to “remove” the duplicated information that remains in RemnantsXn whenever abs(RemnantsXn) ≥ 1, and remove the duplicated information that remains in RemnantsYnwhenever abs(RemnantsYn) ≥ 1.

[0103] If abs(RemnantsXn) ≥ 1, then RemnantsXn= RemnantsXn– floor(RemnantsXn), it is read that “the variable RemnantsXnwill receive the value of RemnantsXnminus floor(RemnantsXn).” For example, suppose that Velocity = 2.

[0104] Step 1: x1 = 1

[0105] Step 2: RemnantsX1 = RemnantsX0 +^^1 – ^^0 ^^1 – ^^0^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ - floor ( ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^) = 0 + ½ + 0 = ½

[0106] Step 3: Δx = floor (1 / 2) = 0

[0107] Step 4: x2= 2

[0108] Step 5: RemnantsX2 = RemnantsX1 +^^2 – ^^1 ^^2 – ^^1^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ - floor ( ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^) = ½ + ½ + 0 = 1

[0109] Step 6: Δx = floor (^^2 – ^^1^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^) + floor(RemnantsX ) =

[0110] Step 7: RemnantsX2 = 1 – floor(1) = 0

[0111] Step 8: x3= 3

[0112] Step 9 = RemnantsX3= RemnantsX2+^^3 – ^^2 ^^3 – ^^2^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ - floor ( ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^) = 0 + ½ + 0 = ½

[0113] Step 10 = Δx = floor^^3 – ^^2^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^) = floor(1 / 2) = 0

[0114] IfRemnantsYn= RemnantsYn– floor(RemnantsYn), it is read that “the variable RemnantsYnwill receive the value of RemnantsYnminus floor(RemnantsXn).”

[0115] These remnants help server sync application 426 autocorrect the relative move over time and keep the target mouse pointer synchronized with the user mouse pointer.044617-00026

[0116] It should be noted that relative moves are sent from server sync application 426 using a fixed amount of bits r: r ∈ ℕ0 and 1 < r ≤ 32 for Δx and Δy. This amount of bits is limited to the USB HID class since the connect cable uses this protocol to send the relative mouse data. The relative move must be broken down into smaller pieces that fall within the range of [- (2r-1) -1, (2r-1) -1)]. This is the range related to the maximum absolute value that can be represented by r bits and still keep a symmetry for negative values.

[0117] Because this division can cause sharp curves and abrupt changes in direction, client sync application 424 applies a pan smoothing algorithm that adjusts the original path of the cursor to reduce the number of sharp curves and abrupt changes in direction. The objectives of this algorithm are to be performant, so the remote mouse feels as real-time as possible, and avoid abrupt changes in directions, creating an “organic” movement.

[0118] In some aspects, client sync application 424 applies the pan smoothing algorithm when Δx ∉ Range or Δy ∉ Range.

[0119] FIG.5 illustrates a flow diagram of a method 500 for pan smoothing. At 502, server sync application 426 determines whether Δx = Δy = 0. If yes, no data is transmitted and the method ends. If no, method 500 advances to 504, where server sync application 426 creates a list of moves including the relative move for the horizontal and vertical directions. The list includes n elements, where each element is an ordered pair represented by the value of x and y: [(x1, y1), (x2, y2), …, (xn, yn)] and n = floor(max(|Δx|, |Δy|) / (2r-1-1)] + ceil((max(|Δx|, |Δy|)mod(2r-1) -1) / (2r-1-1)), where ceil(B) rounds B to an upper integer and max(a,b) = a, if a>b, max(a,b) = b, if b>a, and and max(a, a) = a.

[0120] At 506, server sync application 426 determines whether |Δx| = |Δy|. If |Δx| = |Δy| is not true, method 500 advances to 508, where server sync application 426 determines whether Max(|Δx|, |Δy|) = |Δx| or |Δy|. If Max(|Δx|, |Δy|) = |Δx|, method 500 advances to 510, where server sync application 426 populates the first n − 1 elements of the list with ((2r-1-1) · sign(Δx), floor(|Δy| / n) · sign(Δy)) and populates the nth element of the list with ((|Δx| mod (2r-1-1)) · sign(Δx), (|Δy| - floor(|Δy| / n) · (n-1)) · sign(Δy)). From here, method 500 advances to 512, where server sync application 426 transmits the list of moves to client sync application 424.

[0121] If at 508 server sync application 426 determines that Max(|Δx|, |Δy|) does not equal |Δx|, method 500 advances to 514, where server sync application 426 determines044617-00026 whether Max(|Δx|, |Δy|) = |Δy|. When true, method 500 advances to 516, where server sync application 426 populates the first n − 1 elements of the list with (floor(|Δx| / n) · sign(Δx), ((2r-1-1) · sign(Δy)) and populates the nth element of the list with ((|Δx| - floor(|Δx| / n) · (n- 1)) · sign(Δx), (|Δy| mod (2r-1-1)) · sign(Δy)). From 516, method 500 advances to 512.

[0122] If at 506 server sync application 426 determines that |Δx| = |Δy|, method 500 advances to 518, where server sync application 426 populates the first n − 1 elements of the list with ((2r-1-1) · sign(Δx), (2r-1-1) · sign(Δy)) and populates nth element with (|Δx| mod (2r-1-1)) · sign(Δx), (|Δy| mod (2r-1-1)) · sign(Δy)). From 518, method 500 advances to 512.

[0123] FIG.6 illustrates a flow diagram of a method 600 for mouse synchronization when provisioning remote operation of medical equipment. At 602, server sync application 426 receives, from a remote computing device, a first indication of a first position of a cursor on a user interface, wherein the first position is a first ratio of a horizontal cursor position X1over a width of the user interface and a second ratio of a vertical cursor position Y1over a height of the user interface.

[0124] At 604, server sync application 426 subsequently receives, from the remote computing device, a second indication of a second position of the cursor on the user interface, wherein the second position is a third ratio of a new horizontal cursor position Xn over a width of the user interface and a fourth ratio of a new vertical cursor position Yn over a height of the user interface.

[0125] At 606, server sync application 426 calculates a horizontal difference value between the first ratio and the third ratio and a vertical difference value between the second ratio and the fourth ratio, wherein the horizontal difference value is a horizontal relative movement of the cursor and the vertical difference value is a vertical relative movement of the cursor.

[0126] At 608, server sync application 426 determines that the horizontal difference value and the vertical difference value are greater than a predetermined size.

[0127] At 610, server sync application 426 divides the at least one of the horizontal difference value and the vertical difference value into a plurality of movement values.

[0128] At 612, server sync application 426 applies a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the first position to the second position.044617-00026

[0129] At 614, server sync application 426 transmits the plurality of movement values to a local computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

[0130] In some aspects, prior to determining the horizontal difference value and the vertical difference value, server sync application 426 applies a scaling factor to the first ratio and the third ratio and applies a different scaling factor to the second ratio and the fourth ratio. In some aspects, the scaling factor is ratio of the width over a product of a predetermined scale and 224, and the different scaling factor is a ratio of the height over a product of the predetermined scale and 224.

[0131] In some aspects, server sync application 426 generates a corrected horizontal difference value and a corrected vertical difference value by applying a correction factor accounting for a mouse velocity to both the horizontal difference value and the vertical difference value. Server sync application 426 adds a first remnants value to the corrected horizontal difference value, wherein the first remnants value stores a first sum of a decimal portions of each horizontal cursor position that is rounded to an integer. Server sync application 426 adds a second remnants value to the corrected vertical difference value, wherein the second remnants value stores a second sum of a decimal portions of each vertical cursor position that is rounded to an integer. In some aspects, the first remnants value and second remnants value each subtract their integer portion when greater than or equal to one.

[0132] In some aspects, the plurality of movements values includes n values, the horizontal difference value is Δx, the vertical difference value is Δy, sign(x) = 1, if x > 0, sign(x) = 0, if x = 0, and sign(x) = -1, if x < 0, floor(x) rounds x to a lower integer, and wherein applying the pan smoothing adjustment value comprises server sync application 426 determining that the horizontal difference value is greater than the vertical difference value, populating a first n-1 elements of the plurality of movement values with ((2r-1-1) · sign(Δx), floor(|Δy| / n) · sign(Δy)), and populating an nth element of the plurality of movement values with ((|Δx| mod (2r-1-1)) · sign(Δx), (|Δy| - floor(|Δy| / n) · (n-1) · sign(Δy)).

[0133] In some aspects, the plurality of movements values includes n values, the horizontal difference value is Δx, the vertical difference value is Δy, sign(x) = 1, if x > 0, sign(x) = 0, if x = 0, and sign(x) = -1, if x < 0, floor(x) rounds x to a lower integer, and wherein applying the pan smoothing adjustment value comprises server sync application 426 determining that044617-00026 the vertical difference value is greater than the horizontal difference value, populating a first n-1 elements of the plurality of movement values with (floor(|Δx| / n) · sign(Δx), ((2r-1-1) · sign(Δy)), and populating an nth element of the plurality of movement values with (floor(|Δx| / n) · sign(Δx), ((2r-1-1) · sign(Δy)).

[0134] In some aspects, server sync application 426 receives, from the remote computing device, a third indication of a mouse click, wherein the third indication comprises an identifier of a button clicked on a mouse connected to the remote computing device, a timestamp, and a state of the mouse, and transmits the third indication to the local computing device that is configured to replicate the mouse click.

[0135] In some aspects, server sync application 426 receives, from the remote computing device, a third indication of a keyboard selection, wherein the third indication comprises an identifier of a key clicked on a keyboard connected to the remote computing device and a timestamp, and transmits the third indication to the local computing device that is configured to replicate the keyboard selection.

[0136] FIG.7 illustrates a flow diagram of method 700 for mouse synchronization when provisioning remote operation of medical equipment using an origin position. At 702, a second computing device (e.g., computing device 404) with a cable (e.g., cable 406) emulating a mouse device establishes a connection with a first computing device (e.g., computing device 403). In some aspects, the cable emulates the mouse device using a USB HID protocol.

[0137] At 704, the second computing device executes a request that shifts a cursor within a desktop environment of the first computing device to an origin position. In some aspects, the origin position is a zero position along a horizontal axis and a zero position along a vertical axis.

[0138] At 706, the second computing device receives, from a remote computing device (e.g., remote computing device 420), a first indication of a first position of a cursor on a user interface. Here, the first position is a first ratio of a horizontal cursor position X1 over a width of the user interface and a second ratio of a vertical cursor position Y1 over a height of the user interface.

[0139] At 708, the second computing device determines a projected first position by calculating a horizontal projection and a vertical projection of the first position using a desktop specification of the first computing device. In some aspects, the second computing044617-00026 device calculates the horizontal projection by applying a scaling factor that is a ratio of the width over a predetermined scale, and calculates the vertical projection by applying a scaling factor that is a ratio of the height over the predetermined scale.

[0140] At 710, the second computing device determines a plurality of moves between the origin position and the projected first position.

[0141] At 712, the second computing device divides at least one of a horizontal difference value and a vertical difference value into a plurality of movement values based on an amount of the plurality of moves. In some aspects, the horizontal difference value is a horizontal relative movement of the cursor from the origin position and the projected first position and the vertical difference value is a vertical relative movement of the cursor from the origin position and the projected first position.

[0142] At 714, the second computing device applies a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the origin position to the projected first position.

[0143] At 716, the second computing device transmits the plurality of movement values to the first computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

[0144] FIG. 8 is a block diagram illustrating a computer system 20 on which aspects of systems and methods for mouse and keyboard synchronization when provisioning remote operation of medical equipment may be implemented in accordance with an exemplary aspect. The computer system 20 can be in the form of multiple computing devices, or in the form of a single computing device (e.g., computing device 206, remote computing device 216, etc.), for example, a desktop computer, a notebook computer, a laptop computer, a mobile computing device, a smart phone, a tablet computer, a server, a mainframe, an embedded device, and other forms of computing devices.

[0145] As shown, the computer system 20 includes a central processing unit (CPU) 21, a system memory 22, and a system bus 23 connecting the various system components, including the memory associated with the central processing unit 21. The system bus 23 may comprise a bus memory or bus memory controller, a peripheral bus, and a local bus that is able to interact with any other bus architecture. Examples of the buses may include PCI, ISA, PCI-Express, HyperTransport™, InfiniBand™, Serial ATA, I2C, and other suitable interconnects.044617-00026 The central processing unit 21 (also referred to as a processor) can include a single or multiple sets of processors having single or multiple cores. The processor 21 may execute one or more computer-executable code implementing the techniques of the present disclosure. For example, any of commands / steps discussed in FIGS.1-7 may be performed by processor 21. The system memory 22 may be any memory for storing data used herein and / or computer programs that are executable by the processor 21. The system memory 22 may include volatile memory such as a random access memory (RAM) 25 and non-volatile memory such as a read only memory (ROM) 24, flash memory, etc., or any combination thereof. The basic input / output system (BIOS) 26 may store the basic procedures for transfer of information between elements of the computer system 20, such as those at the time of loading the operating system with the use of the ROM 24.

[0146] The computer system 20 may include one or more storage devices such as one or more removable storage devices 27, one or more non-removable storage devices 28, or a combination thereof. The one or more removable storage devices 27 and non-removable storage devices 28 are connected to the system bus 23 via a storage interface 32. In an aspect, the storage devices and the corresponding computer-readable storage media are power- independent modules for the storage of computer instructions, data structures, program modules, and other data of the computer system 20. The system memory 22, removable storage devices 27, and non-removable storage devices 28 may use a variety of computer- readable storage media. Examples of computer-readable storage media include machine memory such as cache, SRAM, DRAM, zero capacitor RAM, twin transistor RAM, eDRAM, EDO RAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM; flash memory or other memory technology such as in solid state drives (SSDs) or flash drives; magnetic cassettes, magnetic tape, and magnetic disk storage such as in hard disk drives or floppy disks; optical storage such as in compact disks (CD-ROM) or digital versatile disks (DVDs); and any other medium which may be used to store the desired data and which can be accessed by the computer system 20.

[0147] The system memory 22, removable storage devices 27, and non-removable storage devices 28 of the computer system 20 may be used to store an operating system 35, additional program applications 37, other program modules 38, and program data 39. The computer system 20 may include a peripheral interface 46 for communicating data from input044617-00026 devices 40, such as a keyboard, mouse, stylus, game controller, voice input device, touch input device, or other peripheral devices, such as a printer or scanner via one or more I / O ports, such as a serial port, a parallel port, a universal serial bus (USB), or other peripheral interface. A display device 47 such as one or more monitors, projectors, or integrated display, may also be connected to the system bus 23 across an output interface 48, such as a video adapter. In addition to the display devices 47, the computer system 20 may be equipped with other peripheral output devices (not shown), such as loudspeakers and other audiovisual devices.

[0148] The computer system 20 may operate in a network environment, using a network connection to one or more remote computers 49. The remote computer (or computers) 49 may be local computer workstations or servers comprising most or all of the aforementioned elements in describing the nature of a computer system 20. Other devices may also be present in the computer network, such as, but not limited to, routers, network stations, peer devices or other network nodes. The computer system 20 may include one or more network interfaces 51 or network adapters for communicating with the remote computers 49 via one or more networks such as a local-area computer network (LAN) 50, a wide-area computer network (WAN), an intranet, and the Internet. Examples of the network interface 51 may include an Ethernet interface, a Frame Relay interface, SONET interface, and wireless interfaces.

[0149] Aspects of the present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0150] The computer readable storage medium can be a tangible device that can retain and store program code in the form of instructions or data structures that can be accessed by a processor of a computing device, such as the computing system 20. The computer readable storage medium may be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. By way of example, such computer-readable storage medium can comprise a random access memory (RAM), a read-only memory (ROM), EEPROM, a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), flash memory, a hard disk, a portable computer diskette, a memory stick, a floppy disk, or even a044617-00026 mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon. As used herein, a computer readable storage medium is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or transmission media, or electrical signals transmitted through a wire.

[0151] Computer readable program instructions described herein can be downloaded to respective computing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network interface in each computing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing device.

[0152] Computer readable program instructions for carrying out operations of the present disclosure may be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language, and conventional procedural programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a LAN or WAN, or the connection may be made to an external computer (for example, through the Internet). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0153] In various aspects, the systems and methods described in the present disclosure044617-00026 can be addressed in terms of modules. The term "module" as used herein refers to a real- world device, component, or arrangement of components implemented using hardware, such as by an application specific integrated circuit (ASIC) or FPGA, for example, or as a combination of hardware and software, such as by a microprocessor system and a set of instructions to implement the module’s functionality, which (while being executed) transform the microprocessor system into a special-purpose device. A module may also be implemented as a combination of the two, with certain functions facilitated by hardware alone, and other functions facilitated by a combination of hardware and software. In certain implementations, at least a portion, and in some cases, all, of a module may be executed on the processor of a computer system. Accordingly, each module may be realized in a variety of suitable configurations, and should not be limited to any particular implementation exemplified herein.

[0154] In the interest of clarity, not all of the routine features of the aspects are disclosed herein. It would be appreciated that in the development of any actual implementation of the present disclosure, numerous implementation-specific decisions must be made in order to achieve the developer’s specific goals, and these specific goals will vary for different implementations and different developers. It is understood that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skill in the art, having the benefit of this disclosure.

[0155] Furthermore, it is to be understood that the phraseology or terminology used herein is for the purpose of description and not of restriction, such that the terminology or phraseology of the present specification is to be interpreted by the skilled in the art in light of the teachings and guidance presented herein, in combination with the knowledge of those skilled in the relevant art(s). Moreover, it is not intended for any term in the specification or claims to be ascribed an uncommon or special meaning unless explicitly set forth as such.

[0156] Additional aspects of the present disclosure may be implemented according to one or more of the following clauses.

[0157] Clause 1. A method for mouse synchronization, comprising: receiving, from a remote computing device, a first indication of a first position of a cursor on a user interface, wherein the first position is a first ratio of a horizontal cursor position X1over a width of the user interface and a second ratio of a vertical cursor position Y1 over a height of the user044617-00026 interface; subsequently receiving, from the remote computing device, a second indication of a second position of the cursor on the user interface, wherein the second position is a third ratio of a new horizontal cursor position Xn over a width of the user interface and a fourth ratio of a new vertical cursor position Ynover a height of the user interface; calculating a horizontal difference value between the first ratio and the third ratio and a vertical difference value between the second ratio and the fourth ratio, wherein the horizontal difference value is a horizontal relative movement of the cursor and the vertical difference value is a vertical relative movement of the cursor; determining whether the horizontal difference value and the vertical difference value are greater than a predetermined size; in response to determining that at least one of the horizontal difference value and the vertical difference value is greater than a predetermined size, dividing the at least one of the horizontal difference value and the vertical difference value into a plurality of movement values; applying a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the first position to the second position; and transmitting the plurality of movement values to a local computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

[0158] Clause 2. The method of clause 1, further comprising prior to determining the horizontal difference value and the vertical difference value: applying a scaling factor to the first ratio and the third ratio; and applying a different scaling factor to the second ratio and the fourth ratio.

[0159] Clause 3. The method of any of the preceding clauses, wherein the scaling factor is ratio of the width over a product of a predetermined scale, and wherein the different scaling factor is a ratio of the height over a product of the predetermined scale.

[0160] Clause 4. The method of any of the preceding clauses, further comprising: generating a corrected horizontal difference value and a corrected vertical difference value by applying a correction factor accounting for a mouse velocity to both the horizontal difference value and the vertical difference value; adding a first remnants value to the corrected horizontal difference value, wherein the first remnants value stores a first sum of a decimal portions of each horizontal cursor position that is rounded to an integer; and adding a second remnants value to the corrected vertical difference value, wherein the second044617-00026 remnants value stores a second sum of a decimal portions of each vertical cursor position that is rounded to an integer.

[0161] Clause 5. The method of any of the preceding clauses, wherein the first remnants value and second remnants value each subtract their integer portion when greater than or equal to one.

[0162] Clause 6. The method of any of the preceding clauses, wherein the plurality of movements values includes n values, the horizontal difference value is Δx, the vertical difference value is Δy, sign(x) = 1, if x > 0, sign(x) = 0, if x = 0, and sign(x) = -1, if x < 0, floor(x) rounds x to a lower integer, and wherein applying the pan smoothing adjustment value comprises: determining that the horizontal difference value is greater than the vertical difference value; populating a first n-1 elements of the plurality of movement values with ((2r-1-1) · sign(Δx), floor(|Δy| / n) · sign(Δy)); and populating an nth element of the plurality of movement values with ((|Δx| mod (2r-1-1)) · sign(Δx), (|Δy| - floor(|Δy| / n) · (n-1) · sign(Δy)).

[0163] Clause 7. The method of any of the preceding clauses, wherein the plurality of movements values includes n values, the horizontal difference value is Δx, the vertical difference value is Δy, sign(x) = 1, if x > 0, sign(x) = 0, if x = 0, and sign(x) = -1, if x < 0, floor(x) rounds x to a lower integer, and wherein applying the pan smoothing adjustment value comprises: determining that the vertical difference value is greater than the horizontal difference value; populating a first n-1 elements of the plurality of movement values with (floor(|Δx| / n) · sign(Δx), ((2r-1-1) · sign(Δy)); and populating an nth element of the plurality of movement values with ((| ^^ ^^| − ^^ ^^ ^^ ^^ ^^ (| ^^ ^^| ^^−1^^) ⋅ ( ^^ − 1)) ⋅ ^^ ^^ ^^ ^^( ^^ ^^), (| ^^ ^^| ^^ ^^ ^^ (2 −receiving, from the remote computing device, a third indication of a mouse click, wherein the third indication comprises an identifier of a button clicked on a mouse connected to the remote computing device, a timestamp, and a state of the mouse; and transmitting the third indication to the local computing device that is configured to replicate the mouse click.

[0165] Clause 9. The method of any of the preceding clauses, further comprising: receiving, from the remote computing device, a third indication of a keyboard selection, wherein the third indication comprises an identifier of a key clicked on a keyboard connected044617-00026 to the remote computing device and a timestamp; and transmitting the third indication to the local computing device that is configured to replicate the keyboard selection.

[0166] Clause 10. A system for mouse synchronization, comprising: at least one memory; at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to: receive, from a remote computing device, a first indication of a first position of a cursor on a user interface, wherein the first position is a first ratio of a horizontal cursor position X1over a width of the user interface and a second ratio of a vertical cursor position Y1 over a height of the user interface; subsequently receive, from the remote computing device, a second indication of a second position of the cursor on the user interface, wherein the second position is a third ratio of a new horizontal cursor position Xnover a width of the user interface and a fourth ratio of a new vertical cursor position Yn over a height of the user interface; calculate a horizontal difference value between the first ratio and the third ratio and a vertical difference value between the second ratio and the fourth ratio, wherein the horizontal difference value is a horizontal relative movement of the cursor and the vertical difference value is a vertical relative movement of the cursor; determine whether the horizontal difference value and the vertical difference value are greater than a predetermined size; in response to determining that at least one of the horizontal difference value and the vertical difference value is greater than a predetermined size, divide the at least one of the horizontal difference value and the vertical difference value into a plurality of movement values; apply a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the first position to the second position; and transmit the plurality of movement values to a local computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

[0167] Clause 11. The system of any of the preceding clauses, further comprising prior to determining the horizontal difference value and the vertical difference value, the at least one hardware processor is configured to: apply a scaling factor to the first ratio and the third ratio; and apply a different scaling factor to the second ratio and the fourth ratio.

[0168] Clause 12. The system of any of the preceding clauses, wherein the scaling factor is ratio of the width over a product of a predetermined scale, and wherein the different scaling factor is a ratio of the height over a product of the predetermined scale.044617-00026

[0169] Clause 13. The system of any of the preceding clauses, wherein the at least one hardware processor is configured to: generate a corrected horizontal difference value and a corrected vertical difference value by applying a correction factor accounting for a mouse velocity to both the horizontal difference value and the vertical difference value; add a first remnants value to the corrected horizontal difference value, wherein the first remnants value stores a first sum of a decimal portions of each horizontal cursor position that is rounded to an integer; and add a second remnants value to the corrected vertical difference value, wherein the second remnants value stores a second sum of a decimal portions of each vertical cursor position that is rounded to an integer.

[0170] Clause 14. The system of any of the preceding clauses, wherein the first remnants value and second remnants value each subtract their integer portion when greater than or equal to one.

[0171] Clause 15. The system of any of the preceding clauses, wherein the plurality of movements values includes n values, the horizontal difference value is Δx, the vertical difference value is Δy, sign(x) = 1, if x > 0, sign(x) = 0, if x = 0, and sign(x) = -1, if x < 0, floor(x) rounds x to a lower integer, and wherein the at least one hardware processor is configured to apply the pan smoothing adjustment value by: determining that the horizontal difference value is greater than the vertical difference value; populating a first n-1 elements of the plurality of movement values with ((2r-1-1) · sign(Δx), floor(|Δy| / n) · sign(Δy)); and populating an nth element of the plurality of movement values with ((|Δx| mod (2r-1-1)) · sign(Δx), (|Δy| - floor(|Δy| / n) · (n-1) · sign(Δy)).

[0172] Clause 16. The system of any of the preceding clauses, wherein the plurality of movements values includes n values, the horizontal difference value is Δx, the vertical difference value is Δy, sign(x) = 1, if x > 0, sign(x) = 0, if x = 0, and sign(x) = -1, if x < 0, floor(x) rounds x to a lower integer, and wherein the at least one hardware processor is configured to apply the pan smoothing adjustment value by: determining that the vertical difference value is greater than the horizontal difference value; populating a first n-1 elements of the plurality of movement values with (floor(|Δx| / n) · sign(Δx), ((2r-1-1) · sign(Δy)); and populating an nth element of the plurality of movement values with ((| ^^ ^^| − ^^ ^^ ^^ ^^ ^^ (| ^^ ^^|^^) ⋅( ^^ − 1)) ⋅ ^^ ^^ ^^ ^^( ^^ ^^), (| ^^ ^^| ^^ ^^ ^^ (2^^−1− 1)) ⋅ ^^ ^^ ^^ ^^( ^^ ^^)).044617-00026

[0173] Clause 17. The system of any of the preceding clauses, wherein the at least one hardware processor is configured to: receive, from the remote computing device, a third indication of a mouse click, wherein the third indication comprises an identifier of a button clicked on a mouse connected to the remote computing device, a timestamp, and a state of the mouse; and transmit the third indication to the local computing device that is configured to replicate the mouse click.

[0174] Clause 18. A method for mouse synchronization, comprising: establishing a connection between a first computing device and a second computing device that has a cable emulating a mouse device; executing a request that shifts a cursor within a desktop environment of the first computing device to an origin position; receiving, from a remote computing device by the second computing device, a first indication of a first position of a cursor on a user interface, wherein the first position is a first ratio of a horizontal cursor position X1 over a width of the user interface and a second ratio of a vertical cursor position Y1 over a height of the user interface; determining a projected first position by calculating a horizontal projection and a vertical projection of the first position using a desktop specification of the first computing device; determining a plurality of moves between the origin position and the projected first position; dividing at least one of a horizontal difference value and a vertical difference value into a plurality of movement values based on an amount of the plurality of moves, wherein the horizontal difference value is a horizontal relative movement of the cursor from the origin position and the projected first position and the vertical difference value is a vertical relative movement of the cursor from the origin position and the projected first position; applying a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the origin position to the projected first position; and transmitting the plurality of movement values to the first computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

[0175] Clause 19. The method of any of the preceding clauses, wherein the cable emulates the mouse device using a Universal Serial Bus (USB) Human Interface Device (HID) protocol.

[0176] Clause 20. The method of any of the preceding clauses wherein the origin position is a zero position along a horizontal axis and a zero position along a vertical axis.044617-00026

[0177] Clause 21. The method of any of the preceding clauses, wherein calculating the horizontal projection comprises applying a scaling factor that is a ratio of the width over a predetermined scale and wherein calculating the vertical projection comprises applying a scaling factor that is a ratio of the height over the predetermined scale.

[0178] The various aspects disclosed herein encompass present and future known equivalents to the known modules referred to herein by way of illustration. Moreover, while aspects and applications have been shown and described, it would be apparent to those skilled in the art having the benefit of this disclosure that many more modifications than mentioned above are possible without departing from the inventive concepts disclosed herein.

Claims

044617-00026 CLAIMS 1. A method for mouse synchronization, comprising: receiving, from a remote computing device, a first indication of a first position of a cursor on a user interface, wherein the first position is a first ratio of a horizontal cursor position X1over a width of the user interface and a second ratio of a vertical cursor position Y1over a height of the user interface; subsequently receiving, from the remote computing device, a second indication of a second position of the cursor on the user interface, wherein the second position is a third ratio of a new horizontal cursor position Xnover a width of the user interface and a fourth ratio of a new vertical cursor position Yn over a height of the user interface; calculating a horizontal difference value between the first ratio and the third ratio and a vertical difference value between the second ratio and the fourth ratio, wherein the horizontal difference value is a horizontal relative movement of the cursor and the vertical difference value is a vertical relative movement of the cursor; determining whether the horizontal difference value and the vertical difference value are greater than a predetermined size; in response to determining that at least one of the horizontal difference value and the vertical difference value is greater than a predetermined size, dividing the at least one of the horizontal difference value and the vertical difference value into a plurality of movement values; applying a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the first position to the second position; and transmitting the plurality of movement values to a local computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

2. The method of claim 1, further comprising prior to determining the horizontal difference value and the vertical difference value: applying a scaling factor to the first ratio and the third ratio; and applying a different scaling factor to the second ratio and the fourth ratio.044617-00026 3. The method of claim 2, wherein the scaling factor is ratio of the width over a product of a predetermined scale, and wherein the different scaling factor is a ratio of the height over a product of the predetermined scale.

4. The method of claim 1, further comprising: generating a corrected horizontal difference value and a corrected vertical difference value by applying a correction factor accounting for a mouse velocity to both the horizontal difference value and the vertical difference value; adding a first remnants value to the corrected horizontal difference value, wherein the first remnants value stores a first sum of a decimal portions of each horizontal cursor position that is rounded to an integer; and adding a second remnants value to the corrected vertical difference value, wherein the second remnants value stores a second sum of a decimal portions of each vertical cursor position that is rounded to an integer.

5. The method of claim 4, wherein the first remnants value and second remnants value each subtract their integer portion when greater than or equal to one.

6. The method of claim 1, wherein the plurality of movements values includes n values, the horizontal difference value is Δx, the vertical difference value is Δy, sign(x) = 1, if x > 0, sign(x) = 0, if x = 0, and sign(x) = -1, if x < 0, floor(x) rounds x to a lower integer, and wherein applying the pan smoothing adjustment value comprises: determining that the horizontal difference value is greater than the vertical difference value; populating a first n-1 elements of the plurality of movement values with ((2r-1-1) · sign(Δx), floor(|Δy| / n) · sign(Δy)); and populating an nth element of the plurality of movement values with ((|Δx| mod (2r-1- 1)) · sign(Δx), (|Δy| - floor(|Δy| / n) · (n-1) · sign(Δy)).

7. The method of claim 1, wherein the plurality of movements values includes n values, the horizontal difference value is Δx, the vertical difference value is Δy, sign(x) = 1, if x > 0,044617-00026 sign(x) = 0, if x = 0, and sign(x) = -1, if x < 0, floor(x) rounds x to a lower integer, and wherein applying the pan smoothing adjustment value comprises: determining that the vertical difference value is greater than the horizontal difference value; populating a first n-1 elements of the plurality of movement values with (floor(|Δx| / n) · sign(Δx), ((2r-1-1) · sign(Δy)); and populating an nth element of the plurality of movement values with ((|^^ ^^|− | ^^ ^^| ^^ ^^ ^^ ^^ ^^ ( ^^ ) ⋅ ( ^^− 1)) ⋅ ^^ ^^ ^^ ^^( ^^ ^^), (|^^ ^^|^^ ^^ ^^ (2^^−1− 1)) ⋅ ^^ ^^ ^^ ^^( ^^ ^^)).

8. The method of claim 1, further comprising: receiving, from the remote computing device, a third indication of a mouse click, wherein the third indication comprises an identifier of a button clicked on a mouse connected to the remote computing device, a timestamp, and a state of the mouse; and transmitting the third indication to the local computing device that is configured to replicate the mouse click.

9. The method of claim 1, further comprising: receiving, from the remote computing device, a third indication of a keyboard selection, wherein the third indication comprises an identifier of a key clicked on a keyboard connected to the remote computing device and a timestamp; and transmitting the third indication to the local computing device that is configured to replicate the keyboard selection.

10. A system for mouse synchronization, comprising: at least one memory; at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to: receive, from a remote computing device, a first indication of a first position of a cursor on a user interface, wherein the first position is a first ratio of a horizontal cursor044617-00026 position X1over a width of the user interface and a second ratio of a vertical cursor position Y1 over a height of the user interface; subsequently receive, from the remote computing device, a second indication of a second position of the cursor on the user interface, wherein the second position is a third ratio of a new horizontal cursor position Xn over a width of the user interface and a fourth ratio of a new vertical cursor position Yn over a height of the user interface; calculate a horizontal difference value between the first ratio and the third ratio and a vertical difference value between the second ratio and the fourth ratio, wherein the horizontal difference value is a horizontal relative movement of the cursor and the vertical difference value is a vertical relative movement of the cursor; determine whether the horizontal difference value and the vertical difference value are greater than a predetermined size; in response to determining that at least one of the horizontal difference value and the vertical difference value is greater than a predetermined size, divide the at least one of the horizontal difference value and the vertical difference value into a plurality of movement values; apply a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the first position to the second position; and transmit the plurality of movement values to a local computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

11. The system of claim 10, further comprising prior to determining the horizontal difference value and the vertical difference value, the at least one hardware processor is configured to: apply a scaling factor to the first ratio and the third ratio; and apply a different scaling factor to the second ratio and the fourth ratio.

12. The system of claim 11, wherein the scaling factor is ratio of the width over a product of a predetermined scale, and wherein the different scaling factor is a ratio of the height over a product of the predetermined scale.044617-00026 13. The system of claim 10, wherein the at least one hardware processor is configured to: generate a corrected horizontal difference value and a corrected vertical difference value by applying a correction factor accounting for a mouse velocity to both the horizontal difference value and the vertical difference value; add a first remnants value to the corrected horizontal difference value, wherein the first remnants value stores a first sum of a decimal portions of each horizontal cursor position that is rounded to an integer; and add a second remnants value to the corrected vertical difference value, wherein the second remnants value stores a second sum of a decimal portions of each vertical cursor position that is rounded to an integer.

14. The system of claim 13, wherein the first remnants value and second remnants value each subtract their integer portion when greater than or equal to one.

15. The system of claim 10, wherein the plurality of movements values includes n values, the horizontal difference value is Δx, the vertical difference value is Δy, sign(x) = 1, if x > 0, sign(x) = 0, if x = 0, and sign(x) = -1, if x < 0, floor(x) rounds x to a lower integer, and wherein the at least one hardware processor is configured to apply the pan smoothing adjustment value by: determining that the horizontal difference value is greater than the vertical difference value; populating a first n-1 elements of the plurality of movement values with ((2r-1-1) · sign(Δx), floor(|Δy| / n) · sign(Δy)); and populating an nth element of the plurality of movement values with ((|Δx| mod (2r-1- 1)) · sign(Δx), (|Δy| - floor(|Δy| / n) · (n-1) · sign(Δy)).

16. The system of claim 10, wherein the plurality of movements values includes n values, the horizontal difference value is Δx, the vertical difference value is Δy, sign(x) = 1, if x > 0, sign(x) = 0, if x = 0, and sign(x) = -1, if x < 0, floor(x) rounds x to a lower integer, and wherein044617-00026 the at least one hardware processor is configured to apply the pan smoothing adjustment value by: determining that the vertical difference value is greater than the horizontal difference value; populating a first n-1 elements of the plurality of movement values with (floor(|Δx| / n) · sign(Δx), ((2r-1-1) · sign(Δy)); and populating an nth element of the plurality of movement values with ((|^^ ^^|− | ^^ ^^| ^^ ^^ ^^ ^^ ^^ ( ^^ ) ⋅ ( ^^− 1)) ⋅ ^^ ^^ ^^ ^^( ^^ ^^), (|^^ ^^|^^ ^^ ^^ (2^^−1− 1)) ⋅ ^^ ^^ ^^ ^^( ^^ ^^)).

17. The system of claim 10, wherein the at least one hardware processor is configured to: receive, from the remote computing device, a third indication of a mouse click, wherein the third indication comprises an identifier of a button clicked on a mouse connected to the remote computing device, a timestamp, and a state of the mouse; and transmit the third indication to the local computing device that is configured to replicate the mouse click.

18. A method for mouse synchronization, comprising: establishing a connection between a first computing device and a second computing device that has a cable emulating a mouse device; executing a request that shifts a cursor within a desktop environment of the first computing device to an origin position; receiving, from a remote computing device by the second computing device, a first indication of a first position of a cursor on a user interface, wherein the first position is a first ratio of a horizontal cursor position X1 over a width of the user interface and a second ratio of a vertical cursor position Y1 over a height of the user interface; determining a projected first position by calculating a horizontal projection and a vertical projection of the first position using a desktop specification of the first computing device; determining a plurality of moves between the origin position and the projected first position;044617-00026 dividing at least one of a horizontal difference value and a vertical difference value into a plurality of movement values based on an amount of the plurality of moves, wherein the horizontal difference value is a horizontal relative movement of the cursor from the origin position and the projected first position and the vertical difference value is a vertical relative movement of the cursor from the origin position and the projected first position; applying a pan smoothing adjustment value to the plurality of movement values representing an original path of the cursor from the origin position to the projected first position; and transmitting the plurality of movement values to the first computing device configured to mirror a movement of the cursor of the remote computing device on a local cursor based on the plurality of movement values.

19. The method of claim 18, wherein the cable emulates the mouse device using a Universal Serial Bus (USB) Human Interface Device (HID) protocol.

20. The method of claim 18, wherein the origin position is a zero position along a horizontal axis and a zero position along a vertical axis.

21. The method of claim 18, wherein calculating the horizontal projection comprises applying a scaling factor that is a ratio of the width over a predetermined scale and wherein calculating the vertical projection comprises applying a scaling factor that is a ratio of the height over the predetermined scale.

Images