Method for transmitting keyboard and mouse event by remote desktop application in Wayland environment

Abstract

The invention discloses a method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment, which comprises the following steps that: a remote desktop application server in a local computer receives mouse and keyboard events sent by a remote desktop application client in a remote computer, and sends the mouse and keyboard events to XWayland through an X11 control key and mouse protocol; keyboard and mouse events and mouse and keyboard states are transmitted between the XWayland and the Wayland synthesizer based on a Wayland extension protocol, and the mouse and keyboard events are sent to the Wayland application. The invention aims to solve the problem that the remote desktop application transmits the keyboard and mouse event to the Wayland application in the Wayland environment, and the remote desktop availability of the Wayland application is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of operating systems, and in particular to a method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment. Background Art

[0002] Wayland is a communication protocol that defines how Wayland clients communicate with Wayland synthesizers. Given Wayland's advantages over X11, especially its better support for high resolution and high refresh rate, the open source community has fully turned to Wayland for development, and major system distributions have also begun to use Wayland as the default graphics display protocol. However, unlike X11's native support for network display, Wayland itself does not support remote desktop control, and various remote desktop protocols are required to implement remote desktop control functions. Currently, the widely used remote desktop protocols are VNC and RDP. The most basic work to implement a remote desktop includes: obtaining screen display data from the synthesizer, sending remote client mouse and keyboard events to the synthesizer, etc. Remote desktop applications control local computers and need to capture input events, such as events generated by keyboards and mice, and finally send them to various apps on the local computer. Generally speaking, in the X11 environment, mouse and keyboard events are generated by the actual mouse and keyboard devices of the remote desktop application client, and then obtained by the kernel input layer and then by the input driver in the Xserver, and sent to the remote desktop server application via remote control protocols such as VNC.

[0003] In the X11 or Wayland environment, the flowchart of the mouse and keyboard event transmission from the remote PC to the local X11 app is as follows Figure 1 See Figure 1, after being received by the remote desktop server application, it is sent to the server Xserver via the X11 protocol (WarpPointer and XTest), and the Xserver then performs the corresponding keyboard and mouse actions and sends these keyboard and mouse events to the corresponding X11 application. In the Wayland environment, after these events are received by the server application, they are also sent to XWayland via the X11 protocol (WarpPointer and XTest). Then XWayland sends them to the X11 application. However, there are some problems with the compatibility of X11 remote control applications using XWayland in the Wayland environment. For example, in remote desktops and conference sharing such as Sunflower Remote Desktop, HWCloudlink and other remote control scenarios, the mouse and keyboard cannot be used normally. There are two implementation interfaces in Xserver: WarpPointer and XTest. WarpPointer supports mouse events, and XTest supports mouse and keyboard events. Although XWayland supports these two interfaces, they cannot be used perfectly. Because XWayland cannot send keyboard and mouse events to Wayland applications, the events received through these two interfaces can only be sent to X11 applications. Remote desktop applications control local computers and need to send input events to server-side remote desktop applications. In the Wayland environment, XWayland forwards received keyboard and mouse events. Since XWayland cannot send events to Wayland applications, how to implement remote desktop applications in the Wayland environment to pass keyboard and mouse events to Wayland applications has become an urgent problem to be solved. Summary of the invention

[0004] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a method for a remote desktop application to transmit keyboard and mouse events in a Wayland environment is provided. The present invention aims to solve the problem of a remote desktop application transmitting keyboard and mouse events to a Wayland application in a Wayland environment, thereby ensuring the remote desktop availability of the Wayland application.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment comprises the following steps: S1, the remote desktop application server in the local computer receives the mouse and keyboard events sent from the remote desktop application client in the remote computer, and sends the mouse and keyboard events to XWayland through the X11 control keyboard and mouse protocol; S2, XWayland and the Wayland compositor transmit keyboard and mouse events and mouse and keyboard status based on the Wayland extension protocol and send the mouse and keyboard events to the Wayland application, including: Xwayland obtains keyboard and mouse event requirements through the request forwarding of the Wayland extension protocol to the Wayland compositor, and the Wayland compositor sends the mouse and keyboard status to XWayland based on the events of the Wayland extension protocol and sends the mouse and keyboard events to the Wayland application through the Wayland protocol.

[0006] Optionally, the request of the Wayland extended protocol in step S2 includes: a mouse movement request pointer_motion(x,y), parameters x and y are the requested movement position; a mouse click request pointer_button(button,state), parameters button and state are the requested mouse button value and pop-up state respectively; a mouse roller request pointer_axis(axis,value), parameters axis and value are the scrolling direction and scrolling value respectively; a keyboard key request keyboard_key(key,state), parameters key and state are the keyboard button value and pop-up state respectively.

[0007] Optionally, Xwayland forwards the keyboard and mouse event requirements to the Wayland synthesizer via the request of the Wayland extension protocol, including: when Xwayland receives mouse and keyboard events sent by the remote desktop application server, it determines the corresponding Wayland extension protocol request based on the received mouse and keyboard events, and forwards the corresponding Wayland extension protocol request to the Wayland synthesizer to obtain the keyboard and mouse event requirements.

[0008] Optionally, the received mouse and keyboard events are WarpPointer events and XTest events.

[0009] Optionally, the events of the Wayland extension protocol in step S2 include: the current state of the mouse cursor_state(x,y), where parameters x and y are the current position; the current state of the keyboard key_state(key,state), where parameters key and state are the key value pressed and the pop-up state of the keyboard respectively.

[0010] Optionally, the Wayland synthesizer sends the mouse and keyboard status to XWayland based on the event of the Wayland extension protocol and sends the mouse and keyboard events to the Wayland application through the Wayland protocol, which means that the Wayland synthesizer sends the current mouse state cursor_state(x,y) and the current keyboard state key_state(key,state) to XWayland based on the event of the Wayland extension protocol and sends the mouse and keyboard events to the Wayland application through the Wayland protocol.

[0011] Optionally, before step S1, the process also includes mouse and keyboard events sent by a remote desktop application client in the remote computer, and both the remote desktop application client in the remote computer and the remote desktop application server in the local computer are X11 applications.

[0012] In addition, the present invention also provides a system for transmitting keyboard and mouse events by remote desktop applications in a Wayland environment, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the method for transmitting keyboard and mouse events by remote desktop applications in the Wayland environment.

[0013] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the method of transmitting keyboard and mouse events by a remote desktop application in the Wayland environment through a processor.

[0014] In addition, the present invention also provides a computer program product, including a computer program or instruction, which is programmed or configured to execute the method of transmitting keyboard and mouse events by a remote desktop application in the Wayland environment through a processor.

[0015] Compared with the prior art, the present invention mainly has the following advantages: 1. In order to solve the problem that XWayland cannot send events to Wayland applications, the present invention adopts the method of sending events through the Wayland synthesizer. The method implements a Wayland extension protocol as a bridge between XWayland and the Wayland synthesizer. The request of the protocol is to forward keyboard and mouse events to the Wayland synthesizer, and the event of the protocol is to send the status of the keyboard and mouse to XWayland. When XWayland receives the keyboard and mouse events of the remote desktop class X11 application, it sends a request for protocol implementation to the Wayland synthesizer, and the Wayland synthesizer sends it to the application through the corresponding Wayland protocol, thereby solving the problem that the mouse and keyboard of the X11 remote control application cannot be used normally, and ensuring the remote desktop availability of the Wayland application.

[0016] 2. In order to solve the technical problem that remote desktop X11 applications cannot transmit keyboard and mouse events to local wayland applications, the present invention adopts a method of adding a keyboard and mouse event transmission protocol while being compatible with the original interface. The implementation of the keyboard and mouse event transmission protocol is added without changing the original code architecture, which has the advantage of good compatibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The present invention is a flowchart of transmitting mouse and keyboard events from a remote PC to a local X11 app in the prior art.

[0018] Figure 2 The flowchart of transmitting keyboard and mouse events in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the description process, unnecessary details and functions for the present invention are omitted to avoid confusion in understanding the present invention.

[0020] like Figure 2 As shown, the method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment in this embodiment includes the following steps: S1, the remote desktop application server in the local computer receives the mouse and keyboard events sent from the remote desktop application client in the remote computer, and sends the mouse and keyboard events to XWayland through the X11 control keyboard and mouse protocol; S2, XWayland and the Wayland compositor transmit keyboard and mouse events and mouse and keyboard status based on the Wayland extension protocol and send the mouse and keyboard events to the Wayland application, including: Xwayland obtains keyboard and mouse event requirements through the request forwarding of the Wayland extension protocol to the Wayland compositor, and the Wayland compositor sends the mouse and keyboard status to XWayland based on the events of the Wayland extension protocol and sends the mouse and keyboard events to the Wayland application through the Wayland protocol.

[0021] The requests of the Wayland extension protocol in step S2 of this embodiment include: a mouse movement request pointer_motion(x,y), where the parameters x and y are the requested movement position; a mouse click request pointer_button(button,state), where the parameters button and state are the requested mouse button value and pop-up state respectively; a mouse axis request pointer_axis(axis,value), where the parameters axis and value are the scrolling direction and scrolling value respectively; a keyboard key request keyboard_key(key,state), where the parameters key and state are the keyboard button value and pop-up state respectively.

[0022] In this embodiment, Xwayland forwards the keyboard and mouse event requirements to the Wayland synthesizer through the request of the Wayland extension protocol, including: when Xwayland receives the mouse and keyboard events sent by the remote desktop application server, it determines the corresponding Wayland extension protocol request according to the received mouse and keyboard events, and forwards the corresponding Wayland extension protocol requests, namely: mouse movement request pointer_motion(x,y), mouse click request pointer_button(button,state), mouse roller request pointer_axis(axis,value) and keyboard key request keyboard_key(key,state) to obtain the keyboard and mouse event requirements to the Wayland synthesizer. In this embodiment, the mouse and keyboard events received are WarpPointer events and XTest events. By adding a request to send the protocol in XWayland, when Xwayland receives the WarpPointer and XTest events, it sends a specific protocol request.

[0023] The events of the Wayland extension protocol in step S2 of this embodiment include: the current state of the mouse cursor_state(x, y), where the parameters x and y are the current position; the current state of the keyboard key_state(key, state), where the parameters key and state are the key value pressed and the pop-up state of the keyboard respectively.

[0024] An event (event) for sending a protocol is added to the Wayland synthesizer. When the Wayland synthesizer receives a request for a protocol, it sends a specific protocol event, returns the current state, and sends specific keyboard and mouse events to the Wayland application. The Wayland synthesizer sends the mouse and keyboard states to XWayland based on the event of the Wayland extension protocol and sends the mouse and keyboard events to the Wayland application through the Wayland protocol. This means that the Wayland synthesizer sends the current mouse state cursor_state(x,y) and the current keyboard state key_state(key,state) to XWayland based on the event of the Wayland extension protocol and sends the mouse and keyboard events to the Wayland application through the Wayland protocol.

[0025] In this embodiment, before step S1, the mouse and keyboard events sent by the remote desktop application client in the remote computer are also included, and the remote desktop application client in the remote computer and the remote desktop application server in the local computer are both X11 applications.

[0026] In the present embodiment, the method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment implements a Wayland extension protocol: the request of the protocol is to forward keyboard and mouse events to the Wayland synthesizer, and the event of the protocol is to send the status of the keyboard and mouse to XWayland; when a remote desktop class X11 application needs to control the local computer, the mouse and keyboard events are sent to the remote desktop class X11 application on the server side, and events such as WarpPointer and XTest for controlling the keyboard and mouse are sent to XWayland through the X11 protocol, and XWayland sends a request for protocol implementation to the Wayland synthesizer, and Wayland returns the corresponding keyboard and mouse status event to XWayland, and at the same time forwards the keyboard and mouse events to the Wayland application through the Wayland protocol, thereby solving the problem that the mouse and keyboard of the X11 remote control application cannot be used normally, and ensuring the remote desktop availability of the Wayland application.

[0027] In addition, this embodiment also provides a system for transmitting keyboard and mouse events by a remote desktop application under a Wayland environment, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the method for transmitting keyboard and mouse events by a remote desktop application under the Wayland environment. In addition, this embodiment also provides a computer-readable storage medium, wherein a computer program or instruction is stored in the computer-readable storage medium, wherein the computer program or instruction is programmed or configured to execute the method for transmitting keyboard and mouse events by a remote desktop application under the Wayland environment through a processor. In addition, this embodiment also provides a computer program product, comprising a computer program or instruction, wherein the computer program or instruction is programmed or configured to execute the method for transmitting keyboard and mouse events by a remote desktop application under the Wayland environment through a processor.

[0028] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0029] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment, characterized in that: The steps include: S1, the remote desktop application server in the local computer receives the mouse and keyboard events sent from the remote desktop application client in the remote computer, and sends the mouse and keyboard events to XWayland through the X11 control keyboard and mouse protocol; S2, XWayland and the Wayland compositor transmit keyboard and mouse events and mouse and keyboard status based on the Wayland extension protocol and send the mouse and keyboard events to the Wayland application, including: Xwayland obtains keyboard and mouse event requirements through the request forwarding of the Wayland extension protocol to the Wayland compositor, and the Wayland compositor sends the mouse and keyboard status to XWayland based on the events of the Wayland extension protocol and sends the mouse and keyboard events to the Wayland application through the Wayland protocol.

2. The method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment according to claim 1, characterized in that: The requests of the Wayland extension protocol in step S2 include: mouse movement request pointer_motion(x,y), parameters x and y are the requested movement position; mouse click request pointer_button(button,state), parameters button and state are the requested mouse button value and pop-up state respectively; mouse roller request pointer_axis(axis,value), parameters axis and value are the scrolling direction and scrolling value respectively; keyboard key request keyboard_key(key,state), parameters key and state are the keyboard button value and pop-up state respectively.

3. The method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment according to claim 2, characterized in that: The Xwayland forwards the keyboard and mouse event requirements to the Wayland synthesizer through the request of the Wayland extension protocol, including: when Xwayland receives the mouse and keyboard events sent by the remote desktop application server, it determines the corresponding Wayland extension protocol request according to the received mouse and keyboard events, and forwards the corresponding Wayland extension protocol request to obtain the keyboard and mouse event requirements to the Wayland synthesizer.

4. The method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment according to claim 3, characterized in that: The received mouse and keyboard events are WarpPointer events and XTest events.

5. The method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment according to claim 1, characterized in that: The events of the Wayland extension protocol in step S2 include: the current state of the mouse cursor_state(x,y), where the parameters x and y are the current position; the current state of the keyboard key_state(key,state), where the parameters key and state are the key value pressed and the pop-up state of the keyboard respectively.

6. The method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment according to claim 5, characterized in that: The Wayland synthesizer sends the mouse and keyboard states to XWayland based on the event of the Wayland extension protocol and sends the mouse and keyboard events to the Wayland application through the Wayland protocol, which means that the Wayland synthesizer sends the current mouse state cursor_state(x,y) and the current keyboard state key_state(key,state) to XWayland based on the event of the Wayland extension protocol and sends the mouse and keyboard events to the Wayland application through the Wayland protocol.

7. The method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment according to claim 1, characterized in that: The process before step S1 also includes mouse and keyboard events sent by a remote desktop application client in the remote computer, and both the remote desktop application client in the remote computer and the remote desktop application server in the local computer are X11 applications.

8. A system for transmitting keyboard and mouse events of a remote desktop application in a Wayland environment, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the method for transmitting keyboard and mouse events by a remote desktop application in a Wayland environment described in any one of claims 1 to 7 through a processor.

Images