Remote input method and device based on open source cross-platform GUI library

Abstract

The invention provides a remote input method and device based on an open source cross-platform GUI (Graphical User Interface) library. The method comprises the following steps: after receiving an input state of a cloud application, a client creates an input box with the same size to cover a native input box of the cloud application of the client; the input content is transmitted to a remote server; the remote server stores the input content into a shared memory and triggers a user-defined virtual key at the same time; converting the custom virtual key event into an input method event; the method comprises the following steps: firstly reading input content in a shared memory, and then converting a virtual key event into an input method event of a GUI (Graphical User Interface) library; the cloud application receives an input method event reported by the GUI library and completes input of input method content, an input method candidate box is aligned with a cloud input cursor, the input speed and experience are almost equal to those of a local input method, and the input method adapts to the language input method of a user.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud application input methods, and in particular to a remote input method and device based on an open source cross-platform GUI library. Background Art

[0002] Cloud applications are cloud computing services that allow users to deploy applications such as Windows / Linux in the cloud and access them over the Internet. This service allows users to use cloud applications on almost any device (including PCs, Macs, iOS devices, Android devices, etc.) without installing them locally. Users can connect to these applications through remote desktops as if they were running on local devices.

[0003] When the current cloud application receives the user's local input method input, there are problems such as inconsistency between the user's local system and the cloud system, inconsistency in the system input method framework, inconsistency in the input method language, etc., which causes the cloud application to be unable to correctly receive the user's local input method input. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a remote input method and device based on an open source cross-platform GUI library, in which the input method candidate box is aligned with the cloud input cursor, the input speed and experience are almost equivalent to the local input method, and the input method is adapted to the user's language.

[0005] In a first aspect, the present invention provides a remote input method based on an open source cross-platform GUI library, including a cloud application, a remote server and a client; specifically comprising the following steps:

[0006] Step 1: The remote server captures the cloud application input status and notifies it to the client. After receiving the cloud application input status, the client creates an input box of the same size to cover the native input box of the client's cloud application and waits for user input.

[0007] Step 2: The client input box receives the input of the local input method and transmits the input content to the remote server;

[0008] Step 3: The remote server receives the input content from the local client, stores the input content in the shared memory, and triggers the custom virtual key;

[0009] Step 4: When the keyboard event handler of the cloud application GUI library captures a key press, it enters the custom key event handler, which determines the key value of the pressed key. If it is a custom virtual key, the custom virtual key event is converted into an input method event; first read the input content in the shared memory, and then convert the virtual key event into an input method event of the GUI library;

[0010] Step 5: The cloud application receives the input method event reported by the GUI library and completes the input method content input.

[0011] In a second aspect, the present invention provides a remote input device based on an open source cross-platform GUI library, including a cloud application, a remote server and a client; specifically including the following modules:

[0012] Input box regeneration module: the remote server captures the cloud application input status and notifies the client. After receiving the cloud application input status, the client creates an input box of the same size to cover the native input box of the client's cloud application and waits for user input.

[0013] Local input module, the client input box receives the input of the local input method and transmits the input content to the remote server;

[0014] Input content delivery module: the remote server receives the input content from the local client, stores the input content in the shared memory, and triggers the custom virtual key;

[0015] Input content conversion module: When the keyboard event handler of the cloud application GUI library captures a key press, it enters the custom key event handler, which determines the key value of the pressed key. If it is a custom virtual key, the custom virtual key event is converted into an input method event. First, read the input content in the shared memory, and then convert the virtual key event into an input method event of the GUI library;

[0016] After the input module is completed, the cloud application receives the input method event reported by the GUI library and completes the input method content input.

[0017] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0018] The method of the present invention enables the input method candidate box to align with the cloud input cursor, and the input speed and experience are almost equivalent to the local input method, and is adapted to the user's language input method;

[0019] The current remote input method based on the cross-platform GUI library avoids the problem of secondary conversion due to different user-side systems by modifying the underlying GUI library. The user-side input events are not processed and converted by the system, which is more efficient and has lower latency. At the same time, when multiple people are remote on the cloud platform, it can well meet the scenario of multiple users inputting at the same time.

[0020] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0022] Figure 1 is a flow chart of the method in Embodiment 1 of the present invention;

[0023] Figure 2 It is a schematic diagram of the structure of the device in the second embodiment of the present invention. DETAILED DESCRIPTION

[0024] The embodiment of the present application provides a remote input method and device based on an open source cross-platform GUI library to solve the technical problem that when the user terminal and the cloud system are inconsistent, the application cannot correctly accept the input content of the user's local input method.

[0025] The technical solution in the embodiment of the present application has the following general idea:

[0026] This embodiment includes a client application and a remote server. The client application is connected to the remote server via a network. Users can input text on the client application, which is then transmitted to the remote server for processing via the network and converted into input method events for the cloud application. The process is as follows:

[0027] ① When the cloud application input box gets the focus, the cloud application passes the event to the remote server;

[0028] ②The remote server passes the input method activation event to the user's client program, and the client program creates an input box and waits for user input;

[0029] ③The client application passes the input data to the remote server;

[0030] ④The remote server generates input events to the cloud application, and the cloud application receives the events and responds;

[0031] ⑤When the cloud application input box loses focus, the event is passed to the remote server;

[0032] ⑥The remote server passes the focus loss event to the client program, and the client program destroys the created input box.

[0033] In this process, we mainly need to complete the adaptation of the cloud application to the cloud platform remote input method, so that it can directly receive the client's local input. The specific method is as follows:

[0034] Cross-platform GUI libraries need to implement platform adapters, such as the GTK input method framework that implements the adaptation of Windows and Linux platforms. On this basis, the adaptation of the cloud platform is added, and RemoteImeContex is set to be the cloud platform input method adapter.

[0035] RemoteImeContex is used to realize data interaction between the input method module in the GUI library and the cloud application. The input method is adapted to the cloud application platform based on the original GUI library, thus avoiding the problem of input failure caused by the client system and language.

[0036] The specific implementation method is as follows:

[0037] Step 1: The remote server captures the cloud application input status, which includes the input box cursor position, input box position, etc. and notifies the user-side client of the information. After receiving the information, the user-side client creates an input box of the same size to cover the native input box of the cloud application and waits for user input;

[0038] Step 2: The user-side client input box receives the user's local input method input and transmits the input content to the cloud remote server;

[0039] Step 3: The cloud remote server receives the input content from the user's local client, stores the input content D1 in the shared memory S1, and triggers the custom virtual key K1;

[0040] Step 4: The keyboard event handler of the cloud application GUI library captures the key press and enters the custom key event handler. The custom key handler determines that if the key value of the keyboard event is the virtual key K1, the K1 key event is converted into an input method event. The specific method is to first read the input content D1 in the shared memory S1, and then convert the K1 key event into a GTK input method event (the input content is D1);

[0041] Step 5: The cloud application receives the input method event reported by the GUI library and completes the input method content input.

[0042] The implementation of RemoteImeContex is related to the specific interface method of the open source GUI library. In the open source GTK library, the RemoteImeContex adapter is implemented as follows:

[0043] To implement the remote input method adapter in GTK, you need to implement the interface class method in GtkIMContextClass in GTK. The interface class can be found in the GTK source code: gtk / gtk / gtkimcontext.h file;

[0044] GtkIMContextClass is a class structure used for input method context in GTK. It defines the behavior and properties of input method context. In GTK, input method context is used to handle input method related operations during text input and editing, specifically implementing:

[0045] 1. The focus_in interface and focus_out interface in GtkIMContextClass are used to handle the operation of input method context getting focus and losing focus. RemoteImeContex implements the above interfaces and passes the input method status to the remote server through shared memory, and then to the user client. After receiving the message, the client creates a hidden input box.

[0046] 2. The set_cursor_location interface in GtkIMContextClass is used to set the cursor position so that the input method can be displayed in the correct position. RemoteImeContex implements this interface and transmits the cursor position to the user client through the remote server. The client modifies the position of the hidden input box.

[0047] 3. The filter_keypress interface in GtkIMContextClass is used to filter and process key events. The input method context can process key events here. RemoteImeContex implements this interface. After the current user client enters text, the content is written to the shared memory through the remote server and a custom virtual key is triggered. The virtual key is captured in the implementation of the filter_keypress interface, and the input text data in the shared memory is read, thereby converting the keypress event into an input method event.

[0048] 4. The get_preedit_string interface in GtkIMContextClass is used to access the text being pre-edited. When the user is entering text, the input method may display the pre-edited text before the user confirms the input. RemoteImeContex implements this interface. The current user client writes the pre-edited text to the shared memory through the remote server, so that the pre-edited text obtained by the get_preedit_string method is the pre-edited text of the user client input method.

[0049] Embodiment 1

[0050] like Figure 2 As shown, this embodiment provides a remote input method based on an open source cross-platform GUI library, including a cloud application, a remote server and a client; specifically, the following steps are included:

[0051] Step 1: The remote server captures the cloud application input status and notifies it to the client. After receiving the cloud application input status, the client creates an input box of the same size to cover the native input box of the client's cloud application and waits for user input.

[0052] Step 2: The client input box receives the input of the local input method and transmits the input content to the remote server;

[0053] Step 3: The remote server receives the input content from the local client, stores the input content in the shared memory, and triggers the custom virtual key;

[0054] Step 4: When the keyboard event handler of the cloud application GUI library captures a key press, it enters the custom key event handler, which determines the key value of the pressed key. If it is a custom virtual key, the custom virtual key event is converted into an input method event; first read the input content in the shared memory, and then convert the virtual key event into an input method event of the GUI library;

[0055] Step 5: The cloud application receives the input method event reported by the GUI library and completes the input method content input.

[0056] In this embodiment, preferably, the step 1 is specifically as follows: the remote server captures the cloud application input state, the input state including: the input box cursor position and the input box position; the cloud application input state is notified to the client, and after receiving the cloud application input state, the client creates an input box of the same size to cover the native input box of the client's cloud application, and waits for user input.

[0057] In this embodiment, preferably, step 5 is specifically as follows: the cloud application receives the input method event reported by the GUI library and completes the input of the input method content; at this time, the input box of the cloud application loses focus, the remote server transmits the focus loss event to the client, and the client destroys the created input box.

[0058] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0059] Embodiment 2

[0060] like Figure 2 As shown, in this embodiment, a remote input device based on an open source cross-platform GUI library is provided, including a cloud application, a remote server and a client; specifically, the following modules are included:

[0061] Input box regeneration module: the remote server captures the cloud application input status and notifies the client. After receiving the cloud application input status, the client creates an input box of the same size to cover the native input box of the client's cloud application and waits for user input.

[0062] Local input module, the client input box receives the input of the local input method and transmits the input content to the remote server;

[0063] Input content delivery module: the remote server receives the input content from the local client, stores the input content in the shared memory, and triggers the custom virtual key;

[0064] Input content conversion module: When the keyboard event handler of the cloud application GUI library captures a key press, it enters the custom key event handler, which determines the key value of the pressed key. If it is a custom virtual key, the custom virtual key event is converted into an input method event. First, read the input content in the shared memory, and then convert the virtual key event into an input method event of the GUI library;

[0065] After the input module is completed, the cloud application receives the input method event reported by the GUI library and completes the input method content input.

[0066] In this embodiment, preferably, the input box regeneration module is specifically as follows: the remote server captures the cloud application input state, and the input state includes: the input box cursor position and the input box position; the cloud application input state is notified to the client, and after the client receives the cloud application input state, it creates an input box of the same size to cover the native input box of the client's cloud application, and waits for user input.

[0067] In this embodiment, preferably, the input completion module is specifically as follows: the cloud application receives the input method event reported by the GUI library and completes the input method content input; at this time, the input box of the cloud application loses focus, the remote server transmits the focus loss event to the client, and the client destroys the created input box.

[0068] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0069] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0070] In the prior art, remote input methods are all based on input method event mapping and conversion at the system level. This method has the problems of long data path (data conversion is required and the input method framework of the system is passed), inability to align the cursor with the input method candidate box, and inability to meet the requirements of multiple users inputting simultaneously.

[0071] The current remote input method based on the cross-platform GUI library avoids the problem of secondary conversion due to different user-side systems by modifying the underlying GUI library. The user-side input events are not processed and converted by the system, which is more efficient and has lower latency. At the same time, when multiple people are remote on the cloud platform, since it can adapt to the input methods of multiple systems, it can well meet the scenario of multiple users inputting at the same time.

[0072] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A remote input method based on an open source cross-platform GUI library, characterized in that: It includes cloud applications, remote servers and clients; specifically includes the following steps: Step 1: The remote server captures the cloud application input status and notifies it to the client. After receiving the cloud application input status, the client creates an input box of the same size to cover the native input box of the client's cloud application and waits for user input. Step 2: The client input box receives the input of the local input method and transmits the input content to the remote server; Step 3: The remote server receives the input content from the local client, stores the input content in the shared memory, and triggers the custom virtual key; Step 4: When the keyboard event handler of the cloud application GUI library captures a key press, it enters the custom key event handler, which determines the key value of the pressed key. If it is a custom virtual key, the custom virtual key event is converted into an input method event; first read the input content in the shared memory, and then convert the virtual key event into an input method event of the GUI library; Step 5: The cloud application receives the input method event reported by the GUI library and completes the input method content input.

2. The remote input method based on an open source cross-platform GUI library according to claim 1, characterized in that: The step 1 is specifically as follows: the remote server captures the cloud application input state, the input state including: the input box cursor position and the input box position; the cloud application input state is notified to the client, and after receiving the cloud application input state, the client creates an input box of the same size to cover the native input box of the client's cloud application, and waits for user input.

3. The remote input method based on an open source cross-platform GUI library according to claim 1, characterized in that: The step 5 is specifically as follows: the cloud application receives the input method event reported by the GUI library and completes the input of the input method content; at this time, the input box of the cloud application loses focus, the remote server transmits the focus loss event to the client, and the client destroys the created input box.

4. A remote input device based on an open source cross-platform GUI library, characterized in that: It includes cloud applications, remote servers and clients; specifically includes the following modules: Input box regeneration module: the remote server captures the cloud application input status and notifies the client. After receiving the cloud application input status, the client creates an input box of the same size to cover the native input box of the client's cloud application and waits for user input. Local input module, the client input box receives the input of the local input method and transmits the input content to the remote server; Input content delivery module: the remote server receives the input content from the local client, stores the input content in the shared memory, and triggers the custom virtual key; Input content conversion module: When the keyboard event handler of the cloud application GUI library captures a key press, it enters the custom key event handler, which determines the key value of the pressed key. If it is a custom virtual key, the custom virtual key event is converted into an input method event. First, read the input content in the shared memory, and then convert the virtual key event into an input method event of the GUI library; After the input module is completed, the cloud application receives the input method event reported by the GUI library and completes the input method content input.

5. The remote input method based on an open source cross-platform GUI library according to claim 4, characterized in that: The input box regeneration module is specifically as follows: the remote server captures the cloud application input state, the input state includes: the input box cursor position and the input box position; the cloud application input state is notified to the client, and after the client receives the cloud application input state, it creates an input box of the same size to cover the native input box of the client's cloud application, waiting for user input.

6. The remote input method based on an open source cross-platform GUI library according to claim 4, characterized in that: The input completion module is specifically as follows: the cloud application receives the input method event reported by the GUI library and completes the input method content input; at this time, the input box of the cloud application loses focus, the remote server transmits the focus loss event to the client, and the client destroys the created input box.

Images