Cloud-based user input method, apparatus, device, and storage medium

Abstract

The application provides a cloud-based user input method, device, equipment and storage medium, and relates to the technical field of computers. The method comprises the following steps: deploying a preset cloud input method to a server; in response to receiving a trigger operation of an input operation sent by a client, calling the preset cloud input method; generating video stream data comprising the preset cloud input method according to interface content coding of the preset cloud input method; and sending the video stream data to the client to display a video screen comprising interface content of the cloud input method on the client. Compared with the prior art, the method avoids compatibility problems between different devices, thereby preventing some function keys of the cloud input method from being disabled during user interaction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a user input method and device based on cloud, equipment and storage medium. BACKGROUND

[0002] Cloud gaming is a new way of running games based on cloud computing. This technology was first proposed by Onlive company. In 2018, with the participation of Google, Microsoft, Nvidia, Sony and other giants, the popularity of cloud gaming is rising. In the cloud gaming mode, the rendering of game pictures is completed on the server side. The rendered game pictures are encoded and streamed to the game player's terminal. In the client, the game player no longer needs high-end game equipment, but only needs basic video decoding capability to play the game.

[0003] There are many user input interaction scenarios in cloud gaming. In the prior art, the client generally pops up a local keyboard in the game interface after receiving the interaction notification, and the client listens to the user's input, transmits the user's input content to the cloud, and the cloud forwards the received input content to the game and displays the user's input content in the game interface.

[0004] However, such a way may have compatibility problems, resulting in the failure of some function keys of the cloud input method, and may have subsequent access device compatibility problems. SUMMARY

[0005] The purpose of the present application is to solve the problem of compatibility between different devices in the prior art, which causes some function keys of the cloud input method to fail in the user interaction process.

[0006] To achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a user input method based on cloud, which comprises:

[0008] Deploying a preset cloud input method to the server;

[0009] In response to receiving the trigger operation of the input operation sent by the client, calling the preset cloud input method;

[0010] Generating video stream data including the preset cloud input method according to the interface content coding of the preset cloud input method;

[0011] send the video stream data to the client to display a video picture including interface content of the cloud input method on the client.

[0012] Optionally, the deploying the preset cloud input method to the server comprises:

[0013] preinstalling the preset cloud input method into an image of a server system;

[0014] installing a corresponding preset cloud input method in each cloud container according to the preset cloud input method preinstalled in the image of the server system.

[0015] Optionally, the preinstalling the preset cloud input method into an image of a server system comprises:

[0016] adding files of the preset cloud input method into a compilation folder of the image of the server system;

[0017] recompiling compilation source code in the compilation folder in a preset compilation manner.

[0018] Optionally, the invoking the preset cloud input method in response to receiving a trigger operation of an input operation sent by the client comprises:

[0019] invoking the preset cloud input method in a corresponding cloud container in response to receiving a trigger operation of an input operation sent by the client.

[0020] Optionally, the invoking the preset cloud input method in response to receiving a trigger operation of an input operation sent by the client comprises:

[0021] determining a device type of an input device corresponding to the input operation in response to receiving the trigger operation of the input operation sent by the client; the device type of the input device comprises any one of a terminal device corresponding to the client and an external device;

[0022] invoking a preset type of cloud input method corresponding to the device type of the input device in a corresponding cloud container according to the device type of the input device.

[0023] Optionally, the determining the device type of the input device corresponding to the input operation comprises:

[0024] obtaining the device type of the current input device sent by the client through a connection established with the client.

[0025] Optionally, the method further comprises:

[0026] obtaining input operation data sent by the client based on a graphical user interface of the preset cloud input method;

[0027] determining output content corresponding to the input operation data according to the input operation data;

[0028] sending the output content generated according to the input operation data to the client.

[0029] Optionally, the obtaining of the input operation data sent by the client based on the graphical user interface of the preset cloud input method comprises:

[0030] obtaining the serialized input operation data sent by the client through the connection established with the client.

[0031] Optionally, the determining of the output content corresponding to the input operation data according to the input operation data comprises:

[0032] deserializing the serialized input operation data to determine the output content corresponding to the input operation data.

[0033] Optionally, the method further comprises:

[0034] sending an establishment response corresponding to the long connection establishment request to the client in response to the received long connection establishment request sent by the client.

[0035] In a second aspect, another embodiment of the present application provides a cloud-based user input device applied to a server, comprising a deployment module, a calling module, a generation module and a sending module, wherein:

[0036] The deployment module is configured to deploy a preset cloud input method to the server.

[0037] The calling module is configured to call the preset cloud input method in response to receiving a trigger operation of input operation sent by a client.

[0038] The generation module is configured to generate video stream data comprising the preset cloud input method according to interface content coding of the preset cloud input method.

[0039] The sending module is configured to send the video stream data to the client to display a video screen comprising interface content of the cloud input method on the client.

[0040] Optionally, the device further comprises an installation module configured to preinstall the preset cloud input method into an image of a server system, and install a corresponding preset cloud input method in each cloud container according to the preset cloud input method preinstalled in the image of the server system.

[0041] Optionally, the device further comprises an adding module and a compiling module, wherein:

[0042] The adding module is configured to add the file of the preset cloud input method into a compiling folder of a mirror of the server system.

[0043] The compiling module is configured to recompile compiling source code in the compiling folder in a preset compiling manner.

[0044] Optionally, the calling module is specifically configured to call the preset cloud input method in the corresponding cloud container in response to receiving a trigger operation of an input operation sent by the client.

[0045] Optionally, the apparatus further includes a determining module configured to determine a device type of an input device corresponding to the input operation in response to a received trigger operation of an input operation; the device type of the input device includes any one of a terminal device corresponding to the client and an external device.

[0046] The calling module is specifically configured to call a preset type of cloud input method corresponding to the device type of the input device in the corresponding cloud container according to the device type of the input device.

[0047] Optionally, the apparatus further includes an obtaining module configured to obtain the device type of the current input device sent by the client through a connection established with the client.

[0048] Optionally, the obtaining module is specifically configured to obtain input operation data sent by the client based on a graphical user interface of the preset cloud input method.

[0049] The determining module is specifically configured to determine output content corresponding to the input operation data according to the input operation data.

[0050] The generating module is specifically configured to send the output content generated according to the input operation data to the client.

[0051] Optionally, the obtaining module is specifically configured to obtain serialized input operation data sent by the client through a connection established with the client.

[0052] Optionally, the determining module is specifically configured to deserialize the serialized input operation data to determine output content corresponding to the input operation data.

[0053] Optionally, the sending module is specifically configured to send an establishment response corresponding to the long connection establishment request to the client in response to receiving a long connection establishment request sent by the client.

[0054] In a third aspect, another embodiment of the present application provides a cloud-based user input device, comprising a processor, a storage medium and a bus, the storage medium storing machine readable instructions executable by the processor, when the cloud-based user input device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine readable instructions to perform the steps of any of the methods of the first aspect.

[0055] In a fourth aspect, another embodiment of the present application provides a storage medium, the storage medium storing a computer program, when the computer program is run by a processor, the steps of any of the methods of the first aspect are performed.

[0056] The beneficial effects of the present application are: the cloud-based user input method provided by the present application, because the preset cloud input method is pre-deployed on the server, when the server receives the trigger operation of the input operation sent by the client, the preset cloud input method is called, and the video stream data including the preset cloud input method is generated according to the preset cloud input method and the display content of the client, and the generated video stream data including the preset input method is sent to the client. Such input method is directly deployed on the server using the preset cloud input method, rather than directly using the local input method of each client, so as to overcome the compatibility problem of displaying different cloud input methods in the graphical user interface of the client due to different devices of the client. That is, when determining the graphical user interface including the preset cloud input method, the method provided by the present application does not need to consider the compatibility problem, which not only improves the user's experience, but also improves the development time and efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0058] Figure 1 Flowchart of the cloud-based user input method provided by an embodiment of the present application;

[0059] Figure 2 Flowchart of the cloud-based user input method provided by another embodiment of the present application;

[0060] Figure 3 Flowchart of the cloud-based user input method provided by another embodiment of the present application;

[0061] Figure 4A structural schematic diagram of a cloud-based user input device according to an embodiment of the present application is provided.

[0062] Figure 5 A structural schematic diagram of a cloud-based user input device according to another embodiment of the present application is provided.

[0063] Figure 6 A structural schematic diagram of a cloud-based user input device according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application.

[0065] The components of the embodiments of the present application generally described and illustrated in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0066] In addition, the flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application by those skilled in the art.

[0067] The cloud-based user input method in one of the embodiments of the present disclosure can run on a server. When the cloud-based user input method runs on the server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a terminal device, and the terminal device is installed with a client.

[0068] In an optional embodiment, various cloud applications, such as cloud games, can run under the cloud interaction system. Taking cloud games as an example, cloud games refer to a game mode based on cloud computing. In the running mode of cloud games, the running subject of a game program and the presentation subject of a game picture are separated, and the storage and running of a user input method based on a cloud end are completed on a cloud game server. A client is used for receiving and sending data and presenting a game picture. For example, the client can be installed and run on a display device close to a user side, such as a mobile terminal, a television, a computer, a palm computer, and the like, and an information processing device can be a cloud game server of a cloud end. When playing a game, a player operates the client to send an operation instruction to the cloud game server, the cloud game server runs the game according to the operation instruction, encodes and compresses game picture data, returns the data to the client through a network, and finally, the client decodes and outputs the game picture.

[0069] To facilitate the understanding of the embodiments of the present application, the following explains some terms related to the present application:

[0070] Mirror image: mirror image is a mirror image server. The service contents of the mirror image server and the main server are the same, only placed in a different place, sharing the load of the main server. It can be used, but it is not the original. Two or more servers with the same content and synchronous updates on the Internet, except for the main server, are called mirror image servers. Mirror image is a type of redundancy, and the data on one disk has a complete copy on another disk.

[0071] Android application package (APK): is an application package file format used by the Android operating system for distributing and installing mobile applications and middleware. The code of an Android application wants to run on an Android device, must be compiled first, and then packaged into a file that can be recognized by the Android system before it can be run, and this file format that can be recognized and run by the Android system is APK.

[0072] Container technology: effectively divides the resources of a single operating system into isolated groups to better balance conflicting resource usage requirements between isolated groups. This technology is called container technology.

[0073] The following explains a cloud-based user input method provided by the embodiments of the present application in combination with a plurality of specific application examples. Figure 1 The flowchart of a cloud-based user input method provided by an embodiment of the present application is illustrated by taking a server as an execution subject, as shown in Figure 1As shown, the method comprises:

[0074] S101: deploying a preset cloud input method to a server.

[0075] In an embodiment of the present application, the server can be a cloud server, for example, and the preset cloud input method data can be pre-installed in the image of the server system in the following manner: the preset cloud input method is pre-installed in the image of the server system using the adb install<path_to_apk> command, then the deployed preset cloud input method is made available through the ime enable<your.ime.service> command, and finally the preset cloud input method is set through the ime set<your.ime.service> command. However, this deployment manner is mainly used in development and testing. Since there are many cloud instances online, the installation of each instance separately is very time-consuming. Therefore, in an embodiment of the present application, the pre-installed input method data can be pre-installed in the image, thereby improving the installation efficiency of each instance, reducing the installation time, and ensuring that each instance is synchronized with the installation of the preset cloud input method, that is, ensuring the unity of the cloud input method in each instance.

[0076] In some possible embodiments, the deployment manner of the preset cloud input method can be, for example, pre-installing the preset cloud input method into the image of the server system, and installing the corresponding preset cloud input method in each cloud container according to the preset cloud input method pre-installed in the image of the server system. Each cloud container of the cloud server is installed with the preset cloud input method.

[0077] For example, the installation of the corresponding preset cloud input method in each cloud container can be performed when each cloud container is started for the first time, or the installation of the corresponding preset cloud input method in each cloud container can be performed directly according to the preset cloud input method pre-installed in the image of the server system after the preset cloud input method is pre-installed in the image of the server system. The timing of installing the preset cloud input method in each cloud container can be flexibly adjusted according to user needs, and is not limited to the above embodiments.

[0078] In other possible embodiments, when the preset cloud input method in the image of the server system is updated or changed, each cloud container adjusts the preset cloud input method installed in the cloud container according to the changed preset cloud input method. The adjusted preset cloud input method is consistent with the changed preset cloud input method in the image of the server system.

[0079] For example, in an embodiment of the present application, the pre-installed manner of the preset cloud input method in the server system can be, for example, adding the file of the preset cloud input method to the compiling folder of the image of the server system; re-encoding the compiling source code in the compiling file by using the preset compiling manner, and the specific pre-installed steps are as follows:

[0080] 1) creating a new folder in the packages / apps / <folder_name> in the image of the server system.

[0081] 2) putting the pre-deployed apk file of the preset cloud input method into the new folder in step 1.

[0082] 3) creating the compiling source code Android.mk file corresponding to the preset cloud input method in the new folder in step 1, and filling the related information of the apk of the preset cloud input method in the compiling source code file.

[0083] 4) modifying the compiling file build / target / product / core.mk in the image of the server system, and adding the file names of the preset cloud input method apk file and the Android.mk file newly added in steps 2 and 3 in the compiling folder.

[0084] 5) re-compiling the compiling source code in the compiling folder according to the modified compiling folder.

[0085] S102: in response to receiving the trigger operation of the input operation sent by the client, calling the preset cloud input method.

[0086] In an embodiment of the present application, in the game scene requiring the input method, the client can trigger the trigger operation of sending the input operation to the client by pulling up the preset input window corresponding to the preset cloud input method pre-set in the cloud in the game interface, for example, the client can trigger the preset input window corresponding to the preset cloud input method pre-set in the cloud by clicking the preset dialog control in the graphical user interface, or by triggering the preset pull-up shortcut key.

[0087] For example, in the embodiments of the present application, the server also pre-deploys the pull-up logic for the input operation. By default, when the cloud container in the server needs to pull up the input method, the pulled-up input method is the pre-deployed preset cloud input method. Therefore, after the server receives the trigger operation of the input operation sent by the client, the preset cloud input method is directly pulled up and called in the corresponding cloud container according to the pre-deployed pull-up logic, so as to ensure that the preset cloud input method is called after the trigger operation of the input operation is received, instead of the input method corresponding to the terminal device of the client, thereby ensuring that the preset cloud input method is called when the input method is called by the client using different terminal devices, and ensuring the uniformity of the input method called by the containers in the server and avoiding the problem that the compatibility of the input method corresponding to different terminal devices cannot be ensured.

[0088] S103: generating video stream data including the preset cloud input method according to the interface content coding of the preset cloud input method.

[0089] In an embodiment of the present application, the way of generating the graphical user interface including the preset cloud input method may be, for example, that in response to the received trigger operation of the input operation, the display content in the graphical user interface in the current client is obtained; the called preset cloud input method and the display content of the graphical user interface are coded and processed to generate the video stream data including the preset cloud input method.

[0090] S104: sending the video stream data to the client to display the video screen including the interface content of the cloud input method on the client.

[0091] Correspondingly, after the client obtains the video stream data including the preset cloud input method sent by the server, the video stream data including the preset cloud input method after the coding processing needs to be decoded, and the data after the decoding needs to be rendered, and then the video stream data including the preset input window corresponding to the preset cloud input method is displayed on the client.

[0092] Since the preset input window and the game interface are in the same interface in the cloud, the screen content captured by the cloud server contains the preset cloud input method and the game interface, and after the coding, the video stream data is transmitted to the client. After the client obtains the coded video stream data, the coded video stream data is decoded and rendered, and the game interface including the preset input window corresponding to the preset cloud input method is displayed on the graphical user interface. Therefore, the user can see the cloud game interface and the preset input window corresponding to the preset cloud input method, so the client no longer needs to pop up the local keyboard, but directly displays the preset input window corresponding to the preset cloud input method pre-deployed by the cloud server, that is, the preset virtual keyboard corresponding to the preset cloud input method.

[0093] The preset input window corresponding to the preset cloud input method can be at a focus position in the graphical user interface.

[0094] With the cloud-based user input method provided in the present application, since the preset cloud input method is pre-deployed on the server, when the server receives the trigger operation of the input operation sent by the client, the preset cloud input method is called, and video stream data including the preset cloud input method is generated according to the preset cloud input method and the display content of the client, and the generated video stream data including the preset input method is sent to the client. Such an input method overcomes the compatibility problem of displaying different cloud input methods in the graphical user interface of the client due to different devices of the client itself, that is, the method provided in the present application does not need to consider the compatibility problem when determining the graphical user interface including the preset cloud input method, which not only improves the user experience, but also improves the development time and efficiency.

[0095] Optionally, on the basis of the above-mentioned embodiments, the present application embodiment can also provide a cloud-based user input method, and the implementation process of displaying the preset input window in the above-mentioned method will be exemplarily described below in combination with the accompanying drawings. Figure 2 A flowchart of a cloud-based user input method provided for another embodiment of the present application is shown in FIG. 2, and S102 can include: Figure 2

[0096] S105: In response to the received trigger operation of the input operation, determining the device type of the input device corresponding to the input operation.

[0097] For example, the device type of the input device can include any one of a terminal device and an external device; that is, when the current input device is a display screen of a terminal device, the player can realize the input of the input operation by touching the display screen of the terminal device. The external device can also be an external touch screen, an external mouse, an external handle, an external remote controller, or an external keyboard, etc., and the player can realize the trigger operation of the input operation through the external device. It should be understood that the above-mentioned embodiments are only exemplary descriptions, and the types of the external device can be flexibly adjusted according to user needs, and are not limited to the above-mentioned embodiments.

[0098] S106: According to the device type of the input device, calling the preset type of cloud input method corresponding to the device type of the input device in the corresponding cloud container.

[0099] ​In an embodiment of the present application, in a game scenario requiring an input method, the server needs to acquire the device type of the current input device sent by the client in advance, and then the server switches the cloud input method to the preset cloud input method corresponding to the current device type according to the device type of the different input device. Different device types can correspond to different preset cloud input methods.

[0100] For example, in the mirror of the server system, a plurality of preset cloud input methods corresponding to different device types can be pre-installed, and then in the use process, the preset cloud input method corresponding to the current device type can be directly called according to the device type of the different input device after the device type of the current input device is determined.

[0101] In some possible embodiments, when the external device is initially connected to the client, the client sends the device type of the currently connected external device to the server, so that the server directly determines the preset cloud input method corresponding to the external device according to the device type of the external device. If the external device is changed during the game, for example, the external device is detected to be removed and then reconnected, when the new external device is detected to be connected, the client sends the device type of the newly connected external device to the server, so as to realize the synchronization of the external device type update.

[0102] In another possible embodiment, the client can also acquire the device type of the current external device when the input operation is triggered, and send the acquired device type of the current external device to the server, so that the server determines the preset input window corresponding to the current external device according to the device type of the current external device. It should be understood that the above embodiments are only exemplary, and the timing and manner of sending the device type of the external device can be flexibly adjusted according to user needs, and are not limited by the above embodiments.

[0103] Optionally, based on the above embodiments, the embodiments of the present application can also provide a cloud-based user input method, and the implementation process of displaying the preset input window in the above method will be exemplarily described below with reference to the accompanying drawings. Figure 3 A flowchart of a cloud-based user input method provided for another embodiment of the present application is shown in FIG. 2, which can further include the following steps. Figure 3

[0104] S107: Acquire the input operation data sent by the graphical user interface of the client based on the preset cloud input method.

[0105] ​In some possible embodiments, the client can obtain the serialized input operation data sent by the graphical user interface of the preset cloud input method, for example, in the form of ProtoBuf, so as to improve the efficiency, reduce the space occupation of the input operation data, and improve the processing speed of the input operation data. After the input operation data is serialized, the client can directly send the serialized input operation data to the server.

[0106] In an embodiment of the present application, after the graphical user interface of the preset cloud input method is sent to the client, the client can listen to the input operation data of the player in real time. For example, the player touches a certain position of the display screen of the client, or the client listens to the player clicking a certain position of the display screen of the client through an external mouse. Then, the client transmits the input operation data to the cloud server, so that the cloud server obtains the input instruction corresponding to the input operation data according to the received input operation data.

[0107] In some possible embodiments, the client encapsulates the input operation data of the current player in the form of KeyEvent. The input operation data is encapsulated to prepare for subsequent serialization.

[0108] S108: determining the output content corresponding to the input operation data according to the input operation data.

[0109] Correspondingly, after the serialized input operation data is obtained, the server can deserialize the serialized input operation data to determine the output content corresponding to the input operation data.

[0110] Through the serialization and deserialization, the space occupation of the input operation data is reduced, the processing speed of the input operation data is improved, the accurate transmission of the input operation data is ensured, and the accuracy of the output content corresponding to the input instruction obtained according to the input operation data is ensured.

[0111] In an embodiment of the present application, after the input operation sent by the client is obtained, the cloud server injects the obtained input operation into the recognition processing system in the cloud. The recognition processing system processes the obtained input operation according to the preset cloud input method corresponding to the current external device and the processing mode corresponding to the preset cloud input method, obtains the output content corresponding to the input operation, and then sends the obtained output content to the client, thereby completing the process of processing the input operation of the player.

[0112] In some possible embodiments, the input operation injection manner may, for example, be using an Android Instrumentation object, through which an injection instruction is sent by a method starting with send(), where send() is a series of functions starting with send, for example, sendStringSync is a text instruction sending function, and sendKeySync is a key instruction sending function.

[0113] In some possible embodiments of the present application, the input operation injection manner may, for example, be that the Android system based on Linux can directly inject an instruction into / dev / input / eventX; / dev is a directory under the Linux operating system, used to store hardware device information, through which a hardware can be directly accessed; input is a directory, storing input device files; eventX stores event queue files, and X is an integer representing the number of event queues, for example, event0 and event1, and writing an input instruction into an event queue file implements instruction injection; it should be understood that the above embodiments are only exemplary, and the specific injection manner can be flexibly adjusted according to user needs, and is not limited to the above embodiments.

[0114] In an embodiment of the present application, since the preset input window is the focus interface in the current graphical user interface, the input operation is processed by the system according to the input method corresponding to the current external device and the processing manner of the input method, that is, the priority of processing the input operation is the highest event in the current graphical user interface.

[0115] S108: sending the output content generated according to the input operation data to the client.

[0116] For example, in some possible embodiments, before S101, the device type of the current input device needs to be sent to the server through the connection established with the server, so that the server determines the preset input window corresponding to the current input device according to the device type of the current input device. The present application supports more external devices as input devices of the game, and different input devices may correspond to different input windows, and different input windows may correspond to different interaction manners.

[0117] In an embodiment of the present application, Table 1 is the correspondence between different external devices and interaction manners provided by an embodiment of the present application.

[0118]

[0119] Table 1

[0120] As shown in Table 1, when the external device is a touch screen or a mouse, only a preset virtual keyboard needs to be displayed on the graphical user interface, and input operation data can be directly input through a finger clicking operation of a user on the touch screen or a clicking operation of the user on the graphical user interface by using the mouse. The touch screen can be a touch screen of the external device or a display touch screen of the terminal device itself. When the external device is a handle or a remote controller, a virtual keyboard can be provided on the graphical user interface, and then the player moves the handle or the remote controller to move the cursor to a position of a target key to be selected, and then confirms the target key by using a confirmation key or a preset key, for example, an A key. When the external device is a keyboard, the input operation data can be directly received from the physical keyboard, and it should be understood that the above embodiments are only exemplary, and other types of external devices can be used, and the corresponding interaction mode can be adjusted as required.

[0121] For example, in another embodiment of the present application, the server can provide a game word library corresponding to the current game or a game shortcut and an expression package related to the current game in the input window based on the current game, in addition to determining different preset input windows according to different external devices. It should be understood that the above embodiments are only exemplary, and the content included in the input window can be adjusted as required.

[0122] In the embodiments provided in the present application, the preset input method window and the interaction are mainly rewritten Input Method Service (Input Method Service), a window designed in advance is displayed in the graphical user interface when an input view is created (on Create Input View), and the input operation of the key is processed in the key down (on Key Down) and the key up (on Key Up). It should be understood that the above embodiments are only exemplary, and the specific implementation can be adjusted as required.

[0123] In an embodiment of the present application, before the cloud game runs, a long connection needs to be established between the client and the cloud server, which is used for the client and the cloud server to exchange information or data, negotiate cloud game running configuration information, and the like, and through the long connection, the cloud player's external device type information and the like are informed; for example, the server needs to obtain the device type of the current input device sent by the client through the connection established with the client, or the server obtains the serialized input operation data sent by the client based on the preset cloud input method graphical user interface through the connection established with the client. Specifically, the cloud server sending data to the client can also be sent through the long connection.

[0124] The pre-established connection mode can be, for example, that the client sends a long connection establishment request to the server; the server sends a long connection establishment request corresponding establishment response to the client in response to the received long connection establishment request sent by the client, and the client completes the establishment of the long connection with the server after receiving the establishment response fed back by the server according to the long connection establishment request; the establishment of the long connection avoids the influence of the transmission of other data, and ensures the low latency of instruction transmission. In some possible embodiments, the long connection can be, for example, a long connection established based on the transmission control protocol (TCP), and it should be understood that the above embodiment is only exemplary, and the establishment mode and form of the long connection can be flexibly adjusted according to user needs, and are not limited to the above embodiment.

[0125] The cloud-based user input method provided in the application has the following advantages: the preset cloud input method is deployed on the server in advance, so when the server receives the trigger operation of the input operation sent by the client, the preset cloud input method is called and a graphical user interface including the preset cloud input method is generated, and the generated graphical user interface including the preset input method is sent to the client, so that the client displays the preset input window corresponding to the preset input method on the graphical user interface, wherein the preset cloud input method is determined according to the device type of the current input device, and different device types may correspond to different preset cloud input methods; after determining the preset input window corresponding to the preset cloud input method, the client sends the input operation data on the preset input window to the server, and then displays the output content corresponding to the input instruction on the preset input window according to the output content corresponding to the input instruction sent by the server. Such an input method overcomes the compatibility problem caused by different local input windows of different devices, avoids the compatibility problem caused by the client listening to the input method, does not need to consider the compatibility problem when implementing the user input function, improves the user's experience, shortens the development time and improves the development efficiency; in addition, the application can also provide related game word library, shortcut language and emoticon package based on the currently played game, further facilitating and enriching the input of players during the game process, and improving the interest and convenience of the communication process of the players.

[0126] The cloud-based user input device provided in the application is explained and described below in conjunction with the drawings. The cloud-based user input device can execute the above Figures 1-3 Any cloud-based user input method, its specific implementation and advantages are described above, and the following will not be described again.

[0127] Figure 4 The structure diagram of the cloud-based user input device provided in an embodiment of the application is shown in Figure 4 The device is applied to a server, and the device includes a deployment module 201, a calling module 202, a generation module 203 and a sending module 204, wherein:

[0128] The deployment module 201 is configured to deploy the preset cloud input method to the server;

[0129] The calling module 202 is configured to call the preset cloud input method in response to receiving the trigger operation of the input operation sent by the client;

[0130] The generation module 203 is configured to generate video stream data including the preset cloud input method according to the interface display content coding of the preset cloud input method;

[0131] The sending module 204 is used to send video stream data to the client so that the video screen, including the interface content of the cloud input method, can be displayed on the client.

[0132] Optionally, based on the above embodiments, this application embodiment may also provide a cloud-based user input device, as described below with reference to the accompanying drawings. Figure 4 The implementation process of the given device is illustrated with examples. Figure 5 A schematic diagram of the structure of a cloud-based user input device provided in another embodiment of this application is shown below. Figure 5 As shown, the device also includes: an installation module 205, used to pre-install a preset cloud input method into the image of the server system; and to install the corresponding preset cloud input method in each cloud container according to the preset cloud input method pre-installed in the image of the server system.

[0133] like Figure 5 As shown, the device also includes: an adding module 206 and a compiling module 207, wherein:

[0134] Add module 206 to add the files of the preset cloud input method to the compilation folder of the server system image;

[0135] Compiler module 207 is used to recompile the source code in the compilation folder using the preset compilation method.

[0136] Optionally, module 202 is invoked in response to a trigger operation that receives an input operation from the client, and a preset cloud input method is invoked in the corresponding cloud container.

[0137] like Figure 5 As shown, the device further includes: a determining module 208, used to determine the device type of the input device corresponding to the input operation in response to a triggering operation of the received input operation; the device type of the input device includes: either the terminal device corresponding to the client or an external device;

[0138] The calling module 202 is specifically used to call the preset type of cloud input method corresponding to the device type of the input device in the corresponding cloud container according to the device type of the input device, and generate a graphical user interface that displays the preset type of cloud input method.

[0139] like Figure 5 As shown, the device also includes an acquisition module 209, used to acquire the device type of the current input device sent by the client through a connection established with the client.

[0140] Optionally, the acquisition module 209 is specifically used to acquire input operation data sent by the client based on the graphical user interface of the preset cloud input method;

[0141] The determination module 208 is specifically used to determine the output content corresponding to the input operation data based on the input operation data.

[0142] The generation module 203 is specifically used to send the output content generated based on the input operation data to the client.

[0143] Optionally, the acquisition module 209 is specifically used to acquire serialized input operation data sent by the client through a connection established with the client.

[0144] Optionally, the determining module 208 is specifically used to deserialize the serialized input operation data and determine the output content corresponding to the input operation data.

[0145] Optionally, the sending module 204 is specifically used to send an establishment response corresponding to the long connection establishment request to the client in response to the long connection establishment request received from the client.

[0146] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0147] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0148] Figure 6 This is a schematic diagram of the structure of a cloud-based user input device provided in an embodiment of this application. The cloud-based user input device can be integrated into a terminal device or a chip of a terminal device.

[0149] like Figure 6 As shown, the cloud-based user input device includes a processor 501, a storage medium 502, and a bus 503.

[0150] Processor 501 is used to store programs, and processor 501 calls the programs stored in storage medium 502 to execute the above-mentioned programs. Figures 1-3Corresponding method embodiments. The specific implementation manners and technical effects are similar, and will not be described here.

[0151] Optionally, the present application also provides a program product, for example, a storage medium, which stores a computer program, including a program which, when executed by a processor, performs the above-mentioned method corresponding embodiments.

[0152] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0153] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0154] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0155] The integrated unit realized in the form of software functional unit can be stored in a computer readable storage medium. The software functional unit stored in a storage medium includes a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) execute part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, and various program code storage media.

Claims

1. A cloud-based user input method, comprising: The method applied to a server comprises: deploying a preset cloud input method to the server; in response to receiving a triggering operation of an input operation sent by a client, calling the preset cloud input method corresponding to a device type of an input device corresponding to the input operation based on the device type of the input device; generating video stream data comprising the preset cloud input method according to interface content coding of the preset cloud input method; sending the video stream data to the client to display video stream data comprising a preset input window corresponding to the preset cloud input method on the client, the preset input window being a preset virtual keyboard corresponding to the preset cloud input method; the generating of the video stream data comprising the preset cloud input method according to the interface content coding of the preset cloud input method comprises: in response to the received triggering operation of the input operation, acquiring display content in a graphical user interface in the current client; and encoding and processing the called preset cloud input method and the display content of the graphical user interface to generate the video stream data comprising the preset cloud input method.

2. The method of claim 1, wherein, the deploying of the preset cloud input method to the server comprises: preinstalling the preset cloud input method into an image of a server system; installing corresponding preset cloud input methods in respective cloud containers according to the preset cloud input method preinstalled in the image of the server system.

3. The method of claim 2, wherein, the preinstalling of the preset cloud input method into the image of the server system comprises: adding files of the preset cloud input method to a compilation folder of the image of the server system; and recompiling compilation source codes in the compilation folder in a preset compilation mode.

4. The method of claim 2, wherein, the calling of the preset cloud input method in response to the received triggering operation of the input operation sent by the client comprises: in response to the received triggering operation of the input operation sent by the client, calling the preset cloud input method in a corresponding cloud container.

5. The method of claim 2, wherein, the calling of the preset cloud input method in response to the received triggering operation of the input operation sent by the client comprises: in response to the received triggering operation of the input operation sent by the client, determining a device type of an input device corresponding to the input operation; the device type of the input device comprising any one of a terminal device corresponding to the client and an external device; according to the device type of the input device, calling a preset type of cloud input method corresponding to the device type of the input device in a corresponding cloud container.

6. The method of claim 5, wherein, the determining of the device type of the input device corresponding to the input operation comprises: acquiring the device type of the current input device sent by the client through a connection established with the client.

7. The method of claim 1, wherein, the method further comprises: acquiring input operation data sent by the client based on a graphical user interface comprising the preset cloud input method; determining output content corresponding to the input operation data according to the input operation data; sending the output content generated according to the input operation data to the client.

8. The method of claim 7, wherein, the acquiring of the input operation data sent by the client based on the graphical user interface comprising the preset cloud input method comprises: Obtaining serialized input operation data sent by the client based on a graphical user interface including the preset cloud input method.

9. The method of claim 8, wherein, The method further comprises: Deserializing the serialized input operation data to determine the output content corresponding to the input operation data.

10. The method of claim 1, wherein, The method further comprises: In response to a received long connection establishment request sent by the client, sending an establishment response corresponding to the long connection establishment request to the client.

11. A cloud-based user input device, comprising: The device applied to a server comprises a deployment module, a calling module, a generation module and a sending module, wherein: The deployment module is configured to deploy a preset cloud input method to the server. The calling module is configured to, in response to a received trigger operation of an input operation sent by a client, call the preset cloud input method corresponding to a device type of an input device based on the device type of the input device. The generation module is configured to generate video stream data including the preset cloud input method according to interface content coding of the preset cloud input method. The sending module is configured to send the video stream data to the client to display video stream data including a preset input window corresponding to the preset cloud input method on the client, the preset input window being a preset virtual keyboard corresponding to the preset cloud input method. The generation module is specifically configured to, in response to the received trigger operation of the input operation, obtain display content in a graphical user interface in the current client; and encode the called preset cloud input method and the display content of the graphical user interface to generate the video stream data including the preset cloud input method.

12. A cloud-based user input device, comprising: The device comprises a processor, a storage medium and a bus, the storage medium stores machine readable instructions executable by the processor, when the cloud-based user input device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine readable instructions to execute the method of any one of claims 1-10.

13. A storage medium, characterized by The storage medium stores a computer program, and the computer program is executed by the processor to execute the method of any one of claims 1-10.

Images