A cloud container and a human-computer interaction method and device based on the cloud container

Abstract

The present specification discloses a cloud container and a human-computer interaction method and device based on the cloud container. In the embodiments of the present specification, the driver programs for driving various hardware of the device where the client is located are loaded into the cloud container, and virtual interfaces corresponding to these hardware are virtualized in the cloud container. When the cloud container is applied, in response to the triggering of a first interaction event, the corresponding driver program is run to obtain driver information, and the driver information is output through the virtual interface. Once the driver information is output by the virtual interface, the driver information is sent to the client, so that the client drives the corresponding hardware based on the driver program, thereby realizing human-computer interaction. In addition, the method does not need to run an operating system on the cloud container, so it can realize rich interaction while meeting the lightweight requirements of cloud deployment.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a cloud container and a human-computer interaction method and device based on the cloud container. Background Art

[0002] With the development of cloud technology, more and more client computing power is being migrated to the cloud, more and more software is being migrated to the cloud, and even more and more traditional operating systems are being deployed on the cloud for end customers to use. Containers are an infrastructure platform deployed on cloud hosts to provide a secure, trusted, and isolated operating environment for various cloud applications. The above containers are referred to as cloud containers below.

[0003] Generally, applications deployed on the cloud run in the environment provided by the cloud container, and the client can remotely connect to the cloud container to access various cloud applications running in the cloud container.

[0004] The cloud container in the prior art can only provide a very single means of interaction for the client to access the cloud application: the terminal command line. In other words, the cloud container in the prior art can only provide limited cloud computing power as background services for these cloud applications, such as FasS (Functions as a Service), Paas (Platform as a Service), Iaas (Infrastructure as a Service), etc., which also limits the types and forms of cloud applications, which are usually some background service and interface program software. Obviously, this cannot meet the needs of more and more rich interactive application scenarios (the so-called rich interaction refers to a variety of human-computer interaction forms, including but not limited to touch, keyboard and mouse, gestures, face recognition and other inputs, as well as physical outputs such as images, sounds, vibrations, etc.) and related software.

[0005] Although there are cloud-based remote desktop technologies in the prior art that can realize rich interaction of cloud applications, cloud-based remote desktop technologies require installing and running an operating system in a cloud container, and then running cloud applications through the operating system. In other words, the realization of rich interaction of cloud applications cannot be separated from the operating system, which obviously does not meet the lightweight requirements of cloud deployment.

[0006] Therefore, how to achieve rich interaction of cloud applications while meeting the lightweight requirements of cloud deployment has become a problem that needs to be solved urgently. Summary of the invention

[0007] The embodiments of this specification provide a cloud container and a human-computer interaction method and device based on the cloud container to partially solve the problems existing in the above-mentioned prior art.

[0008] The embodiments of this specification adopt the following technical solutions:

[0009] This specification provides a cloud container, the cloud container is used to run a computer program, and the cloud container includes a user-mode microkernel;

[0010] The user-mode microkernel is used to run a driver corresponding to each hardware on the device where the client is located;

[0011] The user-mode microkernel also includes a virtual interface corresponding to each hardware;

[0012] The user-state microkernel determines, in response to a first interaction event from a computer program running on the client or the cloud container, a target hardware corresponding to the first interaction event, runs a driver corresponding to the target hardware to obtain driver information for driving the target hardware, and outputs the driver information through a virtual interface corresponding to the target hardware;

[0013] The cloud container sends the driver information output through the virtual interface corresponding to the target hardware to the client, so that the client drives the target hardware on the device to perform corresponding operations based on the driver information.

[0014] Optionally, the user-state microkernel is also used to, in response to a second interaction event input from the target hardware through a virtual interface corresponding to the target hardware, report the second interaction event to a computer program running on the cloud container through a driver corresponding to the target hardware, so that the computer program processes the second interaction event.

[0015] Optionally, the cloud container further includes a rendering engine;

[0016] The rendering engine responds to a rendering event from a computer program running on the cloud container, generates a rendering instruction according to the graphics to be displayed contained in the rendering event, and performs one of the following operations:

[0017] Sending the rendering instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the graphics to be displayed according to the rendering instruction, and sends the drawn graphics to be displayed to the device where the client is located for display; or

[0018] Sending the rendering instruction to the graphics processing hardware of the device where the client is located, so that the graphics processing hardware of the device draws the graphics to be displayed according to the rendering instruction, so that the device displays the graphics to be displayed; or

[0019] The rendering instruction is split into a first instruction and a second instruction, and the first instruction is sent to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the first part of the graphics to be displayed according to the first instruction, and sends the drawn part of the graphics to be displayed to the device where the client is located, and the second instruction is sent to the graphics processing hardware of the device where the client is located, so that the graphics processing hardware of the device draws the second part of the graphics to be displayed according to the second instruction, so that the device splices and displays the first part and the second part of the graphics to be displayed.

[0020] This specification provides a human-computer interaction method based on a cloud container, wherein the cloud container preloads various drivers, each driver corresponding to each hardware on the device where the client is located; the method includes:

[0021] The cloud container determines, in response to a first interaction event from the client or a computer program running on the cloud container, target hardware corresponding to the first interaction event from hardware corresponding to each pre-loaded driver program;

[0022] Running a driver corresponding to the target hardware, and obtaining driver information output by the running driver for driving the target hardware;

[0023] Outputting the driving information through a preset virtual interface corresponding to the target hardware;

[0024] For a preset virtual interface corresponding to each hardware on the device, when driver information is detected to be output through the virtual interface, the output driver information is sent to the client, so that the client drives the hardware corresponding to the virtual interface on the device to perform corresponding operations based on the received driver information.

[0025] Optionally, the method further comprises:

[0026] The cloud container responds to a second interaction event input from the target hardware through a virtual interface corresponding to the target hardware, and reports the second interaction event to a computer program running on the cloud container through a driver corresponding to the target hardware, so that the second interaction event is processed by the computer program.

[0027] Optionally, the method further comprises:

[0028] The cloud container generates a rendering instruction according to the graphics to be displayed contained in the rendering event in response to a rendering event from a computer program running on the cloud container;

[0029] Based on the rendering instruction, the device is controlled to display the graphics to be displayed.

[0030] Optionally, based on the rendering instruction, controlling the device to display the graphics to be displayed specifically includes:

[0031] Sending the rendering instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the graphics to be displayed according to the rendering instruction;

[0032] The drawn graphics to be displayed are sent to the device for display.

[0033] Optionally, based on the rendering instruction, controlling the device to display the graphics to be displayed specifically includes:

[0034] The rendering instruction is sent to the graphics processing hardware of the device, so that the graphics processing hardware of the device draws the graphics to be displayed according to the rendering instruction, so that the device displays the graphics to be displayed.

[0035] Optionally, based on the rendering instruction, controlling the device to display the graphics to be displayed specifically includes:

[0036] Splitting the rendering instruction into a first instruction and a second instruction;

[0037] Sending the first instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the first part of the graphics to be displayed according to the first instruction, and sends the drawn part of the graphics to be displayed to the device where the client is located;

[0038] The second instruction is sent to the graphics processing hardware of the device where the client is located, so that the graphics processing hardware of the device draws the second part of the graphic to be displayed according to the second instruction, so that the device splices and displays the first part and the second part of the graphic to be displayed.

[0039] This specification provides a human-computer interaction device, the device comprising:

[0040] Loading module, pre-loading various drivers, each driver corresponding to each hardware on the device where the client is located;

[0041] A receiving module, in response to a first interaction event from the client or a computer program running on the device, determines, from among the hardware corresponding to each pre-loaded driver program, a target hardware corresponding to the first interaction event;

[0042] An operation module is used to operate a driver corresponding to the target hardware and obtain driver information output by the operating driver for driving the target hardware;

[0043] An output module, outputting the driving information through a preset virtual interface corresponding to the target hardware;

[0044] The sending module, for a preset virtual interface corresponding to each hardware on the device, sends the output driver information to the client when it is monitored that driver information is output through the virtual interface, so that the client drives the hardware corresponding to the virtual interface on the device to perform corresponding operations based on the received driver information.

[0045] Optionally, the receiving module is also used to, in response to a second interaction event input from the target hardware through a virtual interface corresponding to the target hardware, report the second interaction event to a computer program running on the device through a driver corresponding to the target hardware, so that the second interaction event can be processed by the computer program.

[0046] Optionally, the receiving module is further configured to, in response to a rendering event from a computer program running on the cloud container, generate a rendering instruction according to the graphics to be displayed contained in the rendering event;

[0047] The device also includes:

[0048] A rendering module controls the device to display the graphics to be displayed based on the rendering instruction.

[0049] Optionally, the rendering module sends the rendering instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the graphics to be displayed according to the rendering instruction, and sends the drawn graphics to be displayed to the device for display.

[0050] Optionally, the rendering module sends the rendering instruction to the graphics processing hardware of the device, so that the graphics processing hardware of the device draws the graphics to be displayed according to the rendering instruction, so that the device displays the graphics to be displayed.

[0051] Optionally, the rendering module splits the rendering instruction into a first instruction and a second instruction; sends the first instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the first part of the graphic to be displayed according to the first instruction, and sends the drawn part of the graphic to be displayed to the device where the client is located; sends the second instruction to the graphics processing hardware of the device where the client is located, so that the graphics processing hardware of the device draws the second part of the graphic to be displayed according to the second instruction, so that the device splices and displays the first part and the second part of the graphic to be displayed.

[0052] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the human-computer interaction method based on the cloud container is implemented.

[0053] An electronic device provided in this specification includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned human-computer interaction method based on a cloud container when executing the program.

[0054] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0055] The embodiments of this specification load the drivers for driving various hardware of the device where the client is located into the cloud container, and virtualize virtual interfaces corresponding to these hardware in the cloud container. When the cloud container is applied, in response to the triggering of the first interaction event, the corresponding driver is run to obtain driver information and output it through the virtual interface. Once the virtual interface outputs the driver information, the driver information is sent to the client, so that the client drives the corresponding hardware based on the driver, thereby realizing human-computer interaction. In addition, this method does not need to run an operating system on the cloud container, so it can achieve rich interaction while meeting the lightweight requirements of cloud deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0057] Figure 1 A schematic diagram of the system architecture of a human-computer interaction method based on a cloud container provided in an embodiment of this specification;

[0058] Figure 2 A schematic diagram of a human-computer interaction process based on a cloud container provided in an embodiment of this specification;

[0059] Figure 3 A schematic diagram of the structure of a human-computer interaction device provided in an embodiment of this specification;

[0060] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0061] The so-called human-computer interaction refers to the process in which a user performs an operation on a device, and the device responds to the user's operation and gives the user physical output as feedback. The rich interaction described in the embodiments of this specification refers to the various forms of operations performed by the user on the device, and the various forms of physical output given to the user by the device. Among them, the various forms of operations performed by the user on the device include but are not limited to touch, keyboard and mouse, gestures, and face recognition. The various forms of physical output given to the user by the device include but are not limited to images, sounds, and vibrations.

[0062] Regardless of the form of human-computer interaction, for the device, how to drive the corresponding hardware on the device is the basis for realizing human-computer interaction. To realize human-computer interaction based on the cloud, it is necessary to remotely drive the hardware on the device used by the user through the cloud. For this reason, in the prior art, it is often necessary to run an operating system in a cloud container to realize human-computer interaction, but the operating system does not meet the lightweight requirements of cloud deployment. Therefore, the main idea of ​​the cloud container provided in this specification is to directly run the driver for driving the device used by the user on the cloud container, and virtualize the hardware environment of the device used by the user on the cloud container, so that the driver can run directly on the cloud container and output the driving information, and the cloud container will then send the driving information to the device used by the user, so that the device drives its own hardware based on the received driving information.

[0063] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0064] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0065] Figure 1 A schematic diagram of the system architecture of a human-computer interaction method based on a cloud container provided in an embodiment of this specification includes: a cloud container 100 and a device 200 used by a user.

[0066] The cloud container 100 is an infrastructure platform deployed on a cloud device to provide a secure, trusted and isolated operating environment for computer programs running in the cloud, and can be used to run computer programs. In the embodiments of this specification, the computer program running on the cloud container 100 can be an operating system, a cloud application, or a cloud component.

[0067] Since the user needs to use the device 200 to obtain the services provided by the computer program running on the cloud container 100, the device 200 needs to run a client 201 that can establish a communication connection with the cloud container 100. In addition, the device 200 also includes a number of hardware 202, which can specifically be a keyboard, a mouse, a camera, a linear motor, a touch screen, a speaker, etc. The device 200 described in this specification can be any computing device that can be used by the user, such as a mobile phone, a computer, etc.

[0068] In order to remotely drive the hardware 202 on the device 200 , the cloud container 100 includes a user-mode microkernel 101 . The user-mode microkernel 101 can load and run a driver 1011 corresponding to each hardware 202 on the device 200 .

[0069] Since the cloud container 100 runs a computer program in the cloud, the physical device where the cloud container 100 is located does not actually have the same hardware 202 as the device 200. Therefore, in order to virtualize the hardware environment of the device 200 on the cloud container 100 so that the user-state microkernel 101 can normally run the driver 1011 corresponding to each hardware 202, in the embodiment of this specification, the user-state microkernel 101 also includes a virtual interface 1012 corresponding to each hardware 202.

[0070] The interface described in this specification refers to the interface program, not the physical interface or interface circuit. Since the physical device where the cloud container 100 is located does not actually have each hardware 202, the interface 1012 corresponding to each hardware 202 set on the user-mode microkernel 101 is called a virtual interface 1012.

[0071] based on Figure 1 The system architecture shown in this specification provides a human-computer interaction method based on cloud container, such as Figure 2 shown.

[0072] Figure 2 The schematic diagram of the human-computer interaction process based on the cloud container provided in the embodiment of this specification specifically includes the following steps:

[0073] S200: In response to a first interaction event from the client or a computer program running on the cloud container, the cloud container determines, from among hardware corresponding to each pre-loaded driver program, target hardware corresponding to the first interaction event.

[0074] In the embodiments of this specification, the first interaction event refers to an event that needs to drive the hardware on the device used by the user to perform a corresponding operation. The first interaction event can be generated by the user's operation, that is, the user performs an input operation on the client to generate the first interaction event. Specifically, through the input operation performed by the user on the client, the first interaction event can be directly generated by the client, or sent by the client to the cloud container, and the first interaction event is generated by a computer program running on the cloud container. Regardless of whether the first interaction event is generated by the client or by a computer program running on the cloud container, the first interaction event needs to carry the identifier of the target hardware corresponding to the first interaction event, so that the user-state microkernel determines the target hardware corresponding to the first interaction event based on the identifier.

[0075] For example, when a user clicks a virtual button on the interface displayed by the client, the client can send the user's operation of clicking the virtual button to the cloud container, and the computer program running in the cloud container generates a first interaction event, which can be to make the linear motor on the device where the client is located vibrate in a preset manner to simulate the touch feeling of the user pressing a physical button. At this time, the user state microkernel can respond to the triggering of the first interaction event and determine that the target hardware corresponding to the first interaction event is a linear motor.

[0076] S202: Running a driver program corresponding to the target hardware, and obtaining driver information output by the running driver program for driving the target hardware.

[0077] S204: Outputting the driving information through a preset virtual interface corresponding to the target hardware.

[0078] S206: For a preset virtual interface corresponding to each hardware on the device, when it is monitored that driver information is output through the virtual interface, the output driver information is sent to the client, so that the client drives the hardware corresponding to the virtual interface on the device to perform corresponding operations based on the received driver information.

[0079] Since the hardware environment of the device used by the user needs to be virtualized in the cloud container, the purpose of setting virtual interfaces corresponding to the hardware on the device used by the user in the user state microkernel in this specification is to "deceive" the user state microkernel. From the user state microkernel's point of view, since it is set with virtual interfaces corresponding to the hardware, the user state microkernel will think that these hardware exist on the physical device where it is located. In this way, the corresponding hardware environment is virtualized for the user state microkernel.

[0080] Thus, in step S202, the user-mode microkernel can run the driver according to the first interaction event to obtain the driver information. The driver information may include a specific driving method for driving the target hardware. The first interaction event may include the operation that the target hardware needs to perform.

[0081] Continuing with the above example, since the first interactive event can be to make the linear motor on the device where the client is located vibrate in a preset manner to simulate the touch of the user pressing a physical button, the operation that the linear motor needs to perform is: vibrate in a preset manner. The driving information obtained by the user-mode microkernel running the driver is how to drive the linear motor so that the linear motor can vibrate in a preset manner, such as driving the linear motor to vibrate along the x-axis with intensity q and frequency f.

[0082] Because the virtual interface described in this specification refers to an interface program, when step S204 is specifically executed, the virtual interface of the target hardware can be called, and the driver instructions can be encapsulated based on the virtual interface and the driver information.

[0083] Therefore, in step S206, the cloud container can monitor whether each virtual interface is called. Once the virtual interface is detected to be called, the driver instruction encapsulated by calling the virtual interface can be determined as the output of the virtual interface and sent to the client.

[0084] After receiving the driver instruction, the client can use the driver instruction to directly drive the target hardware of the device where the client is located to perform the corresponding operation, or drive the target hardware to perform the corresponding operation based on the driver instruction through the operating system of the device where the client is located.

[0085] Through the above method, it is possible to remotely drive various hardware on the device where the client is located in the cloud without running an operating system on the cloud container, thereby achieving rich interaction while meeting the lightweight requirements of cloud deployment.

[0086] In addition, for the device used by the user, some hardware on the device can not only be used to provide physical output to the user, but also receive user input, such as keyboard, mouse, touch screen, camera and other hardware. For these hardware, the driver in the user-mode microkernel not only needs to drive these hardware to perform corresponding operations, but also needs to report the user input operations collected by these hardware as second interaction events to the computer program running on the cloud container, and the computer program will process the second interaction event accordingly.

[0087] Specifically, Figure 1The user-mode microkernel is also used to respond to a second interaction event from the target hardware input through the virtual interface corresponding to the target hardware, and report the second interaction event to the computer program running on the cloud container through the driver corresponding to the target hardware, so that the computer program processes the second interaction event. Specifically, the target hardware can call the virtual interface corresponding to the target hardware through the client, and input the second interaction event into the user-mode microkernel through the called virtual interface.

[0088] For example, cloud containers are Figure 2 After the method shown drives the camera on the device used by the user, when the camera captures an image, the client can call the virtual interface corresponding to the camera, and encapsulate the captured image as a second interactive event, which is input into the user-state microkernel of the cloud container through the called virtual interface. Correspondingly, the user-state microkernel of the cloud container reports the second interactive event input through the virtual interface to the computer program running on the cloud container to process the image.

[0089] Furthermore, since one of the more important forms of human-computer interaction is graphic display, such as windows and pages, Figure 1 In the system architecture shown, the cloud container 100 may also include a rendering engine 102, which may be any graphics rendering engine or a 3D rendering engine, such as DircectX or OpenGL.

[0090] For the rendering engine 102, it is not a driver, so it is not necessary to virtualize the corresponding hardware environment for the rendering engine 102 in the cloud container 100. The rendering engine 102 can respond to the rendering event from the computer program running on the cloud container 100, generate a rendering instruction according to the graphics to be displayed contained in the rendering event, and control the device 200 used by the user to display the graphics to be displayed based on the rendering instruction. The rendering instruction is an instruction for controlling the graphics processing hardware to draw the graphics to be displayed.

[0091] Furthermore, the cloud device where the cloud container 100 is located may have graphics processing hardware, such as a graphics processing unit (GPU). Therefore, according to different actual needs, this specification provides the following three methods for drawing graphics to be displayed.

[0092] The first is that the graphics to be displayed are completely drawn by the cloud device. Specifically, the rendering engine 102 can send rendering instructions to the graphics processing hardware of the cloud device, so that the graphics processing hardware of the cloud device draws the graphics to be displayed according to the rendering instructions, and then sends the drawn graphics to be displayed to the device 200 used by the user for display. The cloud device can be the cloud device where the cloud container 100 is located, or it can be other cloud devices in the cloud where the cloud container 100 is located.

[0093] The second method is that the graphics to be displayed are completely drawn by the device 200 used by the user. Specifically, the rendering engine 102 can send a rendering instruction to the graphics processing hardware of the device 200 used by the user, so that the graphics processing hardware of the device 200 used by the user draws the graphics to be displayed according to the rendering instruction, so that the device 200 used by the user displays the graphics to be displayed.

[0094] The third type is that a part of the graphics to be displayed is drawn by the cloud device, and the other part is drawn by the device 200 used by the user. Specifically, the rendering engine 102 splits the rendering instruction into a first instruction and a second instruction, sends the first instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the first part of the graphics to be displayed according to the first instruction, and sends the drawn first part of the graphics to be displayed to the device 200 used by the user; sends the second instruction to the graphics processing hardware of the device 200 used by the user, so that the graphics processing hardware of the device 200 used by the user draws the second part of the graphics to be displayed according to the second instruction, so that the device 200 used by the user splices and displays the first part and the second part of the graphics to be displayed.

[0095] Through the above method, the computing power used to draw graphics can be flexibly allocated between the device 200 used by the user and the cloud. The specific method used to draw the graphics depends on the computer program running on the cloud container 100, and this specification does not limit this.

[0096] In summary, the user-mode microkernel 101 in the cloud container 100 can realize human-computer interaction based on the cloud container, and the rendering engine 102 can realize the view output of the cloud container. Using these two, multiple functions can be realized. For example, a window / user interface (UI) system component ( Figure 1 (not shown in the figure), the window / UI system component can use the user-mode microkernel 101 to drive the keyboard and mouse of the device 200 used by the user, and use the rendering engine 102 to make the device 200 used by the user display the UI, so that the user can perform keyboard and mouse operations through the UI.

[0097] The above are the cloud container and the human-computer interaction method based on the cloud container provided in the embodiments of this specification. Based on the same idea, this specification also provides corresponding devices, storage media and electronic devices.

[0098] Figure 3 A schematic diagram of the structure of a human-computer interaction device provided in an embodiment of this specification, wherein the device comprises:

[0099] A loading module 301 preloads various drivers, each driver corresponding to each hardware on the device where the client is located;

[0100] The receiving module 302 determines, in response to an interaction event from the client or a computer program running on the device, target hardware corresponding to the first interaction event from hardware corresponding to each pre-loaded driver program;

[0101] The running module 303 runs the driver corresponding to the target hardware and obtains the driver information output by the running driver for driving the target hardware;

[0102] The output module 304 outputs the driving information through a preset virtual interface corresponding to the target hardware;

[0103] The sending module 305, for a preset virtual interface corresponding to each hardware on the device, when it is monitored that driver information is output through the virtual interface, sends the output driver information to the client, so that the client drives the hardware corresponding to the virtual interface on the device to perform corresponding operations based on the received driver information.

[0104] Optionally, the receiving module 302 is also used to, in response to a second interaction event input from the target hardware through a virtual interface corresponding to the target hardware, report the second interaction event to a computer program running on the device through a driver corresponding to the target hardware, so that the second interaction event can be processed by the computer program.

[0105] Optionally, the receiving module 302 is further configured to, in response to a rendering event from a computer program running on the cloud container, generate a rendering instruction according to the graphics to be displayed contained in the rendering event;

[0106] The device also includes:

[0107] The rendering module 306 controls the device to display the graphics to be displayed based on the rendering instruction.

[0108] Optionally, the rendering module 306 sends the rendering instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the graphics to be displayed according to the rendering instruction, and sends the drawn graphics to be displayed to the device for display.

[0109] Optionally, the rendering module 306 sends the rendering instruction to the graphics processing hardware of the device, so that the graphics processing hardware of the device draws the graphics to be displayed according to the rendering instruction, so that the device displays the graphics to be displayed.

[0110] Optionally, the rendering module 306 splits the rendering instruction into a first instruction and a second instruction; sends the first instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the first part of the graphic to be displayed according to the first instruction, and sends the drawn first part of the graphic to be displayed to the device where the client is located; sends the second instruction to the graphics processing hardware of the device where the client is located, so that the graphics processing hardware of the device draws the second part of the graphic to be displayed according to the second instruction, so that the device splices and displays the first part and the second part of the graphic to be displayed.

[0111] This specification also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can be used to execute the human-computer interaction method based on the cloud container provided above.

[0112] Based on the human-computer interaction method based on cloud container provided above, this specification embodiment also provides Figure 4 The structural diagram of the electronic device shown in FIG. Figure 4 At the hardware level, the unmanned device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned human-computer interaction method based on the cloud container.

[0113] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0114] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0115] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0116] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0117] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0118] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0119] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0123] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0124] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0125] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0126] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0128] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0129] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A cloud container, characterized in that: The cloud container is used to run a computer program, and the cloud container includes a user-mode microkernel; The user-mode microkernel is used to run a driver corresponding to each hardware on the device where the client is located; The user-mode microkernel also includes a virtual interface corresponding to each hardware; The user-state microkernel determines the target hardware corresponding to the first interaction event in response to the first interaction event from the computer program running on the client or the cloud container, and runs a driver corresponding to the target hardware to obtain driver information for driving the target hardware, and outputs the driver information through a virtual interface corresponding to the target hardware; wherein the first interaction event of the computer program running on the cloud container is a first interaction event generated by the computer program running on the cloud container based on an operation of a user sent by a client to the cloud container; The cloud container sends the driver information output through the virtual interface corresponding to the target hardware to the client, so that the client drives the target hardware on the device to perform corresponding operations based on the driver information.

2. The cloud container according to claim 1, wherein: The user-state microkernel is also used to, in response to a second interaction event input from the target hardware through a virtual interface corresponding to the target hardware, report the second interaction event to a computer program running on the cloud container through a driver corresponding to the target hardware, so that the computer program processes the second interaction event.

3. The cloud container according to claim 1, wherein: The cloud container also includes a rendering engine; The rendering engine responds to a rendering event from a computer program running on the cloud container, generates a rendering instruction according to the graphics to be displayed contained in the rendering event, and performs one of the following operations: Sending the rendering instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the graphics to be displayed according to the rendering instruction, and sends the drawn graphics to be displayed to the device where the client is located for display; or Sending the rendering instruction to the graphics processing hardware of the device where the client is located, so that the graphics processing hardware of the device draws the graphics to be displayed according to the rendering instruction, so that the device displays the graphics to be displayed; or The rendering instruction is split into a first instruction and a second instruction, and the first instruction is sent to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the first part of the graphics to be displayed according to the first instruction, and sends the drawn part of the graphics to be displayed to the device where the client is located, and the second instruction is sent to the graphics processing hardware of the device where the client is located, so that the graphics processing hardware of the device draws the second part of the graphics to be displayed according to the second instruction, so that the device splices and displays the first part and the second part of the graphics to be displayed.

4. A human-computer interaction method based on cloud container, characterized in that: The cloud container preloads various drivers, each driver corresponding to each hardware on the device where the client is located; the method includes: The cloud container determines, in response to a first interaction event from the client or a computer program running on the cloud container, target hardware corresponding to the first interaction event among hardware corresponding to each pre-loaded driver; wherein the first interaction event of the computer program running on the cloud container is a first interaction event generated by the computer program running on the cloud container based on an operation of a user sent by the client to the cloud container; Running a driver corresponding to the target hardware, and obtaining driver information output by the running driver for driving the target hardware; Outputting the driving information through a preset virtual interface corresponding to the target hardware; For a preset virtual interface corresponding to each hardware on the device, when driver information is detected to be output through the virtual interface, the output driver information is sent to the client, so that the client drives the hardware corresponding to the virtual interface on the device to perform corresponding operations based on the received driver information.

5. The method according to claim 4, characterized in that The method further comprises: The cloud container responds to a second interaction event input from the target hardware through a virtual interface corresponding to the target hardware, and reports the second interaction event to a computer program running on the cloud container through a driver corresponding to the target hardware, so that the second interaction event is processed by the computer program.

6. The method according to claim 4, characterized in that The method further comprises: The cloud container generates a rendering instruction according to the graphics to be displayed contained in the rendering event in response to a rendering event from a computer program running on the cloud container; Based on the rendering instruction, the device is controlled to display the graphics to be displayed.

7. The method according to claim 6, characterized in that Based on the rendering instruction, controlling the device to display the graphics to be displayed specifically includes: Sending the rendering instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the graphics to be displayed according to the rendering instruction; The drawn graphics to be displayed are sent to the device for display.

8. The method according to claim 6, characterized in that Based on the rendering instruction, controlling the device to display the graphics to be displayed specifically includes: The rendering instruction is sent to the graphics processing hardware of the device, so that the graphics processing hardware of the device draws the graphics to be displayed according to the rendering instruction, so that the device displays the graphics to be displayed.

9. The method according to claim 6, characterized in that Based on the rendering instruction, controlling the device to display the graphics to be displayed specifically includes: Splitting the rendering instruction into a first instruction and a second instruction; Sending the first instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the first part of the graphics to be displayed according to the first instruction, and sends the drawn first part of the graphics to be displayed to the device where the client is located; The second instruction is sent to the graphics processing hardware of the device where the client is located, so that the graphics processing hardware of the device draws the second part of the graphic to be displayed according to the second instruction, so that the device splices and displays the first part and the second part of the graphic to be displayed.

10. A human-computer interaction device, characterized in that: The device comprises: Loading module, pre-loading various drivers, each driver corresponding to each hardware on the device where the client is located; A receiving module, in response to a first interaction event from the client or the computer program running on the device, determines the target hardware corresponding to the first interaction event in the hardware corresponding to each pre-loaded driver; wherein the first interaction event of the computer program running on the device is a first interaction event generated by the computer program running on the device based on the user operation sent by the client to the device; An operation module is used to operate a driver corresponding to the target hardware and obtain driver information output by the operating driver for driving the target hardware; An output module, outputting the driving information through a preset virtual interface corresponding to the target hardware; The sending module, for a preset virtual interface corresponding to each hardware on the device, sends the output driver information to the client when it is detected that driver information is output through the virtual interface, so that the client drives the hardware corresponding to the virtual interface on the device to perform corresponding operations based on the received driver information.

11. The device according to claim 10, characterized in that The receiving module is also used to, in response to a second interaction event input from the target hardware through a virtual interface corresponding to the target hardware, report the second interaction event to a computer program running on the device through a driver corresponding to the target hardware, so that the second interaction event can be processed by the computer program.

12. The device according to claim 10, characterized in that The receiving module is further used to, in response to a rendering event from a computer program running on the device, generate a rendering instruction according to the graphics to be displayed contained in the rendering event; The device also includes: A rendering module controls the device to display the graphics to be displayed based on the rendering instruction.

13. The device according to claim 12, characterized in that The rendering module sends the rendering instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the graphics to be displayed according to the rendering instruction, and sends the drawn graphics to be displayed to the device for display.

14. The device according to claim 12, characterized in that The rendering module sends the rendering instruction to the graphics processing hardware of the device, so that the graphics processing hardware of the device draws the graphics to be displayed according to the rendering instruction, so that the device displays the graphics to be displayed.

15. The device according to claim 12, characterized in that The rendering module splits the rendering instruction into a first instruction and a second instruction; sends the first instruction to the graphics processing hardware in the cloud, so that the graphics processing hardware in the cloud draws the first part of the graphics to be displayed according to the first instruction, and sends the drawn first part of the graphics to be displayed to the device where the client is located; The second instruction is sent to the graphics processing hardware of the device where the client is located, so that the graphics processing hardware of the device draws the second part of the graphic to be displayed according to the second instruction, so that the device splices and displays the first part and the second part of the graphic to be displayed.

16. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 4 to 9 is implemented.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 4 to 9 is implemented.

Images