Method and device for synchronizing rendering at pull stream end of cloud desktop and local event

Abstract

The invention relates to a method and a device for synchronizing rendering and local events at a stream pulling end of a cloud desktop, and belongs to the technical field of video rendering and event synchronization, and the method comprises the following steps: registering an event hook for a rendering service; detecting whether an event message needs to be synchronized to a corresponding window; if the detection result is that the event message needs to be synchronized to the corresponding window, the event synchronization service intercepts the event message; the event synchronization service synchronizes the event message to the cloud server and intercepts the event message; and the rendering service receives the video plug flow data of the media forwarding server, and renders the video plug flow data through the rendering service. The event synchronization service is used for intercepting the keyboard and mouse events of the program and synchronizing the keyboard and mouse events to the cloud server at the first time, and the rendering service realizes rendering of the video plug-flow data through cooperation of the video rendering module and the main thread, so that the rendering fluency of the video plug-flow data is optimized, and the synchronization efficiency of the local event is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video rendering and event synchronization, and in particular to a method and device for cloud desktop streaming end rendering and local event synchronization. Background Art

[0002] With the popularization of cloud desktop technology, many manufacturers will provide corresponding services. Optimizing network transmission technology is certainly a feasible direction to improve the cloud desktop experience, but optimizing local strategies can also further improve customers' experience when using cloud desktop clients.

[0003] Existing video rendering and event synchronization are both implemented through message loops, but the rendering fluency is poor and the synchronization efficiency of local events is low. Summary of the invention

[0004] In view of the above deficiencies in the prior art, the purpose of the invention is to provide a method for cloud desktop streaming rendering and local event synchronization and an event synchronization service in the device for intercepting the keyboard and mouse events of the program and synchronizing them to the cloud server as soon as possible. The rendering service realizes the rendering of video streaming data through the cooperation of the video rendering module and the main thread, which not only optimizes the smoothness of the video streaming data rendering, but also improves the synchronization efficiency of local events.

[0005] In a first aspect of the present invention, a method for synchronizing cloud desktop streaming end rendering and local events is proposed, which is applied to a system including an event synchronization service and a rendering service, including:

[0006] Register event hooks for the rendering service;

[0007] Detect whether there are event messages that need to be synchronized to the corresponding window;

[0008] If the detection result is that there is an event message that needs to be synchronized to the corresponding window, the event synchronization service intercepts the event message;

[0009] The event synchronization service synchronizes event messages to the cloud server and intercepts them;

[0010] The rendering service receives the video streaming data from the media forwarding server and renders the video streaming data through the rendering service.

[0011] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, registering an event hook for the rendering service is registered through dll injection technology.

[0012] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, the event hooks at least include: keyboard and mouse hooks.

[0013] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, the event messages at least include: keyboard and mouse messages.

[0014] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, the rendering service receives the video streaming data from the media forwarding server, and the rendering service renders the video streaming data, including:

[0015] The video rendering module in the rendering service receives the video streaming data from the media forwarding server;

[0016] Construct rendering data for video streaming data;

[0017] Synchronize the notification window to refresh the rendering data of the video streaming data;

[0018] Draw the rendering data of the video streaming data through the main thread of the rendering service.

[0019] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, the synchronization notification window refreshes the rendering data of the video streaming data by calling a refresh API.

[0020] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, the rendering data for constructing the video streaming data is constructed using multiple rendering engines.

[0021] In a second aspect of the present invention, a device for synchronizing cloud desktop streaming end rendering and local events is also proposed, which is applied to a system including an event synchronization service and a rendering service, including:

[0022] Registration module: used to register event hooks for rendering services;

[0023] Detection module: used to detect whether there are event messages that need to be synchronized to the corresponding window;

[0024] Synchronization module and interception module: used for event synchronization service to intercept event messages if the detection result shows that there are event messages that need to be synchronized to the corresponding window;

[0025] Interception module: used by the event synchronization service to synchronize event messages to the cloud server and intercept them;

[0026] Rendering module: used for rendering service to receive video streaming data from media forwarding server, and rendering video streaming data through rendering service.

[0027] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0028] The processor calls the program or instruction stored in the memory to execute any of the above methods for cloud desktop streaming end rendering and local event synchronization.

[0029] In a fourth aspect of the present invention, a computer-readable storage medium is further proposed, which stores a program or instruction, and the program or instruction enables a computer to execute any of the above methods for cloud desktop streaming end rendering and local event synchronization.

[0030] The beneficial effects of the present invention are as follows: the present invention registers an event hook for the rendering service; detects whether there is an event message that needs to be synchronized to the corresponding window; if the detection result is that there is an event message that needs to be synchronized to the corresponding window, the event synchronization service intercepts the event message; the event synchronization service synchronizes the event message to the cloud server and intercepts it; the rendering service receives the video push streaming data from the media forwarding server, and renders the video push streaming data through the rendering service. The event synchronization service in the present invention is used to intercept the keyboard and mouse events of the program, and synchronize them to the cloud server at the first time. The rendering service realizes the rendering of the video push streaming data through the cooperation of the video rendering module and the main thread, which not only optimizes the smoothness of the video push streaming data rendering, but also improves the synchronization efficiency of local events. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0032] Figure 1 A method diagram of cloud desktop streaming end rendering and local event synchronization provided by an embodiment of the present invention;

[0033] Figure 2 A method diagram for rendering video streaming data through a rendering service provided by an embodiment of the present invention;

[0034] Figure 3 A diagram of a device for cloud desktop streaming end rendering and local event synchronization provided by an embodiment of the present invention;

[0035] Figure 4 A schematic block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0037] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0038] In the description of the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be internal communication between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0039] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0040] The present invention proposes a method, device, electronic device and storage medium for cloud desktop streaming end rendering and local event synchronization. The event synchronization service is used to intercept the keyboard and mouse events of the program and synchronize them to the cloud server as soon as possible. The rendering service realizes the rendering of video streaming data through the cooperation of the video rendering module and the main thread, which not only optimizes the smoothness of the video streaming data rendering, but also improves the synchronization efficiency of local events.

[0041] Before introducing the embodiments of the present invention, professional terms involved in the present invention are first introduced.

[0042] dll injection: dll injection is an interception technology under Windows, which can intercept Windows process message responses or load dynamic library interfaces, etc.

[0043] Method Embodiment

[0044] Figure 1A method diagram of cloud desktop streaming end rendering and local event synchronization provided by an embodiment of the present invention.

[0045] In a first aspect of the present invention, a method for synchronizing cloud desktop streaming end rendering and local events is proposed, which is applied to a system including an event synchronization service and a rendering service, and combined with Figure 1 , including five steps from S1 to S5:

[0046] S1: Register event hooks for the rendering service.

[0047] Specifically, in the embodiment of the present invention, an event hook is registered for the rendering service through the dll injection technology, and the event hook at least includes: keyboard and mouse hooks.

[0048] S2: Detect whether there is an event message that needs to be synchronized to the corresponding window.

[0049] Specifically, in the embodiment of the present invention, it is detected whether there are keyboard and mouse messages that need to be synchronized to the corresponding window.

[0050] S3: If the detection result is that there is an event message that needs to be synchronized to the corresponding window, the event synchronization service intercepts the event message.

[0051] Specifically, in the embodiment of the present invention, if the detection result is that there are keyboard and mouse messages that need to be synchronized to the corresponding window, the event synchronization service intercepts the keyboard and mouse messages.

[0052] S4: The event synchronization service synchronizes the event message to the cloud server and intercepts it.

[0053] Specifically, in the embodiment of the present invention, the event synchronization service steps the keyboard and mouse messages to the cloud server and intercepts them. It should be understood that the event synchronization service is used to intercept the keyboard and mouse events of the program and synchronize them to the cloud server as soon as possible.

[0054] S5: The rendering service receives the video streaming data from the media forwarding server and renders the video streaming data through the rendering service.

[0055] Specifically, in an embodiment of the present invention, a method for rendering video streaming data through a rendering service is described in detail below.

[0056] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, registering an event hook for the rendering service is registered through dll injection technology.

[0057] Specifically, in an embodiment of the present invention, the dll injection technology is to inject the dynamic link library file into the memory space of the target process, so that the target process can load and execute the code in the dynamic link library file at runtime, and can intercept the windows process message response, or the loaded dynamic library interface, etc.

[0058] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, the event hooks at least include: keyboard and mouse hooks.

[0059] Specifically, in the embodiment of the present invention, the event hook includes keyboard and mouse hooks, but is not limited to keyboard and mouse hooks.

[0060] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, the event messages at least include: keyboard and mouse messages.

[0061] Specifically, in the embodiment of the present invention, the event message includes: keyboard and mouse messages, but is not limited to keyboard and mouse messages.

[0062] Figure 2 A diagram of a method for rendering video streaming data through a rendering service provided in an embodiment of the present invention.

[0063] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, the rendering service receives the video streaming data from the media forwarding server, and renders the video streaming data through the rendering service. Figure 2 , including four steps from S21 to S24:

[0064] S21: The video rendering module in the rendering service receives the video streaming data from the media forwarding server.

[0065] Specifically, in the embodiment of the present invention, the rendering service includes: a video rendering module and a main thread. The video rendering module in the rendering service receives the video streaming data from the media forwarding server.

[0066] S22: Construct rendering data of the video streaming data.

[0067] Specifically, in the embodiment of the present invention, after receiving the video streaming data from the media forwarding server, the video rendering module uses a variety of rendering engines to construct rendering data of the video streaming data.

[0068] S23: Synchronously notify the window to refresh the rendering data of the video streaming data.

[0069] Specifically, in the embodiment of the present invention, after the rendering data of the video streaming data is constructed using multiple rendering engines, the synchronous notification window is refreshed by calling a refresh API.

[0070] S24: Drawing rendering data of the video streaming data through the main thread of the rendering service.

[0071] Specifically, in the embodiment of the present invention, the rendering data of the video streaming data is drawn by the main thread of the rendering service, thereby completing the process of rendering the video data.

[0072] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, the synchronization notification window refreshes the rendering data of the video streaming data by calling a refresh API.

[0073] Specifically, in the embodiment of the present invention, the refresh API is called to synchronously notify the window to refresh the rendering data of the video streaming data, thereby avoiding the refresh message from entering the process message queue and reducing unnecessary loss.

[0074] Furthermore, in the above-mentioned method for synchronizing cloud desktop streaming end rendering and local events, the rendering data for constructing the video streaming data is constructed using multiple rendering engines.

[0075] Specifically, in the embodiment of the present invention, the multiple rendering engines may be various rendering engines, which can be flexibly selected in practice, and the protection scope of the present invention is not limited thereto.

[0076] Device Embodiment

[0077] Figure 3 A diagram of a device for cloud desktop streaming end rendering and local event synchronization provided by an embodiment of the present invention.

[0078] In a second aspect of the present invention, a device for synchronizing cloud desktop streaming end rendering and local events is also proposed, which is applied to a system including an event synchronization service and a rendering service, and is combined with Figure 3 ,include:

[0079] Registration module 31: used to register event hooks for rendering services.

[0080] Specifically, in the embodiment of the present invention, the registration module 31 of the dll injection technology registers event hooks for the rendering service, and the event hooks at least include: keyboard and mouse hooks.

[0081] Detection module 32: used to detect whether there is an event message that needs to be synchronized to the corresponding window.

[0082] Specifically, in the embodiment of the present invention, the detection module 32 detects whether there are keyboard and mouse messages that need to be synchronized to the corresponding window.

[0083] The synchronization module 33 and the interception module 34 are used for the event synchronization service to intercept the event message if the detection result shows that there is an event message that needs to be synchronized to the corresponding window.

[0084] Specifically, in the embodiment of the present invention, if the detection result is that there are keyboard and mouse messages that need to be synchronized to the corresponding window, the synchronization module 33 and the interception module 34 of the event synchronization service intercept the keyboard and mouse messages.

[0085] Interception module 35: used for the event synchronization service to synchronize event messages to the cloud server and intercept them.

[0086] Specifically, in the embodiment of the present invention, the interception module 35 of the event synchronization service synchronizes and intercepts the keyboard and mouse messages to the cloud server. It should be understood that the event synchronization service is used to intercept the keyboard and mouse events of the program and synchronize them to the cloud server as soon as possible.

[0087] Rendering module 36: used for rendering service to receive the video streaming data from the media forwarding server, and rendering the video streaming data through the rendering service.

[0088] Specifically, in the embodiment of the present invention, after the rendering service receives the video streaming data from the media forwarding server, the rendering module 36 of the rendering service renders the video streaming data.

[0089] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0090] The processor calls the program or instruction stored in the memory to execute any of the above methods for cloud desktop streaming end rendering and local event synchronization.

[0091] In a fourth aspect of the present invention, a computer-readable storage medium is further proposed, which stores a program or instruction, and the program or instruction enables a computer to execute any of the above methods for cloud desktop streaming end rendering and local event synchronization.

[0092] Figure 4 It is a schematic block diagram of an electronic device provided by an embodiment of the present invention.

[0093] like Figure 4 As shown, the electronic device includes: at least one processor 401, at least one memory 402 and at least one communication interface 403. The various components in the electronic device are coupled together through a bus system 404. The communication interface 403 is used to transmit information between external devices. It can be understood that the bus system 404 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 404 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 4 Various buses are labeled as bus system 404 .

[0094] It can be understood that the memory 402 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0095] In some implementations, the memory 402 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.

[0096] Among them, the operating system includes various system programs, such as the framework layer, the core library layer, the driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. A program that implements any of the methods of cloud desktop streaming end rendering and local event synchronization provided in an embodiment of the present invention can be included in the application program.

[0097] In an embodiment of the present invention, the processor 401 calls the program or instructions stored in the memory 402, specifically, the program or instructions stored in the application, and the processor 401 is used to execute the steps of each embodiment of a method for cloud desktop streaming end rendering and local event synchronization provided in an embodiment of the present invention.

[0098] Register event hooks for the rendering service;

[0099] Detect whether there are event messages that need to be synchronized to the corresponding window;

[0100] If the detection result is that there is an event message that needs to be synchronized to the corresponding window, the synchronization service intercepts the event message;

[0101] The synchronization service synchronizes event messages to the cloud server and intercepts them;

[0102] The rendering service receives the video streaming data from the media forwarding server and renders the video streaming data through the rendering service.

[0103] Any of the methods for cloud desktop streaming end rendering and local event synchronization provided by an embodiment of the present invention can be applied to the processor 401, or implemented by the processor 401. The processor 401 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 401 or instructions in the form of software. The above-mentioned processor 401 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0104] The steps of any method in the method for cloud desktop streaming end rendering and local event synchronization provided by an embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software units in the decoding processor. The software unit can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 402, and the processor 401 reads the information in the memory 402 and completes the steps of the method in combination with its hardware.

[0105] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0106] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0107] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

[0108] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for synchronizing cloud desktop streaming end rendering and local events, characterized in that: Applicable to systems including event synchronization services and rendering services, including: Register event hooks for the rendering service; Detect whether there are event messages that need to be synchronized to the corresponding window; If the detection result is that there is an event message that needs to be synchronized to the corresponding window, the event synchronization service intercepts the event message; The event synchronization service synchronizes event messages to the cloud server and intercepts them; The rendering service receives the video streaming data from the media forwarding server and renders the video streaming data through the rendering service.

2. According to claim 1, a method for synchronizing cloud desktop streaming end rendering and local events, characterized in that: The event hook registration for the rendering service is registered through the dll injection technology.

3. According to claim 1, a method for synchronizing cloud desktop streaming end rendering and local events, characterized in that: The event hooks at least include: keyboard and mouse hooks.

4. A method for synchronizing cloud desktop streaming end rendering and local events according to claim 1, characterized in that: The event messages at least include: keyboard and mouse messages.

5. A method for synchronizing cloud desktop streaming end rendering and local events according to claim 1, characterized in that: The rendering service receives the video streaming data from the media forwarding server, and renders the video streaming data through the rendering service, including: The video rendering module in the rendering service receives the video streaming data from the media forwarding server; Construct rendering data for video streaming data; Synchronize the notification window to refresh the rendering data of the video streaming data; Draw the rendering data of the video streaming data through the main thread of the rendering service.

6. A method for synchronizing cloud desktop streaming end rendering and local events according to claim 5, characterized in that: The synchronous notification window refreshes the rendering data of the video streaming data by calling a refresh API.

7. A method for synchronizing cloud desktop streaming end rendering and local events according to claim 5, characterized in that: The rendering data for constructing the video streaming data is constructed using a variety of rendering engines.

8. A device for synchronizing cloud desktop streaming end rendering and local events, characterized in that: Applicable to systems including event synchronization services and rendering services, including: Registration module: used to register event hooks for rendering services; Detection module: used to detect whether there are event messages that need to be synchronized to the corresponding window; Synchronization module and interception module: used for event synchronization service to intercept event messages if the detection result shows that there are event messages that need to be synchronized to the corresponding window; Interception module: used by the event synchronization service to synchronize event messages to the cloud server and intercept them; Rendering module: used for rendering service to receive video streaming data from media forwarding server, and rendering video streaming data through rendering service.

9. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute the method of cloud desktop streaming end rendering and local event synchronization as described in any one of claims 1 to 7 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute a method for cloud desktop streaming end rendering and local event synchronization as described in any one of claims 1 to 7.

Images