Cursor display method, device, medium and electronic device based on remote desktop
By capturing and sending mouse displacement values and absolute coordinate proportion data in remote desktop technology, the problem of high cursor control delay is solved, and a smoother cloud gaming operation experience is achieved.
Patent Information
- Application Number
- CN202110158150.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-02-04
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-02-04
AI Technical Summary
The existing remote desktop technology has high latency in cursor control, resulting in poor cloud gaming operations and inability to achieve localized experience effects.
By capturing the relative displacement value of the system mouse and sending it to the remote server, the absolute coordinate proportion data is obtained and sent. The client draws the cursor at the remote desktop window position based on this data, avoiding the cursor data encoding between the server and the client and Decoding delay.
Reduces cursor manipulation delay, improves operational fluency, and makes the user experience closer to localization.
Smart Images

Figure CN113741744B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of remote desktop, and in particular to a cursor display method, device, medium and electronic equipment based on remote desktop. Background Art
[0002] Cloud gaming is a gaming method based on cloud computing. In the cloud gaming mode, all games are run on the server side, and the entire game is played with players through the Internet. That is, users access the server locally through various smart devices through remote control protocols, and control the remote desktop provided by the remote server through various input devices. Compared with using remote control technology for work, using remote control technology to control cloud games requires higher latency data and smoothness, especially in cursor control. If the cursor latency is high and the movement is not smooth, cloud gaming cannot truly achieve a localized experience. Summary of the invention
[0003] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a cursor display method, device, medium and electronic device based on a remote desktop, so as to solve the above-mentioned shortcomings in the prior art.
[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a cursor display method based on a remote desktop, which is applied to a client that displays a remote desktop. The method includes: capturing the relative displacement value of the system mouse movement and sending it to a remote server that provides the remote desktop, so that the remote server can perform corresponding mouse movements; obtaining the ratio data of the absolute coordinates of the mouse after the remote server moves relative to the screen resolution of the remote server provided by the remote server; and drawing the moved cursor at the corresponding position of the remote desktop window on the client according to the ratio data and the position and size of the remote desktop window displayed by the client.
[0005] In one embodiment of the present invention, the relative displacement value is captured in a manner including: an event-triggered acquisition manner and / or a timed polling acquisition manner.
[0006] In one embodiment of the present invention, if the client includes: a mobile app, a computer application client, a web client, and an h5 client; drawing the moved cursor at a corresponding position of the remote desktop window on the client specifically includes: for the mobile app and the h5 client, directly drawing the moved cursor at a corresponding position of the remote desktop window; for the computer application client and the web client, first hiding the local cursor, and then drawing the moved cursor at a corresponding position of the remote desktop window.
[0007] To achieve the above-mentioned purpose and other related purposes, the present invention provides a cursor display method based on a remote desktop, which is applied to a remote server that provides a remote desktop. The method includes: receiving a relative displacement value of a cursor movement sent by a client that displays the remote desktop, and performing a corresponding mouse movement operation based on the relative displacement value; periodically capturing the mouse of the remote server, and sending proportional data of the absolute coordinates of the moved mouse relative to the screen resolution of the remote server to the client, so that the client can draw the moved cursor at a corresponding position of the remote desktop window on the client according to the proportional data and the position and size of the remote desktop window displayed by the client.
[0008] In one embodiment of the present invention, the timing of sending the proportional data includes: calculating and sending the proportional data when the system mouse movement is detected, or, before the cursor moves again, sending the proportional data of the last cursor movement to the client at a preset frequency.
[0009] In one embodiment of the present invention, if the absolute coordinates of the mouse after the movement are (k, l), and the screen resolution of the remote server is c*d, then the ratio data is (k / c, l / d).
[0010] To achieve the above-mentioned purpose and other related purposes, the present invention provides a cursor display device based on a remote desktop, which is applied to a user device that displays a remote desktop. The device includes: a data capture module, which is used to capture the relative displacement value of the system mouse movement and send it to a remote server that provides the remote desktop, so that the remote server can perform corresponding mouse movements accordingly; a data receiving module, which is used to obtain the ratio data of the absolute coordinates of the mouse after the remote server moves relative to the screen resolution of the remote server provided by the remote server; and a cursor display module, which is used to draw the moved cursor at the corresponding position of the remote desktop window on the client according to the ratio data and the position and size of the remote desktop window displayed by the client.
[0011] To achieve the above-mentioned purpose and other related purposes, the present invention provides a cursor display device based on a remote desktop, which is applied to a remote device that provides a remote desktop. The device includes: a relative displacement receiving module, which is used to receive the relative displacement value of the mouse movement sent by the client that displays the remote desktop, and perform the corresponding mouse movement operation accordingly; a proportional data sending module, which is used to periodically capture the mouse of the remote server and send the proportional data of the absolute coordinates of the moved mouse relative to the screen resolution of the remote server to the client, so that the client can draw the moved cursor at the corresponding position of the remote desktop window on the client according to the proportional data and the position and size of the remote desktop window displayed by the client.
[0012] To achieve the above-mentioned purpose and other related purposes, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is loaded and executed by a processor, the cursor display method based on a remote desktop is implemented.
[0013] To achieve the above-mentioned purpose and other related purposes, the present invention provides an electronic device, characterized in that it includes: a processor and a memory; wherein the memory is used to store a computer program; and the processor is used to load and execute the computer program so that the electronic device executes the cursor display method based on the remote desktop.
[0014] As described above, the cursor display method, device, medium and electronic device based on remote desktop of the present invention realize the transformation from the traditional server-side cursor alignment to the client-side cursor alignment, and can make cloud games or other cloud applications develop towards localization. Compared with the conventional implementation method in which the server captures audio, video and cursor data, encodes them and transmits them to the client for decoding, the present invention eliminates the need to process cursor data simultaneously with audio and video data, thus saving the delay caused by server-side encoding and client-side decoding data, and because the single cursor data packet is small, the impact of data packet loss is reduced. Through the technical solution of the present invention, when users use the cursor to control cloud games or other cloud applications, the operation is smooth and the delay is low, the operation effect can gradually tend to be localized, and the user experience is better. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 Shown is a flowchart of a cursor display method based on a remote desktop in one embodiment of the present invention.
[0016] Figure 2 Shown is a module diagram of a cursor display device based on a remote desktop in one embodiment of the present invention.
[0017] Figure 3 Shown is a structural diagram of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION
[0018] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0019] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic manner, and thus the illustrations only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0020] like Figure 1 As shown, this embodiment provides a cursor display method based on remote desktop, which can meet the higher requirements of various delay data and smoothness in controlling cloud games or other cloud applications using remote control technology, so that the delay in mouse control is lower and the mouse movement is smoother, thereby enabling cloud games or other cloud applications to achieve a localized experience effect.
[0021] The cursor display method based on the remote desktop of this embodiment includes the following steps:
[0022] S1: The client captures the relative displacement value of the system mouse movement and sends it to the remote server providing the remote desktop, so that the remote server can perform the corresponding mouse movement accordingly;
[0023] Specifically, the user's client captures the system mouse movement operation data through the operating system API (for example, the web application uses the JS API). The capture methods of the system mouse movement operation data include: event trigger acquisition method, timed polling acquisition method, etc. For example: when a reliable protocol is used to transmit data, an event trigger acquisition method is used, and when an unreliable protocol is used, a timed polling acquisition method is used. The system mouse movement operation data at least includes the relative displacement value of the mouse movement. A coordinate system is established according to the system resolution. When the system mouse moves, the relative displacement value (m, n) of the current mouse position compared to the previous position is calculated according to the coordinate system.
[0024] S2: The remote server receives the relative displacement value of the mouse movement sent by the client that displays the remote desktop, and performs a corresponding mouse movement operation accordingly;
[0025] Specifically, when the remote server receives the relative displacement value (m, n), it converts the relative displacement value (m, n) into the corresponding movement amount of the mouse movement event on the client on the remote server according to the remote desktop window resolution displayed on the client and the remote server desktop resolution, and immediately performs the corresponding operation. For example: if the client window resolution is a*b and the remote server desktop resolution is c*d, the remote server relative displacement value is (c / a*m, d / b*n). At this time, the mouse of the remote server moves according to (c / a*m, d / b*n), thereby performing the mouse movement operation.
[0026] S3: The remote server captures the mouse of the remote server, and sends the ratio data of the absolute coordinates of the moved mouse relative to the screen resolution of the remote server to the client;
[0027] Specifically, after executing the mouse movement operation, the remote server immediately captures the mouse image data through the graphics operation instructions of the operating system, obtains the absolute coordinates (k, l) of the mouse of the remote server through the hook, calculates its proportional data (k / c, l / d) relative to the screen resolution of the remote server, and sends it to the client synchronously. Optionally, the remote server can send proportional data to the client by capturing and sending when new mouse movement data is generated, or by sending the last captured cursor data to the client at a predetermined frequency before new data is generated.
[0028] S4: The client receives the ratio data of the absolute coordinates of the mouse after the remote server moves relative to the screen resolution of the remote server, provided by the remote server;
[0029] S5: The client draws the moved cursor at a position corresponding to the remote desktop window on the client according to the ratio data and the position and size of the remote desktop window displayed on the client.
[0030] Specifically, after receiving the mouse data (mouse image data, proportional data) sent by the remote server, the client confirms the window position and size of the remote desktop window currently displayed on the client, takes a vertex (x, y) of the display window of the remote desktop on the user's local device as the origin, confirms the window size according to the horizontal and vertical sizes of the window, and when the remote desktop window is in full screen, the system resolution is the resolution of the remote desktop window, and maps the proportional data of the received absolute coordinates of the mouse of the remote server relative to the screen resolution of the remote server. The received mouse image data is processed differently according to the push methods of different types of mice, so that the mouse of the remote server can be drawn at the corresponding position (k / c*a, l / d*b) of the remote desktop window of the client, so that the client cursor is consistent with the remote server mouse.
[0031] In addition, for different clients, such as computer application clients and web page clients, the local cursor must be hidden before drawing and displaying the cursor; for mobile apps and h5 clients, since there is no local cursor, they can directly draw and display the cursor data when it is received.
[0032] All or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. Based on such understanding, the present invention also provides a computer program product, including one or more computer instructions. The computer instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state hard disk (SSD)), etc.
[0033] See also Figure 2 This embodiment provides a cursor display device 21 based on a remote desktop and a cursor display device 22 based on a remote desktop. Since the technical principle of this embodiment is similar to the technical principle of the aforementioned method embodiment, the same technical details will not be repeated.
[0034] The cursor display device 21 based on remote desktop is installed in a user device, such as a smart phone, a tablet computer, etc., and specifically includes: a data capture module 21a, a data receiving module 21b, and a cursor display module 21c. The data capture module 21a is used to execute step S1 of the aforementioned method embodiment, the data receiving module 21b is used to execute step S4 of the aforementioned method embodiment, and the cursor display module 21c is used to execute step S5 of the aforementioned method embodiment.
[0035] The cursor display device 22 based on the remote desktop is installed in a remote device, such as a single physical server, a server group, etc., and specifically includes: a relative displacement receiving module 22a and a proportional data sending module 22b. The relative displacement receiving module 22a is used to execute step S2 of the aforementioned method embodiment, and the proportional data sending module 22b is used to execute step S3 of the aforementioned method embodiment.
[0036] The remote device is used to provide a remote desktop to the user device, and the user device displays the remote desktop.
[0037] Those skilled in the art should understand that Figure 2 The division of each module in the embodiment is only a division of logical functions, and in actual implementation, all or part of it can be integrated into one or more physical entities. Moreover, these modules can be implemented in the form of software calling through processing elements, or in the form of hardware, or some modules can be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware.
[0038] See also Figure 3This embodiment provides an electronic device, which may be a desktop computer, a portable computer, a tablet computer, a smart phone, etc. In detail, the electronic device at least includes: a memory and a processor connected via a bus, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to execute all or part of the steps executed by the client in the above method embodiment, or all or part of the steps executed by the remote server.
[0039] The system bus mentioned above can be a Peripheral Pomponent Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage.
[0040] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0041] In summary, the cursor display method, device, medium and electronic device based on remote desktop of the present invention reduce the impact of mouse movement data loss on movement smoothness, improve the mouse experience of playing 3D games through remote desktop under weak network conditions, effectively overcome the various shortcomings of the prior art and have high industrial utilization value.
[0042] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.
Claims
1. A cursor display method based on a remote desktop, characterized in that: Applied to a client for displaying a remote desktop, the method comprises: Capturing the relative displacement value of the system mouse movement and sending it to the remote server providing the remote desktop, so that the remote server can perform the corresponding mouse movement accordingly; Acquire the ratio data of the absolute coordinates of the mouse after the remote server moves relative to the screen resolution of the remote server, provided by the remote server; Drawing the moved cursor at a corresponding position of the remote desktop window on the client according to the ratio data and the position and size of the remote desktop window displayed on the client; After the client receives the mouse data sent by the remote server, the mouse data includes the scale data and the mouse image data captured by the remote server at regular intervals through the graphics operation instructions of the operating system after the remote server performs the mouse movement operation, According to the position and size of the remote desktop window currently displayed on the client, a vertex (x, y) of the remote desktop display window on the user's local device is used to confirm the window position and set it as the origin. The window size is determined based on the horizontal and vertical dimensions of the window. When the remote desktop window is in full screen, the system resolution is the resolution of the remote desktop window. The received absolute coordinates of the mouse on the remote server are mapped to the proportional data of the screen resolution of the remote server. The received mouse image data is processed differently according to the push mode of different types of mice, so that the mouse of the remote server can be drawn at the corresponding position of the remote desktop window of the client.
2. The cursor display method according to claim 1, characterized in that: The relative displacement value capture method includes: event triggering acquisition method and / or timing polling acquisition method.
3. The cursor display method according to claim 1, characterized in that: If the client includes: a mobile app, a computer application client, a web client, or an h5 client; drawing the moved cursor at a position corresponding to the remote desktop window on the client specifically includes: For the mobile app and h5 client, the moved cursor is directly drawn at the corresponding position of the remote desktop window; For the computer application client and the web client, the local cursor is first hidden, and then the moved cursor is drawn at the corresponding position of the remote desktop window.
4. A cursor display method based on a remote desktop, characterized in that: Applied to a remote server providing a remote desktop, the method comprises: Receiving a relative displacement value of a mouse movement sent by a client that displays the remote desktop, and executing a corresponding mouse movement operation accordingly; Capturing the mouse of the remote server, and sending the scale data of the absolute coordinates of the moved mouse relative to the screen resolution of the remote server to the client, so that the client can draw the moved cursor at the corresponding position of the remote desktop window on the client according to the scale data, the position and size of the remote desktop window displayed on the client; After the client receives the mouse data sent by the remote server, the mouse data includes the scale data and the mouse image data captured by the remote server at regular intervals through the graphics operation instructions of the operating system after the remote server performs the mouse movement operation, According to the position and size of the remote desktop window currently displayed on the client, a vertex (x, y) of the remote desktop display window on the user's local device is used to confirm the window position and set it as the origin. The window size is determined based on the horizontal and vertical dimensions of the window. When the remote desktop window is in full screen, the system resolution is the resolution of the remote desktop window. The received absolute coordinates of the mouse on the remote server are mapped to the proportional data of the screen resolution of the remote server. The received mouse image data is processed differently according to the push mode of different types of mice, so that the mouse of the remote server can be drawn at the corresponding position of the remote desktop window of the client.
5. The cursor display method according to claim 4, characterized in that: The timing of sending the proportional data includes: capturing the absolute coordinates of the cursor when an operation of the system mouse is detected, calculating and sending the proportional data.
6. The cursor display method according to claim 4, characterized in that: If the absolute coordinates of the cursor after the movement are (k, l), and the screen resolution of the remote server is c*d, then the proportional data is (k / c, l / d).
7. A cursor display device based on a remote desktop, characterized in that: A user device for displaying a remote desktop, the device comprising: A data capture module, used for capturing the relative displacement value of the system mouse movement and sending it to the remote server providing the remote desktop, so that the remote server can perform the corresponding mouse movement accordingly; A data receiving module, used for acquiring the ratio data of the absolute coordinates of the mouse after the remote server moves relative to the screen resolution of the remote server, provided by the remote server; A cursor display module, used for drawing the moved cursor at a position corresponding to the remote desktop window on the client according to the ratio data, the position and size of the remote desktop window displayed on the client; After the client receives the mouse data sent by the remote server, the mouse data includes the scale data and the mouse image data captured by the remote server at regular intervals through the graphics operation instructions of the operating system after the remote server performs the mouse movement operation, According to the position and size of the remote desktop window currently displayed on the client, a vertex (x, y) of the remote desktop display window on the user's local device is used to confirm the window position and set it as the origin. The window size is determined based on the horizontal and vertical dimensions of the window. When the remote desktop window is in full screen, the system resolution is the resolution of the remote desktop window. The received absolute coordinates of the mouse on the remote server are mapped to the proportional data of the screen resolution of the remote server. The received mouse image data is processed differently according to the push mode of different types of mice, so that the mouse of the remote server can be drawn at the corresponding position of the remote desktop window of the client.
8. A cursor display device based on a remote desktop, characterized in that: A remote device for providing a remote desktop, the device comprising: A relative displacement receiving module, used for receiving a relative displacement value of a mouse movement sent by a client that displays the remote desktop, and performing a corresponding mouse movement operation accordingly; A proportional data sending module, used for capturing the mouse of the remote server, and sending proportional data of the absolute coordinates of the moved mouse relative to the screen resolution of the remote server to the client, so that the client can draw the moved cursor at a corresponding position of the remote desktop window on the client according to the proportional data and the position and size of the remote desktop window displayed on the client; After the client receives the mouse data sent by the remote server, the mouse data includes the scale data and the mouse image data captured by the remote server at regular intervals through the graphics operation instructions of the operating system after the remote server performs the mouse movement operation, According to the position and size of the remote desktop window currently displayed on the client, a vertex (x, y) of the remote desktop display window on the user's local device is used to confirm the window position and set it as the origin. The window size is determined based on the horizontal and vertical dimensions of the window. When the remote desktop window is in full screen, the system resolution is the resolution of the remote desktop window. The received absolute coordinates of the mouse on the remote server are mapped to the proportional data of the screen resolution of the remote server. The received mouse image data is processed differently according to the push mode of different types of mice, so that the mouse of the remote server can be drawn at the corresponding position of the remote desktop window of the client.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is loaded and executed by the processor, the cursor display method based on the remote desktop as described in any one of claims 1 to 6 is implemented.
10. An electronic device, characterized in that: include: processor and memory; wherein, The memory is used to store computer programs; The processor is used to load and execute the computer program so that the electronic device executes the cursor display method based on remote desktop as described in any one of claims 1 to 6.
