Writing display method and device, writing response method and device, terminal equipment and server

Abstract

The embodiment of the invention discloses a writing display method, a writing response method and device, terminal equipment and a server, a cloud desktop client installed in the terminal equipment displays a running interface of a cloud desktop determined by a cloud desktop server during running, and a virtual operating system of the cloud desktop runs in a virtual machine of the cloud desktop server. The writing display method comprises the following steps: when a writing operation of a cloud desktop is detected, reporting a writing event generated in real time in an operation process of the writing operation to a cloud desktop server; receiving a writing state generated and fed back by the cloud desktop server according to the received first writing event; and acquiring a real-time writing event according to the writing state, and drawing a corresponding first writing track according to the writing event. By adopting the method, the technical problem that response delay exists when the terminal equipment displays the writing track when writing is carried out in the cloud desktop in the related technology can be solved.

Description

Writing display method, writing response method and device, terminal equipment and server Technical Field

[0001] The embodiments of the present application relate to the field of cloud desktop technology, and in particular to a writing display method, a writing response method and apparatus, a terminal device, and a server. Background Art

[0002] Cloud desktop, also known as desktop virtualization or cloud computer, is a new model that replaces traditional computers. Unlike traditional desktops, the operating system of the cloud desktop is installed on a remote host machine (such as a server), and data processing and storage are also completed remotely. The local terminal device communicates with the remote host machine over the network to obtain the cloud desktop and display it, as well as receive the local user's control of the cloud desktop and send it to the remote host machine for response. For example, when a local user writes in the cloud desktop, the remote host machine is required to draw the corresponding writing track based on the user's writing, and send the picture containing the writing track to the local terminal device in real time, and the local terminal device displays the picture containing the writing track. This method has relatively high requirements for the network. When the network status between the host machine and the local terminal device is not good, such as when there are abnormal conditions such as delays, packet loss, and jitter, it will cause the local terminal device to display the picture containing the writing track with obvious response delay.

[0003] Summary of the Invention

[0004] The embodiments of the present application provide a writing display method, a writing response method and apparatus, a terminal device, and a server to solve the technical problem in related technologies of response delay in displaying a writing track on a terminal device when writing in a cloud desktop.

[0005] In a first aspect, an embodiment of the present application provides a writing display method, which is applied to a terminal device, wherein the terminal device is installed with a cloud desktop client, and the cloud desktop client displays the operation interface of the cloud desktop when running, and the operation interface is determined by the cloud desktop server. The writing display method includes:

[0006] When a writing operation of the cloud desktop is detected, a writing event generated in real time based on the writing operation is reported to the cloud desktop server;

[0007] receiving a writing status fed back by the cloud desktop server, where the writing status is generated by the cloud desktop server according to a first received writing event;

[0008] According to the writing state, a real-time writing event is acquired, and a corresponding first writing track is drawn according to the writing event.

[0009] In one embodiment of the present application, acquiring a real-time writing event according to the writing state, and drawing a corresponding first writing trajectory according to the writing event, includes:

[0010] Generate a writing layer according to the writing state, and overlay the writing layer on the current running interface of the cloud desktop;

[0011] The writing layer monitors the real-time writing events of the cloud desktop, and draws a corresponding first writing track in the writing layer according to the writing events.

[0012] In one embodiment of the present application, the receiving of the writing status feedback from the cloud desktop server further includes:

[0013] receiving a trajectory drawing color fed back by the cloud desktop server, where the trajectory drawing color is determined by the cloud desktop server;

[0014] Drawing a corresponding first writing trajectory according to the writing event includes:

[0015] The track color of the first writing track to be drawn is set as the track drawing color, and the first writing track corresponding to the writing event is drawn according to the track drawing color.

[0016] In one embodiment of the present application, the receiving of the writing status feedback from the cloud desktop server further includes:

[0017] receiving writing coordinates fed back by the cloud desktop server, where the writing coordinates are writing coordinates obtained by the cloud desktop server based on all currently received writing events;

[0018] The acquiring of a real-time writing event according to the writing state, and drawing a corresponding first writing trajectory according to the writing event, includes:

[0019] According to the writing state, a real-time writing event is acquired, and a corresponding first writing trajectory is drawn based on the writing event and the writing coordinates.

[0020] In one embodiment of the present application, after acquiring a real-time writing event according to the writing state and drawing a corresponding first writing trajectory according to the writing event, the method further includes:

[0021] When it is detected that the writing event stops being generated, the drawing of the corresponding first writing track is stopped; or,

[0022] When receiving the stop writing status fed back by the cloud desktop server, stop drawing the corresponding first writing track.

[0023] In one embodiment of the present application, after reporting the writing event generated in real time based on the writing operation to the cloud desktop server, the method further includes:

[0024] receiving drawing data sent by the cloud desktop server, where the drawing data includes a second writing trajectory drawn by the cloud desktop server according to the writing event;

[0025] According to the drawing data, the current running interface of the cloud desktop is updated, and the drawn first writing track is covered on the second writing track in the running interface.

[0026] In one embodiment of the present application, the terminal device and the cloud desktop server perform data transmission based on the SPICE protocol.

[0027] In one embodiment of the present application, the cloud desktop adopts a VDI technology architecture.

[0028] In a second aspect, an embodiment of the present application provides a writing response method, which is applied to a cloud desktop server, wherein the operating system running in the virtual machine of the cloud desktop server is a virtual operating system of the cloud desktop, and the cloud desktop server sends the running interface of the cloud desktop to the terminal device, and the cloud desktop client installed on the terminal device displays the running interface during operation; the writing response method includes:

[0029] receiving a writing event reported by the terminal device, wherein the writing event is a writing event generated in real time by the terminal device when detecting a writing operation of the cloud desktop;

[0030] When determining, based on the first received writing event, that writing on the cloud desktop occurs on the terminal device, generating a writing state;

[0031] The writing state is fed back to the terminal device, so that the terminal device obtains a real-time writing event according to the writing state, and draws a corresponding first writing track according to the writing event.

[0032] In one embodiment of the present application, when determining, based on the first received writing event, that writing on the cloud desktop occurs on the terminal device, generating a writing state includes:

[0033] Acquire corresponding writing coordinates and cursor coordinates according to the first received writing event;

[0034] When it is determined according to the writing coordinates and the cursor coordinates that the terminal device has written on the cloud desktop, a writing state is generated.

[0035] In one embodiment of the present application, the cursor coordinates include a cursor offset and a cursor physical position.

[0036] The generating of a writing state when determining, based on the writing coordinates and the cursor coordinates, that the terminal device has written on the cloud desktop comprises:

[0037] Calculating the difference between the writing coordinates and the physical position of the cursor;

[0038] When the difference is equal to the cursor offset, it is determined that the terminal device has written on the cloud desktop, and a writing state is generated.

[0039] In one embodiment of the present application, when determining, based on the first received writing event, that writing on the cloud desktop occurs on the terminal device and generating a writing state, the process further includes:

[0040] Determining the color of the track being drawn during writing according to the writing event;

[0041] When the writing status is fed back to the terminal device, the method further includes:

[0042] The trajectory drawing color is fed back to the terminal device, so that the terminal device draws the first writing trajectory according to the trajectory drawing color.

[0043] In one embodiment of the present application, determining the color of the trajectory drawn during writing according to the writing event includes:

[0044] According to the writing event, corresponding texture information is obtained;

[0045] The trajectory drawing color is obtained according to the texture information.

[0046] In one embodiment of the present application, after receiving the writing event reported by the terminal device, the method further includes:

[0047] drawing a second writing trajectory according to the writing event;

[0048] The drawing data containing the second writing track is sent to the terminal device, so that the terminal device updates the current running interface of the cloud desktop according to the drawing data, and the first writing track drawn in the terminal device is covered on the second writing track in the running interface.

[0049] In one embodiment of the present application, after receiving the writing event reported by the terminal device, the method further includes:

[0050] When the writing event is no longer received, a stop writing state is generated;

[0051] The stop writing state is fed back to the terminal device, so that the terminal device stops drawing the corresponding first writing track.

[0052] In a third aspect, an embodiment of the present application further provides a terminal device, the terminal device comprising one or more processors and a display screen, the processor running a cloud desktop client installed on the terminal device, the display screen displaying a cloud desktop operation interface when the cloud desktop client is running, the operation interface being determined by a cloud desktop server,

[0053] The processor reports a writing event generated in real time based on detecting a writing operation of the cloud desktop to the cloud desktop server, obtains the real-time writing event according to a writing state fed back by the cloud desktop server, and draws a corresponding first writing track according to the writing event, wherein the writing state is generated by the cloud desktop server according to a first received writing event;

[0054] The display screen displays the first writing track after drawing in the running interface.

[0055] In one embodiment of the present application, the processor obtains a real-time writing event based on the writing status fed back by the cloud desktop server, and draws a corresponding first writing trajectory based on the writing event, including:

[0056] The processor generates a writing layer according to the writing status fed back by the cloud desktop server, and instructs the display screen to display the writing layer, where the writing layer is covered on the current operation interface of the cloud desktop;

[0057] The processor controls the writing layer to monitor real-time writing events of the cloud desktop, and draws a corresponding first writing track in the writing layer according to the writing event.

[0058] In one embodiment of the present application, when the processor receives the writing status fed back by the cloud desktop server, it also receives the track drawing color fed back by the cloud desktop server, where the track drawing color is determined by the cloud desktop server;

[0059] When the processor draws the corresponding first writing track according to the writing event, the processor further includes:

[0060] The processor sets the track color of the first writing track to be drawn as the track drawing color, and draws the first writing track corresponding to the writing event according to the track drawing color.

[0061] In one embodiment of the present application, when the processor receives the writing status fed back by the cloud desktop server, it receives the writing coordinates fed back by the cloud desktop server, where the writing coordinates are the writing coordinates obtained by the cloud desktop server based on all currently received writing events;

[0062] When the processor draws a corresponding first writing track according to the writing event, the method includes:

[0063] The processor draws a corresponding first writing trajectory based on the writing event and the writing coordinates.

[0064] In one embodiment of the present application, after the processor reports a writing event generated in real time based on detecting a writing operation of the cloud desktop to the cloud desktop server, the process further includes:

[0065] The processor receives drawing data sent by the cloud desktop server, where the drawing data includes a second writing trajectory drawn by the cloud desktop server according to the writing event;

[0066] The processor updates the current operation interface of the cloud desktop according to the drawing data;

[0067] The display screen displays the updated running interface, and the displayed first writing track covers the second writing track in the running interface.

[0068] In a fourth aspect, an embodiment of the present application further provides a writing response device, which is applied to a cloud desktop server, wherein the operating system running in the virtual machine of the cloud desktop server is a virtual operating system of the cloud desktop, and the cloud desktop server sends the running interface of the cloud desktop to the terminal device, and the cloud desktop client installed on the terminal device displays the running interface during operation; the writing response device includes:

[0069] A first receiving unit is configured to receive a writing event reported by the terminal device, wherein the writing event is a writing event generated in real time by the terminal device when a writing operation of the cloud desktop is detected;

[0070] A first generating unit is configured to generate a writing state when determining that writing for the cloud desktop occurs on the terminal device according to the first received writing event;

[0071] The first feedback unit is configured to feed back the writing state to the terminal device, so that the terminal device obtains a real-time writing event according to the writing state and draws a corresponding first writing track according to the writing event.

[0072] In a fifth aspect, an embodiment of the present application further provides a server, wherein the server is a cloud desktop server, and the server includes:

[0073] one or more processors;

[0074] a memory for storing one or more programs;

[0075] When the one or more programs are executed by the one or more processors, the one or more processors implement the writing response method as described in the second aspect.

[0076] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the writing display method as described in the first aspect or the writing response method as described in the second aspect.

[0077] In one embodiment of the present application, a cloud desktop client installed on a terminal device displays the operating interface of the cloud desktop, which is generated by a cloud desktop server and sent to the terminal device. When the terminal device detects a writing operation of the cloud desktop, it reports the writing event generated in real time during the operation to the cloud desktop server. After that, after receiving the writing status feedback from the cloud desktop server based on the first received writing event, it starts to obtain real-time writing events and draws the corresponding first writing track based on the writing event. This technical means realizes the drawing and rendering of the writing track related to the writing operation of the cloud desktop (i.e., the first writing track) on the hardware of the terminal device, which can speed up the display speed of the writing track, reduce the delay caused by network transmission, and improve the timeliness of the response when the user writes in the cloud desktop. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] FIG1 is a flow chart of a writing display method provided by one embodiment of the present application;

[0079] FIG2 is a schematic diagram of a writing trajectory provided by an embodiment of the present application;

[0080] FIG3 is a schematic diagram of another writing trajectory provided by an embodiment of the present application;

[0081] FIG4 is a flow chart of a writing display method provided by one embodiment of the present application;

[0082] FIG5 is a flowchart of a writing response method provided by one embodiment of the present application;

[0083] FIG6 is a first writing response data flow diagram provided by one embodiment of the present application;

[0084] FIG7 is a second writing response data flow diagram provided by one embodiment of the present application;

[0085] FIG8 is a schematic diagram of a first drawing screen provided by an embodiment of the present application;

[0086] FIG9 is a schematic diagram of a second drawing screen provided by one embodiment of the present application;

[0087] FIG10 is a schematic diagram of a command structure of a qxl virtual graphics card provided by one embodiment of the present application;

[0088] FIG11 is a diagram of a writing and drawing framework provided by one embodiment of the present application;

[0089] FIG12 is a schematic structural diagram of a terminal device provided by one embodiment of the present application;

[0090] FIG13 is a schematic structural diagram of a writing response device provided by one embodiment of the present application;

[0091] FIG14 is a schematic diagram of the structure of a cloud desktop server provided by one embodiment of the present application;

[0092] FIG15 is a network diagram of a cloud desktop writing process provided by one embodiment of the present application;

[0093] FIG16 is a schematic diagram of a cloud desktop writing interaction provided by one embodiment of the present application;

[0094] FIG17 is a schematic diagram of a cloud desktop writing interaction provided by an embodiment of the present application. DETAILED DESCRIPTION

[0095] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended to explain the present application, not to limit the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present application, not all structures.

[0096] VDI (Virtual Desktop Infrastructure) is a virtual desktop infrastructure with centralized storage and computing, and is a type of cloud desktop technology. VDI's technical architecture centralizes all data processing for user desktops (i.e., cloud desktops) on servers, with users receiving only the operating system environment's screen.

[0097] In the VDI cloud desktop (i.e., a cloud desktop using the VDI technology architecture) scenario, multiple operating systems can be run on a server (i.e., a host machine) through virtualization technology. For example, operating systems such as Windows and Ubuntu can be run on the server through virtualization technology to implement cloud desktops for Windows and Ubuntu systems. Among them, Ubuntu is a Linux system that mainly uses desktop applications. The terminal device used by the user is connected to the virtual operating system on the server through an internally installed client. After that, the terminal device displays the screen of the virtual operating system, that is, the running screen of the cloud desktop is displayed on the terminal device for user operation. Afterwards, the data generated by the user's operation is sent by the terminal device to the server so that the server's virtual operating system responds and the response screen is fed back to the terminal device for display.

[0098] Among them, virtualization technology refers to the operation of computing elements on a virtual basis rather than a real basis. Virtualization technology can expand the capacity of hardware and simplify the software reconfiguration process. The operating system installed on the physical machine of the server can be recorded as Host OS. This operating system is the operating system used by the server. It can be considered as an operating system that is actually installed and exists in the server. This operating system can be a Linux system or other operating systems. Install virtualization software on this operating system, so that several partitions can be virtualized on the physical machine to install different operating systems respectively. At this time, the operating system installed in the virtualized partition can be considered a virtual operating system. In one embodiment, the virtual operating system in the virtualized partition is the virtual operating system used by the cloud desktop. Each virtual operating system is implemented through a corresponding virtual machine. A virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. Currently, a virtual operating system can be recorded as Guest OS.

[0099] When a terminal device connects to the virtual operating system on a server using a cloud desktop client installed on it, the client uses a VDI protocol such as Spice or RDP. Spice is an open-source virtual desktop transmission protocol, and RDP is a remote display protocol. In one embodiment, the Spice protocol is used as an example for description. In this case, the client used by the terminal device can be considered a Spice client.

[0100] On the server, processor emulation software (QEMU) is used to create multiple virtual machines (VMs). Each VM can run a virtual operating system and is configured with a QXL virtual graphics card (QXL stands for virtual video memory). The open-source SPICE protocol uses the VM's QXL virtual graphics driver to encode the VM's image (i.e., the cloud desktop image) and send it to the terminal device, which decodes and displays the image.

[0101] In this way, the server and terminal devices need to perform multiple encoding, decoding, and data transmission. This makes the VDI cloud desktop very demanding on the network environment and can easily cause display delays on the terminal device. Especially when the user writes in the cloud desktop through the terminal device, the display delay of the writing track will be very obvious.

[0102] It can be understood that when a user writes in the VDI cloud desktop displayed on the terminal device, writing-related events are sent to the corresponding Guest OS on the server via the spice protocol. After receiving the event, the Guest OS draws the writing trajectory and encodes the writing trajectory into an image or a video stream. After encoding it into a video stream, it is sent to the terminal device via the spice protocol. After receiving the image or video stream, the terminal device decodes and renders it and displays it on the current interface so that the user can see the writing response result. Since the user's writing process is a real-time process, the response to the writing should also be real-time. That is, the writing response has very high real-time requirements. However, the writing response method mentioned above is very dependent on network performance, which makes the VDI cloud desktop respond to writing with a high response delay, which seriously affects the user experience.

[0103] Based on this, an embodiment of the present application provides a writing display method, a writing response method and device, a terminal device and a server to solve the technical problem of response delay in displaying writing tracks on a terminal device when writing in a VDI cloud desktop. By using the technical means of drawing writing tracks in the cloud desktop by the terminal device, the dependence of the writing response process on the network is reduced, the delay in displaying the writing tracks is reduced, the display speed of the writing tracks is increased, and the user experience is improved.

[0104] An embodiment of the present application provides a writing display method, which can be executed by a writing display device. The writing display device can be implemented through software and / or hardware. The writing display device can be composed of two or more physical entities or one physical entity.

[0105] In one embodiment, the writing display device is a terminal device, which can be an interactive tablet, a mobile phone, a laptop computer, a tablet computer, or other device. In one embodiment, the description is based on an interactive tablet as an example. An interactive tablet is an integrated device that uses touch technology to control content displayed on the display tablet and implement human-computer interaction. It integrates one or more functions such as a projector, an electronic whiteboard, a screen, audio equipment, a television, and a video conferencing terminal.

[0106] The interactive tablet is installed with at least one type of operating system, and the operating system can be an Android system, a Windows system, or a Linux system, etc. In one embodiment, the operating system is described as an Android system, and the operating system can be understood as an operating system that is actually installed and exists in the interactive tablet. The interactive tablet installs at least one application under the operating system. Among them, the application can be an application that comes with the operating system, or an application downloaded from a third-party device or server, which is not currently limited. The interactive tablet is also configured with at least one display screen. In one embodiment, the display screen is a large-size display screen (such as a display screen larger than 40 inches). The display screen may or may not have a touch function. When the display screen has a touch function, the interactive tablet is also configured with a device for receiving touch operations. Based on the detection type of the touch operation, the device can be a capacitive touch detection device, a resistive touch detection device, an infrared touch detection device, and / or an electromagnetic touch detection device. The user can control the interactive tablet by touching the display screen with a finger or a stylus (also known as a touch pen, a writing pen, a smart pen, etc.), such as writing on the interactive tablet. In addition, the interactive tablet can be connected to external devices such as a mouse and keyboard, allowing users to control the interactive tablet by operating the external devices, such as writing on the interactive tablet with a mouse. Optionally, the interactive tablet is also equipped with a network interface, through which the interactive tablet implements communication functions, for example, communicating data with a server through the network interface.

[0107] In one embodiment, the terminal device is installed with a cloud desktop client, which displays the running interface of the cloud desktop when the cloud desktop client is running. The running interface is determined by the cloud desktop server.

[0108] Exemplarily, a terminal device has an application (i.e., spiceAPP) installed in its operating system as a spice client, which connects to the virtual operating system of the cloud desktop in the server through the spice client. Currently, the spice client is referred to as a cloud desktop client. After running the cloud desktop client, the terminal device can display the effect of running the cloud desktop. Exemplarily, the terminal device displays the cloud desktop's operating interface by running the cloud desktop client. This operating interface can be the desktop interface of the cloud desktop or the current application interface of an application installed in the virtual operating system where the cloud desktop resides. The operating interface is determined by the cloud desktop server. Exemplarily, the server that provides cloud desktop services to the terminal device is referred to as the cloud desktop server. The cloud desktop server includes at least one virtual operating system, each of which runs in a corresponding virtual machine and corresponds to a cloud desktop. Exemplarily, after the cloud desktop client in the terminal device runs, it connects to the virtual operating system running in the cloud desktop server, which is the operating system used by the cloud desktop. The cloud desktop's virtual operating system determines the cloud desktop's operating interface. Thereafter, the cloud desktop server instructs the terminal device to display the operating interface so that the user of the terminal device can clearly understand the current operating status of the cloud desktop. After the terminal device and the cloud desktop server cooperate, the writing display method is implemented. FIG1 is a flow chart of a method for displaying writing provided by an embodiment of the present application. Referring to FIG1 , when a terminal device executes the method for displaying writing, the method includes the following steps:

[0109] Step 110: When a writing operation of the cloud desktop is detected, a writing event generated in real time based on the writing operation is reported to the cloud desktop server.

[0110] Currently, the cloud desktop in the terminal device adopts the VDI technology architecture. In one embodiment, the running interface of the cloud desktop is currently the application interface of the application with writing function in the cloud desktop, and the user of the terminal device can write in the running interface to realize writing on the cloud desktop.

[0111] In one embodiment, when a user writes, the terminal device detects the user's writing operation and generates a writing event. Here, a writing operation refers to an operation performed when writing, and a writing event refers to an event related to writing. For example, when the terminal device has a touch function, the user can write with a finger or a stylus. When the terminal device detects that the finger or stylus is touching the display screen, it can be considered that a writing operation is detected. At this time, the touch event generated based on the writing operation can be used as a writing event. Moreover, while the finger or stylus is touching the display screen and moving, the terminal device continues to detect the writing operation, generates a touch event in real time based on the writing operation and uses it as a writing event. When it is detected that the user or stylus leaves the display screen, it is determined to stop generating the writing event. At this time, the device that detects touch (such as a module for receiving touch operations) can be considered a writing device. For another example, when the user is writing with a mouse, when the terminal device detects the mouse input signal (such as when the user writes by pressing the left mouse button and moving the mouse, the left button is pressed and moved), it can be considered that the writing operation is detected and a mouse event is generated. The mouse event can be used as a writing event. Similarly, when the user continues to operate the mouse, the terminal device continues to generate writing events, and stops generating writing events when it detects that the mouse no longer inputs signals. At this time, the mouse can be considered a writing device. It should be noted that writing events are real-time, that is, during the user's writing process, the terminal device can generate writing events in real time based on the current real-time detected writing operation, and along with the operation process of the writing operation, writing events are continuously generated. After the writing operation stops, writing events are no longer generated. It should be noted that one writing operation can generate a writing track. When the user writes multiple writing tracks, multiple writing operations need to be performed.

[0112] When displaying the operating interface of the cloud desktop, the terminal device can determine that the currently received operation is an operation for the cloud desktop. Therefore, the event generated according to the operation can be considered as an event for the cloud desktop and needs to be reported to the cloud desktop server for response. In one embodiment, when the terminal device displays the operating interface of the cloud desktop and detects a writing operation, it is determined that a writing operation for the cloud desktop (i.e., an operation of writing in the cloud desktop) is detected and a corresponding writing event is generated. It should be noted that since the user is writing in the cloud desktop, and the cloud desktop is responded to by the cloud desktop server, when the terminal device detects a writing operation, it can only determine that the user has performed an operation on the cloud desktop, but cannot determine that the operation is a writing operation. That is, the writing event currently generated based on the writing operation is only used to notify the cloud desktop server: the user of the terminal device has performed an operation on the cloud desktop in the terminal device. As to whether the operation performed by the user is a writing operation, the cloud desktop server needs to make further judgments.

[0113] Exemplarily, during the writing operation, the terminal device reports the writing events generated in real time to the cloud desktop server, so that the cloud desktop server can determine whether the user is writing based on the writing events. In addition, the cloud desktop server can also determine the user's writing trajectory based on the writing events. It can be understood that during the writing process, the continuously generated writing events are reported to the cloud desktop server in real time. In one embodiment, the terminal device and the cloud desktop server perform data transmission based on the spice protocol, that is, the terminal device sends the writing events to the virtual operating system of the cloud desktop in the cloud desktop server through the spice protocol. After receiving the writing events, the virtual operating system performs a subsequent response.

[0114] Step 120: Receive the writing status fed back by the cloud desktop server. The writing status is generated by the cloud desktop server according to the first writing event received.

[0115] In one embodiment, when a user writes a track, the terminal device continuously generates writing events and continuously reports real-time writing events to the virtual operating system of the cloud desktop server. After receiving the writing events, the virtual operating system distributes the real-time writing events to the application currently responding to the writing event in the virtual operating system (the application has a writing function and the application is currently running). After receiving the first writing event, the application begins to determine whether the user is writing in the cloud desktop of the terminal device based on the first writing event. Optionally, the application with a writing function can draw the corresponding writing track in the QXL virtual graphics card of the virtual machine where the virtual operating system is located. At this time, after receiving the first writing event, the application sends the coordinates of the writing operation corresponding to the writing event (currently recorded as writing coordinates) to the QXL virtual graphics card, and the QXL virtual graphics card determines whether writing is being performed. The rules used to determine whether writing is being performed are not currently limited. For example, it is determined whether the writing coordinates are within the writing area. If so, it is determined that writing is being performed; otherwise, it is determined that no writing is being performed. For another example, it is determined whether the user is currently in writing mode. If so, it is determined that writing is being performed; otherwise, it is determined that no writing is being performed. For another example, the cursor coordinates are obtained, and whether writing is in progress is determined based on the cursor coordinates and the writing coordinates. The cursor includes a display cursor and an implicit cursor, and there are many types of cursors, such as a pointer, a hand shape, an hourglass, etc. The cursor coordinates include the physical position of the cursor and the cursor offset (also known as the cursor hotpot). The physical position of the cursor is obtained through the underlying layer of the virtual operating system. The physical position of the cursor is the coordinate of the cursor in the display screen (i.e., the pixel coordinate). The cursor offset is obtained from the graphical environment interface of the virtual operating system, which is equivalent to the offset between the center position of the cursor and the starting position of the cursor. When the user is writing, the difference between the writing coordinates and the physical position of the cursor should be equal to the cursor offset. Therefore, based on the writing coordinates and the cursor coordinates, it can be determined whether writing is currently occurring. For another example, after determining that the user is in writing mode, it can be determined whether writing is currently occurring based on the writing coordinates and the cursor coordinates.

[0116] Exemplarily, after determining that a user is writing in a cloud desktop on a terminal device, the virtual operating system of the cloud desktop server generates a writing status and feeds the writing status back to the terminal device. The writing status is used to notify the terminal device that the current user is writing in the cloud desktop. In one embodiment, the virtual operating system feeds the writing status back to the terminal device via the spice protocol.

[0117] Optionally, while a user is writing a track, the virtual operating system continues to receive writing events. Based on the first writing event received, the virtual operating system determines whether writing is in progress. Subsequent writing events are not re-determined, but instead respond based solely on the determination of the first writing event.

[0118] Optionally, when the user writes a track, the virtual operating system continues to receive writing events, and if no writing events are received, it is determined that the writing is stopped. Thereafter, when the user writes a new track, the above process is repeated.

[0119] Optionally, when the virtual operating system determines that the user is not writing in the cloud desktop of the terminal device, it will no longer respond to the writing. At this time, the virtual operating system can further determine whether the writing event can trigger other functions. If it can trigger other functions, it will respond to other functions.

[0120] Step 130: Acquire a real-time writing event according to the writing state, and draw a corresponding first writing trajectory according to the writing event.

[0121] For example, when the terminal device determines that the user is writing based on the writing status, it begins directly acquiring writing events for the cloud desktop. Specifically, after a writing event is generated, the terminal device directly acquires and processes the writing event. When processing the writing event, the terminal device acquires the writing coordinates corresponding to the writing event and draws the writing track corresponding to the writing event based on the writing coordinates. The writing track is rendered and displayed during drawing so that the user can see it. Currently, the writing track drawn by the terminal device is recorded as the first writing track.

[0122] It is understandable that the current operating interface of the cloud desktop is generated by the cloud desktop server. Therefore, the terminal device cannot directly render the first writing track it draws on the operating interface of the cloud desktop. In this case, when the terminal device draws the corresponding first writing track based on the acquired writing event, it can generate a transparent writing layer and cover the operating interface of the cloud desktop, and draw the first writing track in the transparent writing layer. In this case, step 130 may include steps 131 and 132:

[0123] Step 131: Generate a writing layer according to the writing status, and overlay the writing layer on the current running interface of the cloud desktop.

[0124] The writing layer is a transparent display layer. It is understandable that the screen displayed by the terminal device can be composed of at least one display layer superimposed, and each display layer can be operated separately. After the terminal device receives the writing status, a new writing layer is added. The writing layer is a transparent display layer, and the size of the writing layer is the same as the size of the cloud desktop's operating interface. The writing layer covers the cloud desktop's operating interface. It is understandable that since the writing layer is transparent, from the user's perspective, the writing layer will not be seen, and the cloud desktop's operating interface will continue to be seen.

[0125] In one embodiment, the operating system of the terminal device is the Android system. After the terminal device receives the writing status, it adds an Overlay writing layer (i.e., the writing layer) through the system service that manages the display. Among them, the system service that manages the display is the underlying display service SurfaceFlinger of the Android system. SurfaceFlinger can manage the display and refresh of the Overlay writing layer. The SurfaceFlinger service process uses the GL interface provided by MESA. Among them, MESA is a computer graphics library that implements multiple graphics API specifications, and specifically implements the OpenGL application program interface (i.e., GL interface) in an open source form.

[0126] Step 132: The writing layer monitors the real-time writing events of the cloud desktop, and draws a corresponding first writing track in the writing layer according to the writing events.

[0127] Exemplarily, the writing layer is overlaid on the running interface of the cloud desktop, and the current input event (the current input event is a writing event) can be directly obtained through the input device of the terminal device (currently a writing device). That is, after the writing layer is generated, the writing event currently generated by the terminal device is simultaneously obtained by the writing layer and the cloud desktop server.

[0128] After the writing layer detects the writing event, it can autonomously draw the writing based on the writing event, thereby enabling the terminal device to draw the first writing track. It should be noted that the process of drawing the first writing track by the writing layer can be considered as a writing redirection process, that is, redirecting the writing and drawing originally in the virtual machine of the cloud desktop server to the terminal device.

[0129] It is understandable that the terminal device starts to obtain writing events after receiving the writing status. Therefore, the terminal device cannot obtain writing events that have been generated before receiving the writing status. That is, since the writing layer is generated after receiving the writing status, the writing layer can only monitor writing events generated after receiving the writing status. The cloud desktop server has received the writing event when determining whether writing is in progress (the current cloud desktop server uses the first writing event received when determining whether writing is in progress), and the writing event will not be obtained by the writing layer. Therefore, there will be a difference between the starting position when the writing layer draws the writing trajectory and the starting position when the user actually writes. This difference is generally small and has little impact on the writing content that the user wants to present. In one embodiment, in order to make the starting position of the first writing track drawn by the writing layer the same as the starting position when the user actually writes, when the writing layer starts drawing, in addition to monitoring the current writing event, it can also obtain the writing coordinates corresponding to the writing event that has been generated by this writing operation from the cloud desktop server (that is, when the cloud desktop server feeds back the writing status to the terminal device, all the writing events (generally the first writing event) that have been received) Afterwards, the writing layer can also draw the first writing track based on the writing coordinates obtained from the cloud desktop server. At this time, when the terminal device receives the writing status fed back by the cloud desktop server, it also includes: receiving the writing coordinates fed back by the cloud desktop server, which are the writing coordinates obtained by the cloud desktop server based on all the currently received writing events. Accordingly, according to the writing status, real-time writing events are obtained, and the corresponding first writing track is drawn according to the writing event, including: according to the writing status, real-time writing events are obtained, and the corresponding first writing track is drawn based on the writing event and the writing coordinates.

[0130] For example, when the cloud desktop server reports the writing status, it also reports the writing coordinates corresponding to the currently received writing event to the terminal device. Generally speaking, the currently received writing event is the first writing event. After the cloud desktop server receives the writing event, the application responding to the writing event can determine the corresponding writing coordinates based on the writing event. It is understood that each writing event has corresponding writing coordinates.

[0131] When the terminal device receives the writing status, it also receives the writing coordinates. At this time, when the terminal device draws the first writing track according to the writing status, in addition to drawing according to the acquired writing event, it also draws according to the writing coordinates received when receiving the writing status, to ensure that the terminal device starts drawing according to the first writing event, thereby ensuring that the starting position of the first writing track drawn by the terminal device is the same as the starting position when the user actually writes. Optionally, during the user's writing process, the cloud desktop server can also feed back the writing coordinates of the received writing event to the terminal device in real time, that is, when the terminal device draws the first writing track, it can continuously obtain the writing coordinates fed back by the cloud desktop server. At this time, the terminal device can determine the writing delay when the cloud desktop server responds to the writing event based on the writing coordinates of the writing event currently acquired by itself and the writing coordinates fed back by the cloud desktop server. In an optional manner, when the cloud desktop server feeds back the writing coordinates corresponding to the writing event acquired in real time to the terminal device, the terminal device can also draw the first writing track based on the writing coordinates.

[0132] Optionally, when a writing track is completed, the terminal device no longer generates a writing event. At this point, the writing layer of the terminal device cannot obtain the writing event. Afterwards, the writing layer can stop drawing the first writing track, that is, the terminal device stops drawing the first writing track. Based on this, in one embodiment, after step 130, the further step includes: when monitoring that the writing event stops being generated, stopping drawing the corresponding first writing track.

[0133] For example, taking the example of drawing the first writing track through the writing layer, at this time, during the user's writing process, the writing layer continues to obtain writing events generated on the terminal device and for the cloud desktop. If the writing layer no longer obtains writing events, it is determined that the writing event is no longer generated, that is, the user stops writing. At this time, the writing layer stops drawing the first writing track and no longer continues to listen for writing events for the cloud desktop.

[0134] Optionally, when a writing track is completed, the terminal device no longer generates a writing event. In this case, the cloud desktop server does not receive any writing events and can notify the terminal device to stop drawing the first writing track. Based on this, in one embodiment, after step 130, the process further includes: upon receiving a stop writing status signaled by the cloud desktop server, stopping drawing the corresponding first writing track.

[0135] For example, when the cloud desktop server no longer receives writing events, it determines that the writing has stopped. At this time, the cloud desktop server generates a stop writing status, which is used to notify the terminal device to stop writing. The cloud desktop server then feeds the stop writing status back to the terminal device. The stop writing status and the writing status are transmitted in the same manner and are not described in detail here. When the terminal device receives the stop writing status, it stops drawing the first writing track, that is, the writing layer stops receiving writing events.

[0136] In the above, when the terminal device detects a writing operation for the cloud desktop, it reports the writing event generated in real time during the operation to the cloud desktop server. After that, after receiving the writing status fed back by the cloud desktop server based on the first received writing event, it starts to obtain real-time writing events and draws the corresponding first writing track based on the writing event. This technical means solves the technical problem of response delay in the display of writing tracks by the terminal device when writing in the cloud desktop in the related art. Through the computing resources and rendering capabilities of the terminal device, it is achieved that the writing track (i.e., the first writing track) related to the writing operation of the cloud desktop is drawn and rendered on the hardware of the terminal device, which can speed up the display speed of the writing track, reduce the delay caused by network transmission, and improve the timeliness of the response when the user writes in the cloud desktop. In addition, by adding a transparent writing layer, the writing track is drawn in the terminal device, making the process of drawing the writing track relatively simple and easy to implement. In addition, when the user finishes writing, the terminal device can stop drawing the writing track, which also ensures the timeliness of the writing response. Especially in the cloud desktop scenario of the VDI technology architecture, the display speed of the writing track is improved, and the user experience is improved.

[0137] In one embodiment of the present application, when a user of a terminal device writes in a cloud desktop, the first writing track drawn by the terminal device cannot be directly rendered in the cloud desktop's operating interface. However, the cloud desktop's operating interface also needs to present the user's writing track. Therefore, in addition to the terminal device drawing the first writing track, the cloud desktop server also needs to draw the writing track corresponding to the writing operation. Currently, the writing track drawn by the cloud desktop server is recorded as the second writing track. It can be understood that the second writing track and the first writing track are writing tracks obtained based on the same writing operation, the difference being that the two writing tracks are drawn by different devices. The cloud desktop server draws the second writing track and feeds back a screen containing the second writing track to the terminal device so that the terminal device displays the writing track in the cloud desktop's operating interface. In this case, in the writing display method, after reporting the writing event generated in real time based on the writing operation to the cloud desktop server, the method further includes: receiving drawing data sent by the cloud desktop server, the drawing data including the second writing track drawn by the cloud desktop server based on the writing event; updating the current operating interface of the cloud desktop based on the drawing data, and overlaying the drawn first writing track on the second writing track in the operating interface.

[0138] For example, after receiving a writing event, the cloud desktop server not only determines the writing state (i.e., whether writing is in progress), but also draws a corresponding second writing track based on the writing event. When drawing the second writing track, the cloud desktop server calls an OpenGL interface. At this time, the cloud desktop server sends the writing coordinates corresponding to the writing event to the interface to draw the second writing track.

[0139] During the drawing process, the cloud desktop server generates the current drawing picture in real time. The drawing picture can be considered as a picture containing the currently drawn writing track. Optionally, the cloud desktop server sends a real-time drawing picture to the terminal device. The drawing picture at least contains a second writing track drawn based on the latest writing event, so that the user of the terminal device can determine the drawing process of the second writing track through the real-time drawing picture. Optionally, the data sent by the cloud desktop server when sending the drawing picture is currently recorded as drawing data. The cloud desktop server sends the drawing data to the terminal device in the form of a picture or a video stream. Among them, the format of the picture is not currently limited. For example, a picture in MJPEG format is used as drawing data. At this time, the cloud desktop server encodes the real-time drawing picture into a picture in MJPEG format and sends it to the terminal device. The video stream can be video data using the H264 standard. At this time, the cloud desktop server encodes the real-time drawing picture into a video stream in H264 standard and sends it to the terminal device. It is understandable that the bandwidth required to send pictures is greater than the bandwidth required to send video streams. Therefore, when the bandwidth is sufficient, the cloud desktop server can send pictures to the terminal device. In this case, the clarity of the drawing picture is good. When the bandwidth is insufficient, the cloud desktop server can send video streams to the terminal device. In this case, the smoothness of the drawing picture is good. For example, after receiving the drawing data, the terminal device renders according to the drawing data, and then displays the picture corresponding to the drawing data in the current running interface of the cloud desktop, that is, the running interface is updated to realize the display of the second writing track drawn by the cloud desktop on the terminal device.

[0140] It can be understood that since the first writing track drawn by the terminal device is also displayed on the display screen, that is, the writing layer is covered on the operating interface of the cloud desktop, and the first writing track and the second writing track are drawn based on the same writing operation and have the same writing position from the user's perspective, the first writing track drawn by the terminal device is covered on the second writing track drawn by the cloud desktop. Since the speed at which the terminal device draws the first writing track is faster than the speed at which the terminal device displays the second writing track drawn by the cloud desktop server, from the user's perspective, the first thing the user sees is the first writing track drawn by the terminal device. For example, Figure 2 is a schematic diagram of a writing track provided by an embodiment of the present application, and Figure 3 is another schematic diagram of a writing track provided by an embodiment of the present application. Referring to Figures 2 and 3, they show the first writing track and the second writing track displayed by the terminal device at two moments before and after when the user writes in the cloud desktop of the terminal device using a mouse. The thicker track is the second writing track drawn by the cloud desktop server, and the thinner track is the first writing track drawn by the terminal device. It can be seen from Figures 2 and 3 that the response speed of the first writing track drawn by the terminal device is faster than the response speed of the second writing track drawn by the cloud desktop server, that is, the latest position of the first writing track drawn by the terminal device is closer to the cursor. It should be noted that in Figures 2 and 3, in order to distinguish the writing tracks obtained by the two drawing methods, the first writing track and the second writing track are displayed separately in different thicknesses. In actual applications, the presentation effects of the two writing tracks are the same, and the first writing track drawn by the terminal device covers the second writing track drawn by the cloud desktop server. Therefore, from the user's perspective, only the drawing process of one writing track can be seen.

[0141] Optionally, when writing stops, the terminal device can delete the current writing layer. At this time, the terminal device displays the second writing track drawn by the cloud desktop server. From the user's perspective, the deletion process will not be seen and the writing track will continue to be displayed.

[0142] When the user writes a new track again, the above operation can be repeated, that is, the writing layer is generated again, and the first writing track is drawn through the writing layer. In one embodiment, before deleting the writing layer, the writing layer can be saved as an image by taking a screenshot of the writing layer and stored in the terminal device so that the first writing track in the writing layer can be retrieved when needed.

[0143] In the above, by receiving the drawing picture containing the second writing track drawn by the cloud desktop server through the terminal device and displaying the drawing picture, it is possible to display the second writing track drawn by the cloud desktop server at the same time as the first writing track drawn by the terminal device is displayed, and when writing stops, only the second writing track drawn by the cloud desktop server is displayed to ensure the normal response of the cloud desktop to the writing operation.

[0144] In one embodiment of the present application, when the terminal device receives the writing state and starts to obtain writing events for the cloud desktop and draws the first writing track, it can only obtain the corresponding coordinates. When the writing track actually drawn by the user has color, the terminal device cannot obtain the corresponding color through the writing event. At this time, in order to make the color of the first writing track drawn by the terminal device consistent with the color of the second writing track drawn by the cloud desktop server, the terminal device also needs to obtain the track drawing color from the cloud desktop server. On this basis, Figure 4 is a flow chart of a writing display method provided by an embodiment of the present application. Referring to Figure 4, when the terminal device executes the writing display method, it includes the following steps:

[0145] Step 210: When a writing operation of the cloud desktop is detected, a writing event generated in real time based on the writing operation is reported to the cloud desktop server.

[0146] Step 220: Receive the writing status and the track drawing color fed back by the cloud desktop server. The track drawing color is determined by the cloud desktop server, and the writing status is generated by the cloud desktop server according to the first writing event received.

[0147] Exemplarily, after the cloud desktop server receives the first writing event, the corresponding virtual operating system distributes the writing event to the application currently responding to the writing event. In addition to determining whether the user is writing in the cloud desktop of the terminal device, the color used to draw the writing track can also be determined when determining to write. Currently, the color used is recorded as the track drawing color, and the first writing track and the second writing track both use the currently determined track drawing color. In one embodiment, when the cloud desktop server draws the second writing track, the drawing is performed in the memory of the qxl virtual graphics card. At this time, the application that receives the writing event will determine the texture information when drawing the second writing track based on the current settings. The texture information can reflect the texture of the second writing track. The first 32-bit byte of the memory area where the texture information is located (i.e., the memory of the qxl virtual graphics card) is the specific value of RGBA of the track drawing color, where RGBA is a color space representing Red, Green, Blue, and Alpha.

[0148] After determining the writing status and the color of the track drawn, the cloud desktop server feeds back the writing status and the color of the track drawn to the terminal device. It is understood that in addition to feeding back the writing status and the color of the track drawn to the terminal device, the cloud desktop server can also feed back other required information to the terminal device, such as the writing coordinates corresponding to the writing event, the thickness of the track when drawing the writing track, etc.

[0149] Step 230: Acquire a real-time writing event according to the writing state, set the track color of the first writing track to be drawn as the track drawing color, and draw the first writing track corresponding to the writing event according to the track drawing color.

[0150] For example, after the terminal device monitors a writing event (such as the writing layer monitors a writing event), the color of the first writing track currently required to be drawn is set to the track drawing color fed back by the cloud desktop server. Afterwards, the first writing track is drawn according to the writing event. At this time, the track color of the first writing track is the track drawing color.

[0151] As described above, by utilizing the computing resources and rendering capabilities of the terminal device, the first writing track related to writing on the cloud desktop is drawn and rendered on the terminal device's hardware, which can speed up the display speed of the writing track when the user writes, reduce the delay caused by network transmission, and improve the timeliness of the response when the user writes in the cloud desktop. In addition, by receiving the track drawing color feedback from the cloud desktop server, the color of the first writing track drawn by the terminal device can also be guaranteed to be more accurate, that is, the color of the first writing track drawn by the terminal device is consistent with the color of the second writing track drawn by the cloud desktop server, avoiding the user from viewing two writing tracks with inconsistent colors, thereby improving the user experience.

[0152] One embodiment of the present application provides a writing response method. This writing response method can be performed by a writing response device. The writing response device can be implemented through software and / or hardware. The writing response device can be composed of two or more physical entities, or a single physical entity. In one embodiment, the writing response device is a cloud desktop server, which is a device with computing capabilities that provides cloud desktop-related services to terminal devices.

[0153] In one embodiment, the operating system running in the virtual machine of the cloud desktop server is the virtual operating system of the cloud desktop. The cloud desktop server sends the operating interface of the cloud desktop to the terminal device, and the cloud desktop client installed on the terminal device displays the operating interface during operation.

[0154] After the cloud desktop server and the terminal device cooperate, the writing response method is implemented. Figure 5 is a flow chart of a writing response method provided by an embodiment of the present application. Referring to Figure 5, when the cloud desktop server executes the writing response method, it includes the following steps:

[0155] Step 310: Receive a writing event reported by a terminal device. The writing event is a writing event generated in real time by the terminal device when a writing operation on the cloud desktop is detected.

[0156] The cloud desktop server and the terminal device perform data transmission based on the spice protocol. The reporting process of the writing event can refer to the relevant description of the above embodiment.

[0157] Step 320: When it is determined that writing for the cloud desktop occurs on the terminal device according to the first received writing event, a writing state is generated.

[0158] For example, after the virtual operating system of the cloud desktop in the cloud desktop server receives a first writing event, the virtual operating system distributes the first writing event to the application currently responding to the writing event. After receiving the first writing event, the application determines whether the user is writing in the cloud desktop. It is understood that subsequent writing events received by the cloud desktop server are also distributed to the application currently responding to the writing event, so that the application draws a second writing track.

[0159] Optionally, the application draws a second writing track in the qxl virtual graphics card of the virtual machine where the virtual operating system is located. At this time, after the application receives the first writing event, it determines whether writing is in progress through the qxl virtual graphics card based on the coordinates of the operation corresponding to the first writing event (currently recorded as writing coordinates).

[0160] In one embodiment, the following is described using an example of determining whether writing is currently being performed based on the cursor coordinates and the writing coordinates. In this case, step 320 includes steps 321 and 322:

[0161] Step 321: Obtain corresponding writing coordinates and cursor coordinates according to the first writing event received.

[0162] Exemplarily, the writing coordinates are the coordinates corresponding to the writing operation when the user writes (ie, the operation coordinates), and the cursor coordinates include the cursor offset and the physical position of the cursor.

[0163] For example, after receiving the first writing event, the application responding to the writing event can obtain the current writing coordinates. The application then sends the writing coordinates to the QXL virtual graphics card, which then performs drawing. When the QXL virtual graphics card performs drawing, it can determine whether the user is writing in the cloud desktop.

[0164] In one embodiment, the virtual operating system corresponding to the cloud desktop is Ubuntu 21.10 as an example for description. At this time, the graphics environment of the cloud desktop is Wayland, where Wayland is a simple "display server". After the application that responds to the writing event receives the writing event, the corresponding second writing track is displayed on the Wayland surface. At this time, the application calls the graphics API interface so that the Wayland surface renders the second writing track to the GPU memory of the virtual machine through the graphics API interface, that is, the drawing information corresponding to the second writing track (such as writing coordinates, track color information, etc.) is sent to the qxl virtual graphics card through the Wayland surface to realize drawing on the qxl virtual graphics card. Among them, the GPU of the virtual machine is the qxl virtual graphics card, and the GPU memory is the memory of the qxl virtual graphics card. In one embodiment, the graphics API interface called by the application can be glTexSubImage2D of OpenGL. At this time, the parameters of the glTexSubImage2D api include the writing coordinates obtained by the application. Afterwards, the qxl virtual graphics card can determine whether writing is currently in progress based on the writing coordinates in the glTexSubImage2D api. In one embodiment, when the qxl virtual graphics card determines whether writing is currently in progress, the qxl virtual graphics card driver also obtains cursor coordinates. The cursor offset is obtained from the graphical environment interface (i.e., Wayland), and the cursor physical position is obtained from the underlying driver of the virtual operating system. It is understood that currently, the writing coordinates and cursor coordinates required to determine whether writing is in progress are the writing coordinates and cursor coordinates corresponding to the first writing event. In actual applications, the writing coordinates and cursor coordinates corresponding to multiple writing events can also be used for comprehensive judgment.

[0165] Step 322: When determining that the terminal device has written on the cloud desktop based on the writing coordinates and the cursor coordinates, a writing state is generated.

[0166] In one embodiment, whether the terminal device is writing is determined based on the writing coordinates and the cursor coordinates. Exemplarily, the cursor coordinates include the cursor offset and the cursor physical position. In this case, step 322 may include steps 3221-3222:

[0167] Step 3221: Calculate the difference between the writing coordinates and the physical position of the cursor.

[0168] Exemplarily, after the cloud desktop server receives the first writing event, the coordinates currently obtained by the qxl virtual graphics card include the writing coordinates, cursor physical position, and cursor offset corresponding to the first writing event. Afterwards, the qxl virtual graphics card calculates the difference between the writing coordinates and the physical position of the cursor. It can be understood that both the writing coordinates and the physical position of the cursor can be two-dimensional coordinates (including the coordinate values ​​of the x-axis and the y-axis), and the difference between the writing coordinates and the physical position of the cursor is also a two-dimensional coordinate. That is, the difference between the writing coordinates and the physical position of the cursor includes the difference between the writing coordinates and the physical position of the cursor on the x-axis and the difference on the y-axis.

[0169] Step 3222: When the difference is equal to the cursor offset, it is determined that the terminal device has written on the cloud desktop and a writing state is generated.

[0170] Exemplarily, the qxl virtual graphics card determines whether the difference between the writing coordinates and the physical position of the cursor is consistent with the cursor offset (i.e., whether they are equal). The cursor offset is also a two-dimensional coordinate. If consistent, it means that the user is writing in the cloud desktop, that is, the terminal device has written for the cloud desktop, and thus, a writing state is generated. If inconsistent, it means that the terminal device has not written.

[0171] Step 330: Feedback the writing state to the terminal device, so that the terminal device obtains a real-time writing event according to the writing state, and draws a corresponding first writing track according to the writing event.

[0172] For example, after the qxl virtual graphics card generates a writing state, the writing state can be fed back to the terminal device through the spice protocol, and the terminal device draws a first writing track according to the writing state. For example, the terminal device generates a writing layer covering the cloud desk display interface and draws the first writing track through the writing layer.

[0173] In one embodiment, Figure 6 is a first writing response data flow diagram provided by an embodiment of the present application, which is a data flow diagram when the cloud desktop server generates a writing state based on a writing event and sends the writing state to the terminal device. The virtual operating system of the cloud desktop corresponding to Figure 6 is Ubuntu 21.10, and the operating system of the terminal device is the Android system. Referring to Figure 6, draw refers to an application that responds to a writing event. After the Guest OS distributes the writing event received in real time to draw, draw calls the graphics API interface to enable the Wayland surface to render the second writing track to the qxl virtual graphics card (denoted as qxl in Figure 6) through the graphics API interface. When writing is occurring on the cloud desktop in the terminal device, qxl generates a writing state based on the parameters of the graphics API interface (i.e., writing coordinates) and the cursor coordinates, and passes the writing state to the spice server (i.e., spice service) of the cloud desktop server Host OS. The spice server writes the writing state into the spice protocol, i.e., adds it to the transmission channel. After that, the writing state is sent to the spice client (i.e., the terminal device) through the socket. After that, the terminal device adds an Overlay writing layer according to the writing state so that the terminal device draws the first writing track.

[0174] The above-mentioned technical means, by receiving the writing event generated in real time by the terminal device when detecting the writing operation for the cloud desktop, determining that writing is occurring on the cloud desktop, generating a writing status and feeding it back to the terminal device, so that the terminal device obtains the real-time writing event (the writing event after receiving the writing status) and draws the first writing track, solves the technical problem in the related art that there is a response delay in the display of the writing track by the terminal device when writing in the cloud desktop. The cloud desktop server only needs to feed back the writing status to the terminal device, and it can use the terminal device's own computing resources and rendering capabilities to draw and render the writing track related to the cloud desktop on the terminal device's hardware, which can speed up the display speed of the writing track, reduce the delay caused by network transmission, and improve the timeliness of the response when the user writes in the cloud desktop.

[0175] In one embodiment of the present application, while a user is drawing a track, the terminal device continuously reports a writing event. When the user stops drawing the track, the terminal device no longer reports the writing event, and the cloud desktop server no longer receives the writing event and can notify the terminal device to stop writing. In this case, after the cloud desktop server receives the writing event reported by the terminal device, it further includes: generating a stop writing status when no more writing events are received; and feeding back the stop writing status to the terminal device, so that the terminal device stops drawing the corresponding first writing track.

[0176] For example, when the cloud desktop server does not receive a writing event, it no longer sends a writing event to the application that responds to the writing event. When the application no longer receives a writing event, it determines that the writing is finished. At this time, the application that responds to the writing event no longer calls the graphics API interface, that is, the qxl virtual graphics card no longer receives drawing information. At this time, the qxl virtual graphics card stops drawing and generates a stop writing state. Afterwards, the stop writing state is sent to the terminal device to cause the terminal device to stop drawing the first writing track. The sending process of the stop writing state and the writing state is the same, which will not be described in detail at this time.

[0177] As mentioned above, the cloud desktop server can determine whether the user is writing in the cloud desktop based on whether a writing event is received, and when not writing, generate a stop writing state and feedback it to the terminal device so that the terminal device stops drawing the first writing trajectory, thereby ensuring the timeliness of the stop of the writing response.

[0178] In one embodiment of the present application, when a terminal device draws a first writing track, the cloud desktop server also draws a corresponding second writing track based on the writing event, and feeds back the drawing data containing the second writing track to the terminal device for display. In this case, after receiving the writing event reported by the terminal device, the process further includes: drawing the second writing track based on the writing event; and sending the drawing data containing the second writing track to the terminal device, so that the terminal device updates the current operating interface of the cloud desktop based on the drawing data, with the first writing track drawn on the terminal device overlaying the second writing track in the operating interface.

[0179] FIG7 is a second writing response data flow diagram provided by an embodiment of the present application. When the cloud desktop server draws the second writing track, it is driven by the Guest OS through the QXL virtual graphics card, that is, the second writing track is drawn in the QXL virtual graphics card. Referring to FIG7, when the Guest OS determines that the running interface of the cloud desktop has changed, the virtual machine where the Guest OS is located (denoted as virtual machine in FIG7) obtains the data of the framebuffer screen change area and passes the data of the framebuffer screen change area from the inside of the virtual machine through the QXL virtual graphics card (that is, drawn in the QXL virtual graphics card, denoted as QXL interface in FIG7) to the spice server. Among them, the framebuffer is a video output device that drives a video display device from a memory buffer containing complete frame data. The framebuffer screen change area can be understood as a "dirty" area (denoted as dirty framebuffer in FIG7). For example, when drawing the second writing track, the "dirty" area data is a rectangular area containing the second writing track currently being drawn. The Spice server then retrieves the content (i.e., drawing data) drawn from the QXL virtual graphics card and encodes it (represented as prepare drawable and encode in Figure 7). The Spice server encapsulates the encoded data into the Spice protocol, adding it to the transmission channel. The Spice server then sends the drawing data to the Spice client (i.e., the terminal device) via a socket, which decodes and renders it for display (represented as render in Figure 7). Optionally, the drawing data can be an image or video stream.

[0180] For example, FIG8 is a schematic diagram of a first drawing screen provided by an embodiment of the present application, and FIG9 is a schematic diagram of a second drawing screen provided by an embodiment of the present application. FIG8 and FIG9 are drawing screens drawn by the cloud desktop server at different times during the writing process of a trajectory. Currently, the moment corresponding to the drawing screen shown in FIG8 is the moment before the moment corresponding to the drawing screen shown in FIG9. The width and height of the drawing screen shown in FIG8 are 589×349. The current writing position in FIG8 is located at the end of the lower left corner trajectory (marked by rectangular box 31 in FIG8), and the writing coordinates are (63, 273). The width and height of the drawing screen shown in FIG9 are 589×362. The current writing position in FIG9 is located at the end of the lower left corner trajectory (marked by rectangular box 32 in FIG9), and the writing coordinates are (70, 286). The drawing screen shown in FIG8 to the drawing screen shown in FIG9 describes the writing moving from (63, 273) to (70, 286). At this time, there are only two differences between the drawing data corresponding to the two drawing pictures sent by the spice protocol. One difference is the current writing coordinates displayed in the upper right corner (marked by rectangular boxes 33 and 34 in Figures 8 and 9 respectively), and the other difference is the second writing track newly added in the lower left corner. The spice server needs to integrate the areas where the two differences are located into one area, that is, a large area composed of two small areas with differences is used as a dirty framebuffer. The drawing picture shown in Figure 9 can be understood as a large area composed of two small areas with differences (from the user's perspective, the two small areas are located in the upper right and lower left of the large area, respectively). That is, the spice protocol needs to send the complete drawing picture shown in Figure 9 to the terminal device in the form of drawing data, so that when the terminal device updates the running interface of the cloud desktop, the drawing picture shown in Figure 9 is displayed. It can be understood that other areas of the display interface of the cloud desktop (areas outside the painting interface shown in Figure 9) may not change.

[0181] For example, FIG10 is a schematic diagram of the command structure of a QXL virtual graphics card provided by an embodiment of the present application. Referring to FIG10, during the application of the QXL virtual graphics card, three circular buffers (i.e., buffers) are used to store drawing commands (represented as command in FIG10), mouse commands (represented as cursor in FIG10), and resource commands that need to be released (represented as release in FIG10). The buffers in the three links all adopt a circular data structure (represented by three circles in FIG10). Among them, the drawing command is transmitted for the graphic screen (the drawing screen shown in FIG8 and FIG9), and the mouse command is transmitted for the input input event between the terminal device and the virtual machine in the cloud desktop server (excluding touch-related events). For events generated by touch, the touch event can be sent from the terminal device to the virtual machine of the cloud desktop server by redirecting the USB device. Among them, the part of the terminal device used to detect touch (i.e., the module that receives the touch operation) can be considered as a USB device (such as the infrared touch box in the infrared screen can be considered as a USB device). In Figure 10, the overlap of the two circles corresponding to command and cursor indicates that both types of commands are sent from the corresponding virtual machine to the QXL virtual graphics card. Specifically, the virtual machine stores the corresponding command in a buffer (shown as a push in Figure 10), and then the spice server retrieves the corresponding command from the buffer (shown as a pop in Figure 10). After the command and cursor are executed, the spice server generates a release and stores the command in the buffer (shown as a push in Figure 10). The virtual machine then retrieves the corresponding command from the buffer (shown as a pop in Figure 10) and destroys it, releasing the corresponding resources.

[0182] It can be understood that for the terminal device, it not only needs to draw the first writing track through the writing layer, but also needs to display the drawing data containing the second writing track fed back by the cloud desktop server. At this time, Figure 11 is a writing and drawing framework diagram provided by an embodiment of the present application, which is a response framework diagram of the terminal device. Referring to Figure 11, the spiceAPP of the terminal device (also recorded as spice client, cloud desktop client) obtains the drawing data and writing status of the cloud desktop server through the transmission channel (i.e. channel) of the spice protocol, and then converts the drawing data into a drawable object (i.e. drawable), and then renders and displays it (i.e. render) to update the running interface of the cloud desktop. At the same time, the system service SurfaceFlinger that manages the display of the terminal device manages the display and refresh of the Overlay writing layer to realize the drawing of the first writing track through the Overlay writing layer. SurfaceFlinger uses the GL interface provided by MESA when managing the Overlay writing layer.

[0183] As mentioned above, when the terminal device draws the first writing track, the cloud desktop server also draws the second writing track through the writing event, and sends the drawing data containing the second writing track to the terminal device for display. It can be achieved that when the terminal device displays the first writing track drawn by itself, the second writing track drawn by the cloud desktop server is displayed at the same time. In this way, when writing stops, the second writing track drawn by the cloud desktop server can still be displayed to ensure the normal response of the cloud desktop to the writing operation.

[0184] In one embodiment of the present application, in order to ensure that the color of the first writing track drawn by the terminal device is consistent with the color of the second writing track drawn by the cloud desktop server, the cloud desktop server also needs to determine the track drawing color and feed it back to the terminal device together with the writing status. At this time, based on the first writing event received, when it is determined that the terminal device has written with respect to the cloud desktop, generating the writing status also includes: determining the track drawing color during writing based on the writing event. Accordingly, when feeding back the writing status to the terminal device, it also includes: feeding back the track drawing color to the terminal device, so that the terminal device draws the first writing track based on the track drawing color.

[0185] Exemplarily, the cloud desktop server determines the track drawing color required for currently drawing the second writing track according to the writing event, and feeds back the track drawing color and the writing status to the terminal device.

[0186] In one embodiment, determining the color of the track drawn during writing based on a writing event includes: obtaining corresponding texture information based on the writing event; and obtaining the track drawing color based on the texture information. Exemplarily, an application in the cloud desktop server that responds to the writing event can also obtain texture information based on the writing event. The texture information reflects the texture used when drawing the second writing track. It is understandable that when the application that responds to the writing event calls the graphics API interface, the parameters used by the graphics API interface also include texture information. Afterwards, the qxl virtual graphics card determines the first 32-bit byte of the texture information in the memory, that is, the first 32-bit byte in the memory when the texture information is used to draw the second writing track, and uses the value corresponding to the first 32-bit byte as the RGBA value of the second writing track, that is, as the track drawing color.

[0187] As mentioned above, by setting the track drawing color through the cloud desktop server and feeding it back to the terminal device, it can also ensure that the color of the first writing track drawn by the terminal device is more accurate, that is, the color of the first writing track drawn by the terminal device is consistent with the color of the second writing track drawn by the cloud desktop server, avoiding the user from viewing two writing tracks with inconsistent colors, thereby improving the user experience.

[0188] It should be noted that the relevant technical details when the cloud desktop server executes the writing response method can refer to the relevant technical details when the aforementioned terminal device executes the writing display method, and it has corresponding functions and beneficial effects.

[0189] The following is an exemplary description of the writing display method and writing response method provided in the embodiments of the present application. The writing display method and writing response method can be applied to product fields involving VDI cloud desktops, such as the education market, the medical market, and the business market. Taking the education market as an example, at this time, the school's computer room server can be used as a cloud desktop server and run the virtual machine where the virtual operating system corresponding to the cloud desktop is located. The interactive tablet equipped in each classroom can be used as a terminal device. Afterwards, when a teacher or student uses an interactive tablet to write in the cloud desktop, the interactive tablet can draw a first writing track, and the cloud desktop server also draws a second writing track and displays it on the interactive tablet. The display effects of the two types of writing tracks are consistent, but the display speed of the first writing track drawn by the interactive tablet is faster than the display speed of the second writing track drawn by the cloud desktop server. At this time, the user can quickly view the drawn writing track (that is, the first writing track drawn by the interactive tablet), which is consistent with the real writing experience. That is, from the user's perspective, the display delay of the second writing track when writing in the cloud desktop is reduced, thereby improving the user's usage experience.

[0190] FIG12 is a schematic diagram of the structure of a terminal device provided in one embodiment of the present application. The terminal device includes a processor 40 and a display screen 41. The number of processors 40 in the terminal device can be one or more, and FIG12 takes one processor 40 as an example.

[0191] Exemplarily, the processor 40 includes a central processing unit (CPU), which primarily performs the computing and control functions of the terminal device. Optionally, the processor 40 also includes a graphics processing unit (GPU), which primarily performs graphics and image-related computing tasks. In one embodiment, the terminal device also includes a memory that can be used to store software programs, computer executable programs, and modules. By running the software programs, instructions, and modules stored in the memory, the processor 40 can execute various functional applications and data processing functions of the terminal device. Optionally, the memory includes a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created based on the use of the terminal device. In one embodiment, the terminal device also includes a network interface that communicates with an external terminal (such as a cloud desktop server) via a network connection. In one embodiment, the terminal device also includes at least one display screen 41. Display screen 41 can be an LCD, LED, or OLED display, etc., and is not currently limited. Optionally, display screen 41 has a touch function, and the touch method can be capacitive, resistive, infrared, and / or electromagnetic, and receives touch operations through a corresponding touch detection device. Currently, the imaging content of the display screen 41 is determined by the processor 40. Furthermore, the terminal device may also include audio output and input devices such as speakers and microphones, as well as input devices related to user equipment and control of the terminal device (such as physical buttons), a network interface for communicating with a server, and the like. It is understood that the processor, memory, display screen, and network interface of the terminal device may be connected via a bus or other means.

[0192] In one embodiment, the processor 40 runs a cloud desktop client installed on a terminal device; the display screen 41 displays the cloud desktop's operating interface while the cloud desktop client is running, where the operating interface is determined by the cloud desktop server. Based on this, the processor 40 reports writing events generated in real time based on detection of writing operations on the cloud desktop to the cloud desktop server, obtains the real-time writing events based on the writing status reported by the cloud desktop server, and draws a corresponding first writing track based on the writing events, wherein the writing status is generated by the cloud desktop server based on the first received writing event; the display screen 41 displays the drawn first writing track in the operating interface.

[0193] When the processor 40 only includes a CPU, the steps performed by the above-mentioned processor 40 are executed by the CPU. When the processor 40 includes a CPU and a GPU, the GPU is responsible for rendering the first writing trajectory. Afterwards, the display screen 41 displays the first writing trajectory rendered by the GPU. For example, when the processor 40 runs the cloud desktop client, the display screen displays the operation interface of the cloud desktop. The processor 40 can detect the writing event through the writing device (a device that detects touch or a mouse). Afterwards, the cloud desktop client sends the writing event to the cloud desktop server through the network interface. Afterwards, the processor 40 receives the writing status fed back by the cloud desktop server through the network interface, and starts to obtain writing events according to the writing status, and draws the corresponding first writing trajectory according to the writing event. Accordingly, the display screen 41 displays the first writing trajectory drawn by the processor 40 on the basis of displaying the operation interface of the cloud desktop.

[0194] In one embodiment, the processor 40 obtains a real-time writing event based on the writing status fed back by the cloud desktop server, and draws a corresponding first writing track based on the writing event, including: the processor 40 generates a writing layer based on the writing status fed back by the cloud desktop server, and instructs the display screen to display the writing layer, and the writing layer is covered on the current running interface of the cloud desktop; the processor 40 controls the writing layer to monitor the real-time writing event of the cloud desktop, and draws a corresponding first writing track in the writing layer based on the writing event

[0195] In one embodiment, when the processor 40 receives the writing status feedback from the cloud desktop server, it also receives the trajectory drawing color feedback from the cloud desktop server, and the trajectory drawing color is determined by the cloud desktop server; when the processor 40 draws the corresponding first writing trajectory according to the writing event, it also includes: the processor 40 sets the trajectory color of the first writing trajectory to be drawn as the trajectory drawing color, and draws the first writing trajectory corresponding to the writing event according to the trajectory drawing color.

[0196] In one embodiment, when the processor 40 receives the writing status feedback from the cloud desktop server, it also receives the writing coordinates feedback from the cloud desktop server, where the writing coordinates are the writing coordinates obtained by the cloud desktop server based on all the writing events currently received; when the processor 40 draws the corresponding first writing trajectory according to the writing event, it includes: the processor 40 draws the corresponding first writing trajectory based on the writing event and the writing coordinates.

[0197] In one embodiment, after the processor 40 reports the writing event generated in real time based on the detection of the writing operation of the cloud desktop to the cloud desktop server, it also includes: the processor 40 receives the drawing data sent by the cloud desktop server, and the drawing data includes a second writing track drawn by the cloud desktop server according to the writing event; the processor 40 updates the current running interface of the cloud desktop according to the drawing data; the display screen 41 displays the updated running interface, and the displayed first writing track covers the second writing track in the running interface.

[0198] In one embodiment, after the processor 40 obtains real-time writing events based on the writing status feedback from the cloud desktop server and draws the corresponding first writing trajectory based on the writing event, it also includes: when the processor 40 monitors that the writing event stops being generated, it stops drawing the corresponding first writing trajectory; or, when the processor 40 receives the stop writing status feedback from the cloud desktop server, it stops drawing the corresponding first writing trajectory.

[0199] In one embodiment, the terminal device and the cloud desktop server perform data transmission based on the spice protocol. The cloud desktop adopts the VDI technology architecture.

[0200] The terminal device provided in one embodiment of the present application can be used to execute the writing display method provided in any of the above embodiments, and has corresponding functions and beneficial effects. The relevant technical details when the terminal device executes the writing display method can refer to the relevant technical details in the above embodiments.

[0201] Figure 13 is a schematic diagram of the structure of a writing response device provided by one embodiment of the present application. The writing response device is applied to a cloud desktop server. The operating system running in the virtual machine of the cloud desktop server is the virtual operating system of the cloud desktop. The cloud desktop server sends the cloud desktop's operating interface to the terminal device, and the cloud desktop client installed on the terminal device displays the operating interface during operation. Referring to Figure 13, the writing response device includes: a first receiving unit 501, a first generating unit 502, and a first feedback unit 503.

[0202] Among them, the first receiving unit 501 is used to receive writing events reported by the terminal device, and the writing event is a writing event generated in real time by the terminal device when the writing operation of the cloud desktop is detected; the first generating unit 502 is used to determine, based on the first writing event received, when the terminal device occurs writing for the cloud desktop, and generate a writing state; the first feedback unit 503 is used to feed back the writing state to the terminal device, so that the terminal device obtains real-time writing events according to the writing state, and draws the corresponding first writing trajectory according to the writing event.

[0203] In one embodiment of the present application, the first generation unit 502 includes: a coordinate acquisition subunit, used to obtain corresponding writing coordinates and cursor coordinates based on the first writing event received; a state generation subunit, used to generate a writing state when determining that the terminal device has written for the cloud desktop based on the writing coordinates and cursor coordinates.

[0204] In one embodiment of the present application, the cursor coordinates include the cursor offset and the cursor physical position, and the state generation subunit includes: a difference calculation subunit, which is used to calculate the difference between the writing coordinates and the cursor physical position; a writing determination subunit, which is used to determine that the terminal device has written with respect to the cloud desktop when the difference is equal to the cursor offset, and generate a writing state.

[0205] In one embodiment of the present application, the first generation unit 502 is used to determine, based on the first received writing event, when the terminal device is writing for the cloud desktop, and to generate a writing state, and is also used to determine the color of the trajectory drawn during the writing based on the writing event. Correspondingly, the first feedback unit 503 is used to feed back the writing state to the terminal device so that the terminal device can obtain a real-time writing event based on the writing state and draw a corresponding first writing trajectory based on the writing event, and is also used to feed back the trajectory drawing color to the terminal device so that the terminal device can draw the first writing trajectory based on the trajectory drawing color.

[0206] In one embodiment of the present application, the first generation unit 502 is used to determine the trajectory drawing color during writing according to the writing event, including: a texture acquisition subunit, used to obtain corresponding texture information according to the writing event; and a color determination subunit, used to obtain the trajectory drawing color according to the texture information.

[0207] In one embodiment of the present application, it also includes: a drawing unit, which is used to draw a second writing track according to the writing event after receiving the writing event reported by the terminal device; a sending unit, which is used to send drawing data containing the second writing track to the terminal device, so that the terminal device updates the current running interface of the cloud desktop according to the drawing data, and the first writing track drawn in the terminal device is overlaid on the second writing track in the running interface.

[0208] In one embodiment of the present application, it also includes: a second generation unit, which is used to generate a stop writing state when no writing event is received after receiving the writing event reported by the terminal device; and a second feedback unit, which is used to feed back the stop writing state to the terminal device so that the terminal device stops drawing the corresponding first writing trajectory.

[0209] The writing response device provided in one embodiment of the present application is included in a cloud desktop server and can be used to execute the writing response method provided by the cloud desktop server in any of the above embodiments, and has corresponding functions and beneficial effects.

[0210] It is worth noting that in the embodiment of the above-mentioned writing response device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0211] Figure 14 is a schematic diagram of the structure of a cloud desktop server provided by one embodiment of the present application. As shown in Figure 14, the cloud desktop server includes a processor 60, a memory 61, and a communication device 62. The number of processors 60 in the cloud desktop server can be one or more, and Figure 14 uses one processor 60 as an example. The processor 60, memory 61, and communication device 62 in the cloud desktop server can be connected via a bus or other means, and Figure 14 uses a bus connection as an example.

[0212] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the writing response method executed by the cloud desktop server in the embodiments of the present application (for example, the first receiving unit 501, the first generating unit 502, and the first feedback unit 503 in the writing response device). The processor 60 executes the software programs, instructions, and modules stored in the memory 61 to execute various functional applications and data processing of the cloud desktop server, thereby implementing the writing response method executed by the cloud desktop server.

[0213] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the cloud desktop server, etc. In addition, the memory 61 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely located relative to the processor 60, and these remote memories may be connected to the cloud desktop server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0214] The cloud desktop server may also include an input device that can be used to receive input digital or character information and generate key signal input related to user settings and function control of the cloud desktop server. The cloud desktop server may also include an output device that may include devices such as an audio player. The cloud desktop server may also include a display screen, which may be a touch-enabled display screen that displays according to the instructions of the processor 60. The communication device 62 can also be understood as a network interface for implementing communication functions, such as data transmission with a terminal device.

[0215] It can be understood that the operating system running in the virtual machine of the cloud desktop server is the virtual operating system corresponding to the cloud desktop, and the cloud desktop server determines the running interface of the cloud desktop, and when the cloud desktop client of the terminal device runs, the running interface is displayed by the terminal device.

[0216] The above-mentioned cloud desktop server includes a writing response device, which can be used to execute the writing response method executed by the cloud desktop server and has corresponding functions and beneficial effects.

[0217] One embodiment of the present application also provides a writing response system, which includes at least one terminal device and a cloud desktop server. The terminal device and the cloud desktop server can communicate data via wired or wireless means. Specifically; when the terminal device detects a writing operation on the cloud desktop, it reports a writing event generated in real time based on the writing operation to the cloud desktop server; the cloud desktop server receives the writing event reported by the terminal device; the cloud desktop server determines that the terminal device has written for the cloud desktop based on the first writing event received, and generates a writing status; the cloud desktop server feeds back the writing status to the terminal device; the terminal device receives the writing status fed back by the cloud desktop server; the terminal device obtains real-time writing events based on the writing status, and draws a corresponding first writing trajectory based on the writing event.

[0218] Specifically, FIG15 is a network diagram of a cloud desktop writing process provided by one embodiment of the present application. Referring to FIG15 , a terminal device 71 communicates data with a cloud desktop server 72 via a network. The virtual operating system of the cloud desktop running in the virtual machine of the cloud desktop server 72 generates a cloud desktop operation interface and sends it to the terminal device 71 for display. When the operation interface displayed by the terminal device 71 is related to writing, the user 73 can write on the cloud desktop by writing in the operation displayed by the terminal device 71.

[0219] FIG16 is a schematic diagram of a cloud desktop writing interaction provided by an embodiment of the present application. Referring to FIG16 , user 73 writes on terminal device 71. Terminal device 71 detects the writing event through a writing device (a touch detection device or a mouse). Subsequently, the cloud desktop client of terminal device 71 sends the writing event to the virtual operating system of the cloud desktop in cloud desktop server 72 via the spice protocol. The virtual operating system sends the writing event to the application that responds to the writing. The application calls the graphics API interface, causing the Wayland surface to send the drawing information corresponding to the second writing track (writing coordinates, track drawing color) to the QXL virtual graphics card via the graphics API interface. The QXL virtual graphics card determines that writing has occurred on terminal device 71 based on the writing coordinates, physical cursor position, and cursor offset of the first writing event, generates a writing status, and transmits the writing status to the spice service of the cloud desktop server Host OS. The spice service writes the writing status into the spice protocol and sends it to the cloud desktop client of the terminal device. When the cloud desktop client of the terminal device receives the writing status sent by the cloud desktop server, the display management system service of the terminal device manages the display and refresh of the Overlay writing layer, so that the Overlay writing layer obtains the writing event for the cloud desktop and draws the first writing track.

[0220] On this basis, when the application responding to writing calls the graphics API interface in the virtual operating system of the cloud desktop server, the qxl virtual graphics card also determines the track drawing color based on the texture information contained in the graphics API interface, and feeds back the track drawing color together with the writing status to the cloud desktop client of the terminal device. When the Overlay writing layer of the terminal device draws the first writing track, the track color of the first writing track is set as the track drawing color.

[0221] On this basis, FIG17 is a schematic diagram of a cloud desktop writing interaction provided by an embodiment of the present application. Referring to FIG17 , user 73 writes in terminal device 71. Terminal device 71 detects the writing event through a writing device (a touch detection device or a mouse). Afterwards, the cloud desktop client of terminal device 71 sends the writing event to the virtual operating system of the cloud desktop in cloud desktop server 72 via the spice protocol. The virtual operating system sends the writing event to the application that responds to the writing. The application calls the graphics API interface, causing the Wayland surface to send the drawing information corresponding to the second writing track (writing coordinates, track drawing color) to the qxl virtual graphics card via the graphics API interface. In addition to determining the writing status, the qxl virtual graphics card also draws the corresponding second writing track. Afterwards, the spice service of the virtual operating system obtains the dirty framebuffer based on the second writing track and sends it to the cloud desktop client of the terminal device as drawing data via the spice protocol.

[0222] When the cloud desktop client of the terminal device receives the drawing data sent by the cloud desktop server, it also updates the currently displayed running interface based on the drawing data to display the second writing track in the terminal device.

[0223] Afterwards, when the user finishes writing, the terminal device deletes the first writing track displayed on the Overlay layer, so that the user can view the second writing track displayed on the cloud desktop running interface.

[0224] It is understandable that the above-mentioned writing response system is also used to execute the writing display direction and writing response method provided by any of the aforementioned embodiments, and the specific corresponding functions and beneficial effects.

[0225] One embodiment of the present application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform relevant operations in the writing display method provided in any embodiment of the present application, and have corresponding functions and beneficial effects.

[0226] One embodiment of the present application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform relevant operations in the writing response method provided in any embodiment of the present application, and have corresponding functions and beneficial effects.

[0227] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0228] Therefore, the application can adopt the form of complete hardware embodiment, complete software embodiment, or the embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The application is described with reference to the flow chart and / or block diagram of the method, equipment (system) and computer program product according to the embodiment of the application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow chart and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing equipment to produce a machine, so that the instruction executed by the processor of the computer or other programmable data processing equipment produces a device for realizing the function specified in one flow chart flow chart or multiple flow processes and / or one block or multiple blocks of the block diagram. These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device that implements the functions specified in one or more processes of the flowchart and / or one or more blocks of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes of the flowchart and / or one or more blocks of the block diagram.

[0229] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in a computer-readable medium, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0230] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0231] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0232] Note that the above are only preferred embodiments of the present application and the technical principles employed. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present application. The scope of the present application is determined by the scope of the appended claims.

Claims

1. A writing display method, applied to a terminal device, wherein: The terminal device is installed with a cloud desktop client, which displays the operation interface of the cloud desktop when the cloud desktop client is running, and the operation interface is determined by the cloud desktop server; The writing display method comprises: When a writing operation of the cloud desktop is detected, a writing event generated in real time based on the writing operation is reported to the cloud desktop server; receiving a writing status fed back by the cloud desktop server, where the writing status is generated by the cloud desktop server according to a first received writing event; According to the writing state, a real-time writing event is acquired, and a corresponding first writing track is drawn according to the writing event.

2. The method according to claim 1, wherein The acquiring of a real-time writing event according to the writing state, and drawing a corresponding first writing trajectory according to the writing event, includes: Generate a writing layer according to the writing state, and overlay the writing layer on the current running interface of the cloud desktop; The writing layer monitors the real-time writing events of the cloud desktop, and draws a corresponding first writing track in the writing layer according to the writing events.

3. The method according to claim 1, wherein When receiving the writing status fed back by the cloud desktop server, the method further includes: receiving a trajectory drawing color fed back by the cloud desktop server, where the trajectory drawing color is determined by the cloud desktop server; Drawing a corresponding first writing trajectory according to the writing event includes: The track color of the first writing track to be drawn is set as the track drawing color, and the first writing track corresponding to the writing event is drawn according to the track drawing color.

4. The method according to claim 1, wherein When receiving the writing status fed back by the cloud desktop server, the method further includes: receiving writing coordinates fed back by the cloud desktop server, where the writing coordinates are writing coordinates obtained by the cloud desktop server based on all currently received writing events; The acquiring of a real-time writing event according to the writing state, and drawing a corresponding first writing trajectory according to the writing event, includes: According to the writing state, a real-time writing event is acquired, and a corresponding first writing trajectory is drawn based on the writing event and the writing coordinates.

5. The method according to claim 1, wherein After acquiring a real-time writing event according to the writing state and drawing a corresponding first writing track according to the writing event, the method further includes: When it is detected that the writing event stops being generated, the drawing of the corresponding first writing track is stopped; or, When receiving the stop writing status fed back by the cloud desktop server, stop drawing the corresponding first writing track.

6. The method according to claim 1, wherein After reporting the writing event generated in real time based on the writing operation to the cloud desktop server, the method further includes: receiving drawing data sent by the cloud desktop server, where the drawing data includes a second writing trajectory drawn by the cloud desktop server according to the writing event; According to the drawing data, the current running interface of the cloud desktop is updated, and the drawn first writing track is covered on the second writing track in the running interface.

7. The method according to claim 1, wherein The terminal device and the cloud desktop server perform data transmission based on the SPICE protocol.

8. The method according to claim 1, wherein The cloud desktop adopts VDI technology architecture.

9. A writing response method, applied to a cloud desktop server, wherein: The operating system running in the virtual machine of the cloud desktop server is the virtual operating system of the cloud desktop. The cloud desktop server sends the running interface of the cloud desktop to the terminal device, and the cloud desktop client installed on the terminal device displays the running interface during operation. The writing response method includes: receiving a writing event reported by the terminal device, wherein the writing event is a writing event generated in real time by the terminal device when detecting a writing operation of the cloud desktop; When determining, based on the first received writing event, that writing on the cloud desktop occurs on the terminal device, generating a writing state; The writing state is fed back to the terminal device, so that the terminal device obtains a real-time writing event according to the writing state, and draws a corresponding first writing track according to the writing event.

10. The method according to claim 9, wherein: The step of determining, based on the first received writing event, that writing on the cloud desktop occurs on the terminal device and generating a writing state includes: Acquire corresponding writing coordinates and cursor coordinates according to the first received writing event; When it is determined according to the writing coordinates and the cursor coordinates that the terminal device has written on the cloud desktop, a writing state is generated.

11. The method according to claim 10, wherein: The cursor coordinates include the cursor offset and the cursor physical position, The generating of a writing state when determining, based on the writing coordinates and the cursor coordinates, that the terminal device has written on the cloud desktop comprises: Calculating the difference between the writing coordinates and the physical position of the cursor; When the difference is equal to the cursor offset, it is determined that the terminal device has written on the cloud desktop, and a writing state is generated.

12. The method according to claim 9, wherein When determining, based on the first received writing event, that writing on the cloud desktop occurs on the terminal device and generating a writing state, the method further includes: Determining the color of the track being drawn during writing according to the writing event; When the writing status is fed back to the terminal device, the method further includes: The trajectory drawing color is fed back to the terminal device, so that the terminal device draws the first writing trajectory according to the trajectory drawing color.

13. The method according to claim 12, wherein: The determining of the color of the writing track according to the writing event includes: According to the writing event, corresponding texture information is obtained; The trajectory drawing color is obtained according to the texture information.

14. The method according to claim 9, wherein After receiving the writing event reported by the terminal device, the method further includes: drawing a second writing trajectory according to the writing event; The drawing data containing the second writing track is sent to the terminal device, so that the terminal device updates the current running interface of the cloud desktop according to the drawing data, and the first writing track drawn in the terminal device is covered on the second writing track in the running interface.

15. The method according to claim 9, wherein After receiving the writing event reported by the terminal device, the method further includes: When the writing event is no longer received, a stop writing state is generated; The stop writing state is fed back to the terminal device, so that the terminal device stops drawing the corresponding first writing track.

16. A terminal device comprising: One or more processors and a display screen, wherein the processor runs the cloud desktop client installed on the terminal device, and the display screen displays the cloud desktop operation interface when the cloud desktop client is running, and the operation interface is determined by the cloud desktop server, wherein: The processor reports a writing event generated in real time based on detecting a writing operation of the cloud desktop to the cloud desktop server, obtains the real-time writing event according to a writing state fed back by the cloud desktop server, and draws a corresponding first writing track according to the writing event, wherein the writing state is generated by the cloud desktop server according to a first received writing event; The display screen displays the first writing track after drawing in the running interface.

17. The terminal device according to claim 16, wherein: The processor acquires a real-time writing event according to the writing state fed back by the cloud desktop server, and draws a corresponding first writing track according to the writing event, including: The processor generates a writing layer according to the writing status fed back by the cloud desktop server, and instructs the display screen to display the writing layer, wherein the writing layer is overlaid on the current operation interface of the cloud desktop; The processor controls the writing layer to monitor real-time writing events of the cloud desktop, and draws a corresponding first writing track in the writing layer according to the writing event.

18. The terminal device according to claim 16, wherein: When the processor receives the writing status fed back by the cloud desktop server, it also receives the track drawing color fed back by the cloud desktop server, where the track drawing color is determined by the cloud desktop server; When the processor draws the corresponding first writing track according to the writing event, the processor further includes: The processor sets the track color of the first writing track to be drawn as the track drawing color, and draws the first writing track corresponding to the writing event according to the track drawing color.

19. The terminal device according to claim 16, wherein: When the processor receives the writing status fed back by the cloud desktop server, it also receives the writing coordinates fed back by the cloud desktop server, where the writing coordinates are the writing coordinates obtained by the cloud desktop server based on all currently received writing events; When the processor draws a corresponding first writing track according to the writing event, the method includes: The processor draws a corresponding first writing trajectory based on the writing event and the writing coordinates.

20. The terminal device according to claim 16, wherein: After the processor reports the writing event generated in real time based on the detection of the writing operation of the cloud desktop to the cloud desktop server, the method further includes: The processor receives drawing data sent by the cloud desktop server, where the drawing data includes a second writing trajectory drawn by the cloud desktop server according to the writing event; The processor updates the current operation interface of the cloud desktop according to the drawing data; The display screen displays the updated running interface, and the displayed first writing track covers the second writing track in the running interface.

21. A writing response device, applied to a cloud desktop server, wherein: The operating system running in the virtual machine of the cloud desktop server is the virtual operating system of the cloud desktop. The cloud desktop server sends the running interface of the cloud desktop to the terminal device, and the cloud desktop client installed on the terminal device displays the running interface during operation. The writing response device comprises: A first receiving unit is configured to receive a writing event reported by the terminal device, wherein the writing event is a writing event generated in real time by the terminal device when a writing operation of the cloud desktop is detected; A first generating unit is configured to generate a writing state when determining that writing for the cloud desktop occurs on the terminal device according to the first received writing event; The first feedback unit is configured to feed back the writing state to the terminal device, so that the terminal device obtains a real-time writing event according to the writing state and draws a corresponding first writing track according to the writing event.

22. A server, the server being a cloud desktop server, wherein: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the writing response method according to any one of claims 9 to 15.

23. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the writing display method according to any one of claims 1 to 8 or the writing response method according to any one of claims 9 to 15 is implemented.

Images