Cloud desktop display method and computer readable storage medium

Abstract

The invention discloses a cloud desktop display method and a computer readable storage medium, and the method comprises the steps that a cloud desktop constructs window layers corresponding to task windows in sequence according to the opening sequence of the task windows; when the cloud desktop image frame is intercepted, current coordinate information of each window image layer is acquired, and current frame data of each window image layer is intercepted; encoding the current frame data of each window layer to obtain encoded data of each window layer, and transmitting the encoded data and the current coordinate information of each window layer to a client, the encoded data and the current coordinate information of the most active layer being preferentially transmitted; the client decodes the coded data of the window layer corresponding to each task window to obtain the current frame data of each window layer; and respectively restoring the position of each task window according to the current coordinate information of each window layer, and performing synthetic display according to the current frame data of each window layer. The smoothness of the cloud desktop can be ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud desktops, and in particular, to a cloud desktop display method and a computer-readable storage medium. Background Art

[0002] Cloud desktop technology refers to that a user remotely accesses a virtual machine. An encoder in the virtual machine on the server side intercepts frames of the current desktop image of the virtual machine. After the desktop image sequence is compressed and encoded, it is transmitted to the user's local terminal through the network. The client decoder decodes the image sequence to obtain a real-time desktop image. By real-time encoding and decoding the frame images of the desktop through cloud desktop technology, the user can operate the remote virtual machine as smoothly as operating their own computer.

[0003] Due to limited server computing resources and bandwidth resources, this technology is greatly limited by the encoding and decoding methods of the cloud desktop protocol and the network bandwidth. To ensure the clarity and smoothness of the cloud desktop, both the encoding efficiency and the compression efficiency are crucial.

[0004] The traditional cloud desktop transmission scheme is to intercept, encode and decode images with a fixed frame size for the complete content of a single desktop, and then allocate encoding and decoding computing resources and bandwidth channels for each desktop. However, the cloud desktop system intercepts and encodes and decodes frames for each virtual desktop of each user, and at the same time, if it wants to use super-resolution technology to enhance the image quality of all desktops, it is bound to consume a large amount of computing resources, resulting in the smoothness of the desktop usage and the picture clarity of all users not being satisfied. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: to provide a cloud desktop display method and a computer-readable storage medium that can ensure the smoothness of the cloud desktop.

[0006] To solve the above technical problem, the technical solution adopted by the present invention is: a cloud desktop display method, including: The cloud desktop constructs window layers corresponding to each task window in sequence according to the opening order of the task windows; When intercepting the cloud desktop image frame, respectively obtain the current coordinate information of each window layer, and intercept the current frame data of each window layer according to the current coordinate information of each window layer; Encode the current frame data of each window layer respectively to obtain the encoded data of each window layer, and transmit the encoded data of each window layer and the current coordinate information to the client. Among them, the encoded data and the current coordinate information of the most active layer are preferentially transmitted, and the most active layer is the window layer corresponding to the topmost task window currently; The client decodes the encoded data of the window layer corresponding to each task window respectively to obtain the current frame data of each window layer; Restore the positions of the task windows respectively according to the current coordinate information of each window layer, and perform composite display according to the current frame data of each window layer.

[0007] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0008] The beneficial effect of the present invention is that by performing layer management on the task windows in the cloud desktop, the encoding, decoding and data transmission of the content of the most active window are preferentially ensured, thereby effectively ensuring the smoothness of the cloud desktop. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a flowchart of a cloud desktop display method of the present invention; Figure 2 is a flowchart of the method according to the first embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0010] To describe in detail the technical content, the achieved objectives and the effects of the present invention, the following is described in conjunction with the embodiments and with reference to the drawings.

[0011] Please refer to Figure 1 , a cloud desktop display method, including: The cloud desktop constructs window layers corresponding to the task windows in sequence according to the opening order of the task windows; When intercepting the cloud desktop image frame, respectively obtain the current coordinate information of each window layer, and intercept the current frame data of each window layer according to the current coordinate information of each window layer; Encode the current frame data of each window layer respectively to obtain the encoded data of each window layer, and transmit the encoded data of each window layer and the current coordinate information to the client. Among them, the encoded data and the current coordinate information of the most active layer are preferentially transmitted, and the most active layer is the window layer corresponding to the currently topmost task window; The client decodes the encoded data of the window layers corresponding to the task windows respectively to obtain the current frame data of each window layer; Restore the positions of the task windows respectively according to the current coordinate information of each window layer, and perform composite display according to the current frame data of each window layer.

[0012] As can be seen from the above description, the beneficial effect of the present invention is that the smoothness of the cloud desktop can be ensured.

[0013] Further, when intercepting the cloud desktop image frame, respectively obtaining the current coordinate information of each window layer includes: When intercepting the cloud desktop image frame, respectively obtain the current area positions of the task windows on the desktop; Calculate the difference set respectively according to the union of the current area positions of each task window and the current area positions of the task windows above it, to obtain the unobstructed area positions of each task window, which are used as the current coordinate information of the window layer corresponding to each task window.

[0014] As can be seen from the above description, only intercept the data of the unobstructed parts of each task window to reduce the data transmission volume and improve the encoding efficiency and response speed.

[0015] Furthermore, the step of respectively encoding the current frame data of each window layer to obtain the encoded data of each window layer, and transmitting the encoded data of each window layer and the current coordinate information to the client includes: Calculate the difference degree of each window layer according to the current frame data and the previous frame data of each window layer; If the difference degree of a window layer is greater than a preset difference threshold, then encode the current frame data of the said window layer to obtain the encoded data of the said window layer, and transmit the encoded data of the said window layer and the current coordinate information to the client.

[0016] As can be seen from the above description, by calculating the difference degree between the front and back two frame data of each window layer, if the difference degree between the front and back two frame data of a certain window layer is very small or there is no difference, it means that the window content has hardly changed, then the current frame data of this window layer can be not encoded, decoded and transmitted, so as to reduce the subsequent data transmission volume, and improve the encoding efficiency and response speed.

[0017] Furthermore, the step that the client respectively decodes the encoded data of the window layer corresponding to each task window to obtain the current frame data of each window layer includes: The client decodes the encoded data of the window layer corresponding to each task window in sequence according to the active order of each task window to obtain the current frame data of each window layer.

[0018] As can be seen from the above description, under the condition of limited resources, it is possible to ensure that the computing resources are allocated to the window content that the user cares most about as much as possible, and ensure the desktop picture quality.

[0019] Furthermore, after the client respectively decodes the encoded data of the window layer corresponding to each task window to obtain the current frame data of each window layer, it further includes: The client enhances the picture quality of the current frame data of the most active layer through super-resolution technology.

[0020] As can be seen from the above description, by separately enhancing the picture quality of the frame data of the most active layer, the clarity of the cloud desktop is ensured, and under the condition of limited terminal computing resources, the high-definition interaction and high responsiveness of the picture quality of the active window content are realized.

[0021] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned method is implemented.

[0022] Embodiment 1 Please refer to Figure 2 , Embodiment 1 of the present invention is: A cloud desktop display method, which can be applied to a cloud desktop system, such as Figure 2 shown, including the following steps: S1: The cloud desktop constructs window layers corresponding to each task window in sequence according to the opening order of the task windows.

[0023] Specifically, when the virtual machine of the cloud desktop cluster starts, window layers corresponding to each task window are constructed according to the order of the task windows opened by the user.

[0024] S2: When intercepting the cloud desktop image frame, the current coordinate information of each window layer is obtained respectively, and according to the current coordinate information of each window layer, the current frame data of each window layer is intercepted.

[0025] Specifically, when intercepting the desktop image frame of the cloud desktop, first detect the position of each task window (active window) on the desktop. Since the window is generally rectangular, the coordinates of the upper left corner point and the lower right corner point of each task window at this time can be recorded as the current area position of each task window on the desktop, such as L i [X start , Y start , X end , Y end , where (X start , Y start ) represents the coordinates of the upper left corner point, and (X end , Y end ) represents the coordinates of the lower right corner point.

[0026] Since a task window may be blocked by other task windows above it, by calculating the difference set of the union of the current position area of a task window and the current position areas of all the task windows above it, the current unblocked area of the task window can be obtained as the current coordinate information of the window layer corresponding to the task window. By analogy, the current coordinate information of the window layers corresponding to each task window is determined. Then, according to the current coordinate information of each window layer, the current frame data of each window layer is intercepted. That is, only the frame data of the unblocked part of each task window is intercepted as the current frame data of the window layer corresponding to each task window. If the result obtained by the difference set calculation is less than or equal to 0, it means that the task window is completely covered by the task window above it, and at this time, the frame data of the task window does not need to be intercepted.

[0027] For example, assume that the task windows from bottom to top are L 1 , L 2 , L 3 and L 4 . Then, the position L 1 ' of the uncovered area of task window L 1 = L 1 - (L 2 ∪ L 3 ∪ L 4 ). The position L 2 ' of the uncovered area of task window L 2 = L 2 - (L 3 ∪ L 4 ). The position L 3 ' of the uncovered area of task window L 3 = L 3 - L 4 . Assume that L 1 ' ≤ 0, which means that task window L 1 is completely covered, indicating that the frame data of this task window does not need to be intercepted at this time.

[0028] S3: Mark the window layer corresponding to the current topmost task window as the most active layer.

[0029] S4: Encode the current frame data of each window layer respectively to obtain the encoded data of each window layer, and transmit the encoded data of each window layer and the current coordinate information to the client.

[0030] In this embodiment, the encoded data and the current coordinate information of the most active layer are preferentially transmitted. For example, according to the active order of each task window (i.e., the order from the topmost to the bottommost), the current frame data of the window layer corresponding to each task window can be encoded in sequence, and the encoded data of each window layer and the current coordinate information can be transmitted to the client in sequence.

[0031] Furthermore, according to the current frame data and the previous frame data of each window layer, calculate the difference degree of each window layer. In the actual application scenario, the difference degree can be calculated by the diff operation. If the difference degree between the front and back two frame data of a certain window layer is very small or there is no difference, it means that the content of this task window has hardly changed, and the current frame data of this window layer is not encoded and transmitted. When the difference degree of the window layer is greater than the preset difference threshold, encoding and transmission are performed.

[0032] The cloud desktop system preferentially transmits and encodes the frame data of the active task window layer to ensure the content of the front task window that the user is most concerned about. If the content of other task windows has not changed, no encoding and decoding are performed.

[0033] S5: The client decodes the encoded data of the window layers corresponding to each task window respectively to obtain the current frame data of each window layer.

[0034] In this embodiment, the client decodes the encoded data of the window layers corresponding to each task window in sequence according to the active order of each task window to obtain the current frame data of each window layer.

[0035] Furthermore, the client enhances the image quality of the current frame data of the most active layer through super-resolution technology.

[0036] In the cloud desktop office scenario, users usually only care about the content of the topmost active window the most, and the content and image quality of other windows are not the top concern factors. Therefore, when resources are limited, it is necessary to ensure that computing resources are allocated to the window content that users care about the most as much as possible and ensure the desktop image quality.

[0037] S6: Restore the positions of each task window respectively according to the current coordinate information of each window layer, and perform composite display according to the current frame data of each window layer.

[0038] Specifically, restore the positions of each task window in the cloud desktop according to the current coordinate information of each window layer, and fill the desktop with the current frame data of each window layer in sequence according to the active order of each window layer. The window layer with a higher activity level covers the window layer with a lower activity level. Finally, composite the current frame data of all window layers to form a complete cloud desktop interface and display it on the terminal display.

[0039] This embodiment is based on the method of task window display layer scheduling to uniformly manage the active windows of active users in the cluster. Since the number of active users in the cluster mostly follows a normal distribution in most cases, and users care most about the content in the topmost window at the current time node, this embodiment performs layer management on the task windows in the cloud desktop through the method of task window layer scheduling. At the current time point, only the frame data transmission and super-resolution encoding and decoding processing are preferentially performed on the topmost active task layer window, using part of the storage space of the client to exchange for computing time, and improving the encoding efficiency and response speed of the desktop images of the entire cluster.

[0040] Embodiment 2 This embodiment is a computer-readable storage medium corresponding to the above embodiment, on which a computer program is stored. When the program is executed by a processor, it realizes each step of a cloud desktop display method as described in the above embodiment and can achieve the same technical effects, which will not be repeated here.

[0041] In summary, a cloud desktop display method and a computer-readable storage medium provided by the present invention manage the task windows in the cloud desktop in a layered manner, giving priority to ensuring the encoding, decoding, and data transmission of the content of the most active window. At the same time, by separately enhancing the picture quality of the frame data of the most active layer, the clarity of the cloud desktop is ensured, and high-definition interaction and high responsiveness of the content of the active window are achieved under the condition of limited terminal computing resources.

[0042] The above are only embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in the relevant technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A cloud desktop display method, characterized in that: include: The cloud desktop builds the window layers corresponding to each task window in sequence according to the order in which the task windows are opened; When capturing the cloud desktop image frame, the current coordinate information of each window layer is obtained respectively, and the current frame data of each window layer is captured according to the current coordinate information of each window layer; Encode the current frame data of each window layer respectively to obtain the encoded data of each window layer, and transmit the encoded data and current coordinate information of each window layer to the client, wherein the encoded data and current coordinate information of the most active layer are transmitted first, and the most active layer is the window layer corresponding to the current topmost task window; The client decodes the encoded data of the window layer corresponding to each task window to obtain the current frame data of each window layer; The position of each task window is restored according to the current coordinate information of each window layer, and a composite display is performed according to the current frame data of each window layer.

2. The cloud desktop display method according to claim 1, characterized in that: When capturing the cloud desktop image frame, the current coordinate information of each window layer is obtained respectively, including: When capturing the cloud desktop image frame, the current area position of each task window in the desktop is obtained respectively; Based on the union of the current area position of each task window and the current area position of the task window above it, a difference calculation is performed to obtain the unobstructed area position of each task window as the current coordinate information of the window layer corresponding to each task window.

3. The cloud desktop display method according to claim 1, characterized in that: The current frame data of each window layer is encoded respectively to obtain the encoded data of each window layer, and the encoded data and current coordinate information of each window layer are transmitted to the client, including: Calculate the difference between each window layer according to the current frame data and the previous frame data of each window layer; If the difference of a window layer is greater than a preset difference threshold, the current frame data of the window layer is encoded to obtain the encoded data of the window layer, and the encoded data of the window layer and the current coordinate information are transmitted to the client.

4. The cloud desktop display method according to claim 1, characterized in that: The client decodes the coded data of the window layer corresponding to each task window respectively to obtain the current frame data of each window layer, including: The client decodes the encoded data of the window layers corresponding to each task window in sequence according to the active order of each task window, and obtains the current frame data of each window layer.

5. The cloud desktop display method according to claim 1, characterized in that: The client decodes the coded data of the window layers corresponding to each task window respectively to obtain the current frame data of each window layer, further comprising: The client uses super-resolution technology to enhance the image quality of the current frame data of the most active layer.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the following steps are implemented: The cloud desktop builds the window layers corresponding to each task window in sequence according to the order in which the task windows are opened; When capturing the cloud desktop image frame, the current coordinate information of each window layer is obtained respectively, and the current frame data of each window layer is captured according to the current coordinate information of each window layer; Encode the current frame data of each window layer respectively to obtain the encoded data of each window layer, and transmit the encoded data and current coordinate information of each window layer to the client, wherein the encoded data and current coordinate information of the most active layer are transmitted first, and the most active layer is the window layer corresponding to the current topmost task window; The client decodes the encoded data of the window layer corresponding to each task window to obtain the current frame data of each window layer; The position of each task window is restored according to the current coordinate information of each window layer, and a composite display is performed according to the current frame data of each window layer.

7. The computer-readable storage medium according to claim 6, wherein: When capturing the cloud desktop image frame, the current coordinate information of each window layer is obtained respectively, including: When capturing the cloud desktop image frame, the current area position of each task window in the desktop is obtained respectively; Based on the union of the current area position of each task window and the current area position of the task window above it, a difference calculation is performed to obtain the unobstructed area position of each task window as the current coordinate information of the window layer corresponding to each task window.

8. The computer-readable storage medium according to claim 6, wherein: The current frame data of each window layer is encoded respectively to obtain the encoded data of each window layer, and the encoded data and current coordinate information of each window layer are transmitted to the client, including: Calculate the difference between each window layer according to the current frame data and the previous frame data of each window layer; If the difference of a window layer is greater than a preset difference threshold, the current frame data of the window layer is encoded to obtain the encoded data of the window layer, and the encoded data of the window layer and the current coordinate information are transmitted to the client.

9. The computer-readable storage medium according to claim 6, wherein: The client decodes the coded data of the window layer corresponding to each task window respectively to obtain the current frame data of each window layer, including: The client decodes the encoded data of the window layers corresponding to each task window in sequence according to the active order of each task window, and obtains the current frame data of each window layer.

10. The computer-readable storage medium according to claim 6, wherein: The client decodes the coded data of the window layers corresponding to each task window respectively to obtain the current frame data of each window layer, further comprising: The client uses super-resolution technology to enhance the image quality of the current frame data of the most active layer.

Images