Cloud Desktop System, Cloud Desktop Display Method, Terminal Device, and Storage Medium

Abstract

Embodiments of the present application provide a cloud desktop system, a cloud desktop display method, a terminal device, and a storage medium. In the embodiments of the present application, the layer data required by the cloud desktop is divided into two parts, one part is provided by the cloud server, and the other part is provided locally by the terminal. Further, combining the advantages of the terminal supporting the main layer and the overlay layer, for the layer data provided by the cloud server, the terminal only needs to decode and directly send it to the overlay layer for display, eliminating the need for the GPU to render this part of the layer data, reducing the latency overhead of inter-process communication caused by the GPU rendering process, and also reducing the time-consuming overhead caused by GPU rendering, reducing the display latency of the cloud desktop and improving the display efficiency; in addition, only the part of the layer data provided locally is rendered using the GPU, and the layer data provided by the cloud server is no longer rendered using the GPU, which can also reduce the power consumption of the terminal, and thus reduce the temperature rise of the terminal device.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular, to a cloud desktop system, a cloud desktop display method, a terminal device, and a storage medium. Background Art

[0002] A cloud desktop, also known as desktop virtualization or cloud computer, is a new mode that replaces traditional computers. The logic of a cloud desktop is that the data and computing of a traditional computer desktop are mainly carried out on a cloud server. After adopting a cloud desktop, users no longer need to purchase a computer host, and components such as the CPU, memory, and hard disk required for the host are all virtually obtained by the cloud server. Users can use their cloud desktops through various terminals such as personal computers and thin clients, achieving the same experience effect as traditional computers.

[0003] In a cloud desktop system, a cloud desktop client is installed on a local terminal, and interacts with a cloud server through a specific communication protocol to obtain cloud desktop data provided by the cloud server. Then, the cloud desktop data is rendered by means of the graphics rendering ability of the GPU, and the rendered cloud desktop is displayed. The cloud desktop is finally displayed on the terminal side. The smaller the display latency, the better the user experience. Therefore, it is necessary to optimize the rendering process of the existing cloud desktop to reduce the display latency of the cloud desktop and improve the user experience. Summary of the Invention

[0004] Multiple aspects of this application provide a cloud desktop system, a cloud desktop display method, a terminal device, and a storage medium, which are used to optimize the rendering process of the cloud desktop, reduce the display latency of the cloud desktop, and improve the user experience.

[0005] An embodiment of this application provides a cloud desktop system, including: a cloud server for providing a first part of layer data required for a cloud desktop, and a terminal device for providing a second part of layer data required for the cloud desktop and displaying the cloud desktop. The terminal device includes at least multiple hardware layers, and the multiple hardware layers include a main layer and at least one overlay layer; the cloud server is configured to send the first part of layer data to the terminal device according to the dynamic display requirements of the cloud desktop; the terminal device is configured to decode the first part of layer data sent by the cloud server to obtain first intermediate layer data, send the first intermediate layer data to a first overlay layer in the at least one overlay layer; and load the second part of layer data, call a graphics processing unit (GPU) to render the second part of layer data to obtain second intermediate layer data, and send the second intermediate layer data to the main layer; perform hardware synthesis on the first intermediate layer data and the second intermediate layer data to obtain first data to be displayed, and display the first data to be displayed to obtain the cloud desktop.

[0006] An embodiment of the present application further provides a terminal device, which is used to provide the second part of layer data required for a cloud desktop and display the cloud desktop. The terminal device includes: a computer processor, a memory, a graphics processing unit (GPU), a plurality of hardware layers, and a display controller. The plurality of hardware layers include a main layer and at least one overlay layer; the memory is used to store a computer program corresponding to a cloud desktop client. The computing processor is coupled to the memory and is used to execute the computer program to: receive the first part of layer data required for the cloud desktop sent by a cloud server according to the dynamic display requirements of the cloud desktop; decode the first part of layer data to obtain first intermediate layer data, and send the first intermediate layer data to a first overlay layer among the at least one overlay layer; and load the second part of layer data, call the GPU to render the second part of layer data to obtain second intermediate layer data, and send the second intermediate layer data to the main layer; the display is used to perform hardware composition on the first intermediate layer data in the first overlay layer and the second intermediate layer data in the main layer to obtain first data to be displayed, and display the first data to be displayed to obtain the cloud desktop.

[0007] An embodiment of the present application further provides a cloud desktop display method, which is applied to a terminal device. The terminal device is used to provide the second part of layer data required for a cloud desktop and display the cloud desktop. The method includes: receiving the first part of layer data required for the cloud desktop sent by a cloud server according to the dynamic display requirements of the cloud desktop; decoding the first part of layer data to obtain first intermediate layer data, and sending the first intermediate layer data to a first overlay layer among the at least one overlay layer included in the terminal device; loading the second part of layer data, calling the GPU to render the second part of layer data to obtain second intermediate layer data, and sending the second intermediate layer data to the main layer included in the terminal device; performing hardware composition on the first intermediate layer data and the second intermediate layer data to obtain first data to be displayed, and displaying the first data to be displayed to obtain the cloud desktop.

[0008] An embodiment of the present application further provides a cloud desktop display device, which is applied to a terminal device. The terminal device is used to provide the second part of the layer data required for the cloud desktop and display the cloud desktop. The device includes: a receiving module, configured to receive the first part of the layer data required for the cloud desktop sent by a cloud server according to the dynamic display requirements of the cloud desktop; a decoding module, configured to decode the first part of the layer data to obtain first intermediate layer data; a first display sending module, configured to send the first intermediate layer data to a first overlay layer in at least one overlay layer included in the terminal device; a loading module, configured to load the second part of the layer data; a rendering module, configured to call a GPU to render the second part of the layer data to obtain second intermediate layer data; a second display sending module, configured to send the second intermediate layer data to a main layer included in the terminal device; a display control module, configured to perform hardware synthesis on the first intermediate layer data and the second intermediate layer data to obtain first data to be displayed, and display the first data to be displayed to obtain the cloud desktop.

[0009] An embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is caused to be able to implement the steps in the cloud desktop display method provided by the embodiment of the present application.

[0010] In the embodiment of the present application, the layer data required for the cloud desktop is divided into two parts. One part is provided by the cloud server, and the other part is provided locally by the terminal. Further, combining the advantages of the terminal supporting the main layer and the overlay layer, for the layer data provided by the cloud server, the terminal only needs to directly send the decoded data to the overlay layer after decoding, omitting the link of rendering this part of the layer data by the GPU, reducing the latency overhead of inter-process communication caused by the GPU rendering link, and also reducing the time-consuming overhead caused by GPU rendering, reducing the display latency of the cloud desktop and improving the display efficiency; in addition, only using the GPU to render the part of the layer data provided locally, and no longer using the GPU to render the layer data provided by the cloud server, can also reduce the power consumption of the terminal, and thus reduce the temperature rise of the terminal device. Description of the Drawings

[0011] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0012] Figure 1 It is a schematic structural diagram of a cloud desktop system provided by an embodiment of the present application;

[0013] Figure 2A schematic structural diagram of a terminal device provided by an embodiment of the present application;

[0014] Figure 3a A schematic diagram of a software framework for cloud desktop display of the terminal device provided by an embodiment of the present application;

[0015] Figure 3b Another schematic diagram of a software framework for cloud desktop display of the terminal device provided by an embodiment of the present application;

[0016] Figure 4a Another schematic diagram of a software framework for cloud desktop display of the terminal device provided by an embodiment of the present application;

[0017] Figure 4b Another schematic diagram of a software framework for cloud desktop display of the terminal device provided by an embodiment of the present application;

[0018] Figure 5a A schematic flowchart of a cloud desktop display method provided by an embodiment of the present application;

[0019] Figure 5b Another schematic flowchart of a cloud desktop display method provided by an embodiment of the present application;

[0020] Figure 6 A schematic structural diagram of a cloud desktop display device provided by an embodiment of the present application. Detailed implementation manners

[0021] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0022] In view of the existing technical problem of latency, in some embodiments of the present application, the layer data required by the cloud desktop is divided into two parts, one part is provided by the cloud server, and the other part is provided locally by the terminal. Further, combining the advantages of the terminal in supporting the main layer and the overlay layer, for the layer data provided by the cloud server, the terminal only needs to decode it and directly send it to the overlay layer for display, eliminating the need for the GPU to render this part of the layer data, reducing the latency overhead of inter-process communication caused by the GPU rendering process, and also reducing the time-consuming overhead caused by GPU rendering, reducing the display latency of the cloud desktop and improving the display efficiency; in addition, only the part of the layer data provided locally is rendered using the GPU, and the layer data provided by the cloud server is no longer rendered using the GPU, which can also reduce the power consumption of the terminal and thus reduce the temperature rise of the terminal device.

[0023] The following will detail the technical solutions provided by each embodiment of the present application with reference to the accompanying drawings.

[0024] Figure 1 It is a schematic structural diagram of a cloud desktop system provided by an embodiment of the present application. As Figure 1 shown, the system 100 includes: a cloud server 10 and a terminal device 20. The cloud server 10 is communicatively connected to the terminal device 20. The communication connection method can be a wired connection method or a wireless connection method. The wireless connection method includes, but is not limited to: a wireless connection method based on a mobile communication network, WiFi, Bluetooth, infrared, etc. The present embodiment does not limit the mobile communication network standard. For example, it can be a 4G network, a 5G network, a 6G network, and other networks of other possible future standards.

[0025] In this embodiment, the implementation form of the terminal device is not limited. It can be various terminal devices with a display, communication ability, and certain computing ability, such as a laptop computer, a desktop computer, a tablet computer, a smart phone, a wearable device, etc. Similarly, the implementation form of the cloud server in the embodiments of the present application is not limited either. It can be a traditional cloud server or various cloud products in the form of a server cluster, virtual machine, container, etc. deployed in the cloud.

[0026] In this embodiment, the cloud server and the terminal device cooperate with each other to implement the cloud desktop system. In this embodiment, the cloud desktop system can provide cloud desktops. The cloud desktops in this embodiment require two parts of layer data, namely the first part of layer data and the second part of layer data. Among them, the first part of layer data is provided by the cloud server, and the second part of layer data is provided locally by the terminal device. The terminal device is also used to obtain the first part of layer data provided by the source server and display the cloud desktop locally according to the first part of layer data and the second part of layer data. Among them, the first part of layer data and the second part of layer data are different layer data. In addition, the terminal device 20 in this embodiment at least includes multiple hardware layers, and the multiple hardware layers include a primary layer (Primary Plane) and at least one overlay layer (Overlay Plane). The primary layer and the at least one overlay layer in this embodiment both belong to the hardware layer module and are at the hardware level. In this embodiment, the primary layer usually supports simple layers in RGB format, and the overlay layer usually supports one or more formats of layers, such as, for example, but not limited to, layers in YUV format.

[0027] In this embodiment, the cloud server 10 is mainly responsible for the relevant calculations of the cloud desktop system, such as the storage, rendering, streaming, etc. of the first part of layer data, and is also used to obtain the dynamic display requirements of the cloud desktop and send the first part of layer data to the terminal device 20 according to the dynamic display requirements of the cloud desktop. Among them, the first part of layer data can be some layer data that is required for the cloud desktop and whose content changes relatively frequently. For example, it can be video stream data played in the cloud desktop, and / or some interactive graphic and text data, graphic and text data with a high update frequency, or graphic and text data that is prone to change in the cloud desktop, etc. Among them, handing over these layer data with relatively frequent changes to the cloud server 10 for processing can make full use of the advantages of rich and flexible cloud resources and reduce the resource consumption and processing burden of the terminal device 20. Correspondingly, the second part of layer data can be some layer data that is required for the cloud desktop and whose content changes less frequently or remains basically unchanged. For example, it can be the UI layer data of the hidden menu required for the cloud desktop, and it can also be some layer data with a small amount of data, etc. The processing of these layer data is relatively simple and consumes less terminal resources, so it can be placed on the terminal side for processing, which can reduce the consumption of the bandwidth resources of the terminal device.

[0028] In some application scenarios, video images are played on a cloud desktop. For example, icons of various application software are displayed on the cloud desktop, and the user selects to open a video playback software. At this time, a video image will be played on the cloud desktop. When a video image is played on the cloud desktop, the dynamic display requirements of the cloud desktop are related to the playback progress of the video image played on the cloud desktop, and the display requirements of the cloud desktop will change dynamically with the change of the playback progress of the video image. The cloud server 10 can monitor the playback progress of the video image on the cloud desktop, determine the dynamic display requirements of the cloud desktop according to the playback progress of the video image, and the dynamic display requirements can reflect the video image content that the cloud desktop currently needs to display. The video image content is an example of the first part of the layer data provided by the cloud server 10.

[0029] In other application scenarios, in addition to playing video images, the cloud desktop window can also display some pages, such as web pages, which include some interactive controls, such as like controls, follow controls, favorite controls, comment controls, refresh controls, and link controls that can jump to other pages. Users can interact with the cloud desktop through these controls. When the cloud desktop supports user interaction, when the terminal device 20 displays the cloud desktop, it can also respond to interaction operations initiated by the user through the cloud desktop (such as interactive controls on the cloud desktop), such as refreshing the page, sharing the page, favoriting the page, or like, follow, comment, etc. interaction operations, and send a data acquisition request for the cloud desktop to the cloud server 10. The data acquisition request reflects the content that the cloud desktop currently needs to display. The cloud server 10 can determine the dynamic display requirements of the cloud desktop according to the data acquisition request, and then return the latest graphic and text data (including pictures, texts, or new page data, etc.) required by the cloud desktop to the terminal device 20 according to the dynamic display requirements. These graphic and text data are examples of the first part of the layer data provided by the cloud server 10.

[0030] Of course, in some other application scenarios, the cloud desktop can play video images and display relevant graphic and text data at the same time. For example, the cloud desktop can be implemented as a video playback interface (such as a web page), which includes a video playback area and a graphic and text display area. Users can initiate interaction operations related to the video image while watching the video image, such as controlling the playback progress of the video image, changing the playback episode of the video image, posting comments, displaying bullet screens, and so on.

[0031] When the cloud server 10 sends the first part of the layer data, the terminal device 20 can receive the first part of the layer data sent by the cloud server 10, decode the first part of the layer data sent by the cloud server 10 to obtain the first intermediate layer data, and send the first intermediate layer data to the first overlay layer in at least one overlay layer. In this embodiment, the decoding method for the first part of the layer data is not limited, and it can be determined specifically according to the encoding method adopted by the cloud server 10, such as H264, H265, JPEG, MPEG, etc. It should be noted that the terminal device 20 displays the cloud desktop window, relying on all the layer data required for the cloud desktop window. Therefore, the terminal device 20 will also load the second part of the layer data, call the graphics processing unit (GPU) to render the second part of the layer data to obtain the second intermediate layer data, and send the second intermediate layer data to the main layer; finally, perform hardware synthesis on the first intermediate layer data and the second intermediate layer data to obtain the first layer data to be displayed, and display the first layer data to be displayed to obtain the cloud desktop. Among them, displaying the first layer data to be displayed refers to the process of sending the first layer data to be displayed to the display of the terminal device for display.

[0032] Here it is explained that the first layer data to be displayed is the layer data synthesized from the first intermediate layer data and the second intermediate layer data. The reason for prefixing "first" is to facilitate differentiation from other synthesized layer data. Here, "first" has no limitation on quantity and sequence. Similarly, the "first" and "second" in front of the intermediate layer data are also for facilitating the differentiation of different intermediate layer data, and "first" and "second" have no limitation on quantity and sequence. The sequence of obtaining the first intermediate layer data and the second intermediate layer data is not limited, and the two steps can be executed in any sequence or in parallel.

[0033] In this embodiment, for the first part of the layer data provided by the cloud server, the terminal device only needs to decode it and then directly send it to the first overlay layer without rendering it through the GPU, saving the link of rendering this part of the layer data through the GPU, reducing the latency overhead of inter-process communication caused by the GPU rendering link, and also reducing the time-consuming overhead caused by GPU rendering, reducing the display latency of the cloud desktop, and improving the display efficiency; in addition, in this embodiment, only the second part of the layer data provided locally is rendered using the GPU, and the layer data provided by the cloud server is no longer rendered using the GPU, which can also reduce the power consumption of the terminal and thus reduce the temperature rise of the terminal device.

[0034] Of course, the reason why the GPU can be omitted from rendering the first part of the layer data is that the terminal device uses multiple hardware layers. By leveraging the advantage of the terminal device's support for stacked layers, the two parts of the layer data are hardware synthesized through two hardware layers, instead of using the GPU to render and synthesize the two parts of the layer data. This helps reduce latency, save GPU resources, and lower power consumption, etc.

[0035] In the embodiments of the present application, as Figure 2 shown, the terminal device 20 includes, but is not limited to, the following components: a computer processor 201, a memory 202, a graphics processing unit GPU 203, multiple hardware layers 204, a display controller 205, and a display 206. The multiple hardware layers include a main layer and at least one stacked layer. Among them, the computing processor 201 can be a CPU. The display controller 205 is a hardware module. In an optional embodiment, the computer processor 201, the memory 202, the graphics processing unit GPU 203, the multiple hardware layers 204, and the display controller 205, etc., can be integrated on one hardware chip or distributed on different hardware chips, but they can communicate with each other. The following will describe the process of the terminal device 20 cooperating with the cloud server 10 for cloud desktop display in combination with the implementation structure of the terminal device 20.

[0036] Among them, a cloud desktop client is installed on the terminal device 20. By running the cloud desktop client, it can interact with the cloud server 10. Among them, the cloud desktop client and the cloud server can interact through a cloud desktop transmission protocol. The embodiments of the present application do not limit the cloud desktop transmission protocol used. For example, it can be some general cloud transmission protocols or custom transmission protocols. Optionally, the cloud desktop client can use, but is not limited to, the Adaptive Stream Protocol (ASP) to communicate with the cloud server 10, for obtaining the first part of the layer data provided by the cloud server 10 and completing the display of the cloud desktop based on the first part of the layer data and the second part of the layer data. Specifically, a computer program corresponding to the cloud desktop client is stored in the memory 202. The computing processor 201 is coupled with the memory 202 and is used to execute the computer program corresponding to the cloud client for: receiving the first part of the layer data required for the cloud desktop sent by the cloud server 10 according to the dynamic display requirements of the cloud desktop; decoding the first part of the layer data to obtain the first intermediate layer data and sending the first intermediate layer data to the first stacked layer in at least one stacked layer; and loading the second part of the layer data into the CPU cache, calling the GPU to render the second part of the layer data to obtain the second intermediate layer data, and sending the second intermediate layer data to the main layer.

[0037] The display controller 205 is used to obtain the first intermediate layer data from the first overlay layer and the second intermediate layer data from the main layer, and then perform hardware synthesis on the first intermediate layer data and the second intermediate layer data to obtain the first layer data to be displayed, and display the first layer data to be displayed to obtain the cloud desktop. Displaying the first layer data to be displayed refers to the process of sending the first layer data to be displayed to the display 206 of the terminal device for display.

[0038] In an optional embodiment, the layer data required for the cloud desktop includes multiple types, such as three or more types. Among them, the first part of the layer data includes at least two types of layer data, and the second part of the layer data includes at least one type of layer data. For the convenience of distinction and description, the layer data in the first part of the layer data is called cloud layer data, and the layer data in the second part of the layer data is called local layer data. There are at least two types of cloud layer data, and there is at least one type of local layer data. For different types of cloud layer data, the cloud server 10 will only provide one type of cloud layer data to the terminal device 20 at the same time. For the cloud server 10, according to the dynamic display requirements of the cloud desktop, each time the target cloud layer data that is currently adapted to the dynamic display requirements can be determined from at least two types of cloud layer data, and the target cloud layer data can be sent to the terminal device. The target cloud layer data is one of the at least two types of cloud layer data.

[0039] Accordingly, the computing processor 201 is specifically used for: receiving the target cloud layer data sent by the cloud server 10 each time and adapted to the dynamic display requirement; decoding the target cloud layer data to obtain the first intermediate layer data, and sending the first intermediate layer data to the first overlay layer for display. Wherein, in the case where the first part of the layer data includes at least two types of cloud layer data, the first overlay layer is an overlay layer that simultaneously supports at least two types of cloud layer data in at least one overlay layer. In other words, at least two types of cloud layer data can share the same overlay layer in time-sharing, and the overlay layer simultaneously supports at least two types of data formats. In addition, in the case where the second part of the layer data includes at least one type of local layer data, the computing processor 201 is specifically used for: simultaneously loading at least one type of local layer data, calling the GPU to simultaneously render at least one type of local layer data to obtain the second intermediate layer data, and sending the second intermediate layer data to the main layer for display.

[0040] In a cloud desktop application scenario, the cloud desktop includes three types of layer data, denoted as the first initial layer data, the third initial layer data, and the fourth initial layer data. Among them, the initial layer data is relative to the intermediate layer data and the layer data to be displayed. Its essence is also a type of layer data. Here, the "first", "third", and "fourth" are for convenience of distinction and have no limitations on quantity and sequence. Among them, the first initial layer data belongs to the local layer data, which refers to the window menu content data required by the cloud desktop and can be simply referred to as the hidden menu UI layer data; the third initial layer data belongs to the cloud layer data, which refers to the video stream data required by the cloud desktop; the fourth initial layer data belongs to the cloud layer data, which refers to the graphic and text data required by the cloud desktop.

[0041] Among them, the first initial layer data, that is, the hidden menu UI layer data, is drawn locally on the terminal device through a graphics library. All kinds of graphics libraries are applicable to the embodiments of the present application. For example, QT is an optional graphics library. QT is a cross-platform C++ graphical user interface application framework that provides various functions and APIs required for application developers to build artistic graphical user interfaces. The third initial layer data, that is, the video stream data, is a type of data used when playing videos on the cloud desktop. Its format can be, but is not limited to, YUV420. YUV420 is a graphics pixel format and can be simply referred to as the Stream layer data. This layer data is sent by the cloud server 10. The fourth initial layer data, that is, the graphic and text data, carries the data after changes in some text areas or picture areas when browsing the web on the cloud desktop. Optionally, its format can be, but is not limited to, QXL. QXL is a graphics transmission format and can be simply referred to as the QXL layer data. This layer data is sent by the cloud server 10 and will be displayed in the BGRA format after being sent to the terminal device. The video stream data is usually full-screen data. Of course, for full-screen data, it can only contain video stream data, or can also contain both video stream data and graphic and text data. The graphic and text data can be full-screen data, or can also be regional stream data or command stream data. Among them, full-screen data refers to the data corresponding to the entire hardware layer, while regional stream data or command stream data refers to the data corresponding to some areas in the hardware layer.

[0042] In terms of the layer order, the hidden menu UI layer data of the cloud desktop is placed at the top layer of the cloud desktop, while the video stream data (i.e., the Stream layer data) is in the middle, and the graphic and text data (i.e., the QXL layer data) is at the bottom. For users, among the three types of layer data included in the cloud desktop, the changes in the video stream data and the graphic and text data are the most perceptible parts. Therefore, it is necessary to present them to the display with as small a time delay as possible.

[0043] For the above scenario, as Figure 3aShown is a software framework for a terminal device to perform cloud desktop display. In this software framework, on the one hand, the computing processor 201 receives the third initial layer data (i.e., video stream data) or the fourth initial layer data (i.e., graphic and text data) sent by the cloud server 10, decodes the third initial layer data (i.e., video stream data) or the fourth initial layer data (i.e., graphic and text data) to obtain the first intermediate layer data, and directly writes the first intermediate layer data into the cache of direct memory access (DMA) without GPU rendering, and provides the first intermediate layer data to the first overlay layer through DMA. Among them, the process of providing the first intermediate layer data to the first overlay layer through DMA includes: sending the cache ID of the first intermediate layer data in the DMA cache to the first overlay layer by calling the display driver interface of the first overlay layer, and the first overlay layer reads the first intermediate layer data from the DMA cache according to this cache ID, and this cache ID represents the storage location of the first intermediate layer data in the DMA cache. In Figure 3a In the shown software framework, the computing processor 201 directly calls the display driver interface provided by the operating system at the cloud desktop client layer to set the first overlay layer.

[0044] It should be noted that the display driver interfaces of each overlay layer can be provided by the display driver framework in the operating system. According to the different operating systems used by the terminal device, the display driver framework will also be different. In this embodiment, taking operating systems such as Linux as an example, the Direct Rendering Manager (DRM) is the display driver framework at the Linux kernel layer. It encapsulates the display function into standard interfaces such as open / close / ioctl. Programs in the user space (such as the cloud desktop client) can call these interfaces to drive display-related devices (such as overlay layers or main layers) to perform data display. These standard interfaces provided by DRM are encapsulated into a display driver library, such as the libdrm library, so that users can perform display control more conveniently. In Figure 3a In the shown software framework, the first intermediate layer data is sent to the first overlay layer for display by calling the display driver interface in the display driver library (such as the libdrm library).

[0045] On the other hand, as Figure 3aAs shown, the computing processor 201 is also used to load the first initial layer data (i.e., the hidden menu UI layer data), and initialize the rendering state of the first initial layer data, encapsulating the first initial layer data and its rendering state into a first data packet; then, calling the GPU to render the first data packet to obtain the third intermediate layer data; further, calling the GPU to render and synthesize the second initial layer data provided by the window manager and the third intermediate layer data to obtain the second intermediate layer data, where the second initial layer data refers to the window menu style data required for the cloud desktop. Among them, the rendering state includes physical texture mapping, material properties, and shaders compiled into binary files. Information related to lighting and the camera is also passed to the GPU along with the rendering state.

[0046] It should be noted that according to the different operating systems used by the terminal device, the specific implementation process of rendering the second intermediate layer data as described above will also be different. In the architectures of some operating systems, such as Linux, Ubuntu, Kubuntu, etc., when the operating system needs to provide an interface, the operating system will establish one or several graphics interface servers (such as X-Server / Wayland in the Linux system), interact with the window manager through a window transmission protocol (such as X-Protocol / Wayland Protocol in the Linux system), and an application independent of the operating system is used to generate an interactive interface such as windows, status bars, and buttons. Among them, this application is called a graphics interface client (such as X-Client in the Linux system), which is an application for displaying graphics on the terminal device and refers to the cloud desktop client in the embodiments of the present application.

[0047] As Figure 3a shown, in the architecture of the graphics interface client and the graphics interface server, the graphics interface client is responsible for initializing the rendering state of the first initial layer data, encapsulating the first initial layer data and its rendering state into a first data packet; then, calling the GPU to render the first data packet through OpenGL to obtain the third intermediate layer data, and then calling the interface function in the client interface library to send the third intermediate layer data to the graphics interface server. OpenGL is a set of API specifications for calling GPU functions. It defines a series of function APIs for operating graphics and pictures, and the APIs defined by this specification can be used to call the GPU. Taking the Linux system as an example, the client interface library is Xlib. Xlib is an interface library for the graphics interface client under the X Window System protocol written in the C language, and it contains functions for communicating with the graphics interface server (such as X-Server / Wayland in the Linux system). Further, as Figure 3aAs shown in the figure, the graphics interface server is responsible for obtaining the second initial layer data provided by the window manager, calling the GPU to render and synthesize the second initial layer data and the third intermediate layer data provided by the window manager to obtain the second intermediate layer data, and then calling the display driver interface provided by the display driver library to send the second intermediate layer data to the main layer for display. Finally, the display controller 205 obtains the first intermediate layer data from the first overlay layer and the second intermediate layer data from the main layer, and then performs hardware synthesis on the first intermediate layer data and the second intermediate layer data to obtain the first data to be displayed layer data, and displays the first data to be displayed layer data to obtain the cloud desktop.

[0048] Further, in an optional embodiment, the third initial layer data or the fourth initial layer data may be a command stream or a region stream. For these command streams or region streams, the window manager may further provide the position information of the first intermediate layer data (or the third initial layer data or the fourth initial layer data) in the cloud desktop. Based on this, the computer processor 201 may further call the graphics interface server to send the first intermediate layer data to the corresponding position in the first overlay layer according to the position information of the first intermediate layer data provided by the window manager in the cloud desktop. Further, as Figure 3b shown, it is another software framework for the cloud desktop display of the terminal device. In this software framework, still taking the interaction between the graphics interface client and the graphics interface server as an example, the processing process of the window menu UI layer data (i.e., the first original layer data) is the same as that Figure 3a shown in the figure, which will not be elaborated here. In Figure 3b , the computing processor 201 will also receive the third initial layer data (i.e., video stream data) or the fourth initial layer data (i.e., graphic and text data) sent by the cloud server 10, decode the third initial layer data (i.e., video stream data) or the fourth initial layer data (i.e., graphic and text data) to obtain the first intermediate layer data, and directly write the first intermediate layer data into the cache of the direct memory access (DMA) without GPU rendering, which is different from the Figure 3a embodiment shown in the figure. In Figure 3bIn the shown embodiment, the first intermediate layer data is not directly sent to the first overlay layer by calling the display driver interface. Instead, the first intermediate layer data is sent to the graphics interface server through the interface function in the client interface library. The graphics interface server, according to the position information of the first intermediate layer data (or the third initial layer data or the fourth initial layer data) provided by the window manager in the cloud desktop, sends the first intermediate layer data to the first overlay layer by calling the display driver interface. Finally, the display controller 205 obtains the first intermediate layer data from the first overlay layer and obtains the second intermediate layer data from the main layer. Then, the first intermediate layer data and the second intermediate layer data are hardware synthesized to obtain the first data to be displayed layer, and the first data to be displayed layer is displayed to obtain the cloud desktop.

[0049] Further, in each embodiment of the present application, before sending the first intermediate layer data to the first overlay layer, the computing processor 201 is further configured to: select, from at least one overlay layer, an overlay layer that simultaneously supports at least two types of cloud layer data as the first overlay layer according to the capability information of at least one overlay layer. The capability information of each overlay layer includes the data format supported by the overlay layer, such as only supporting the YUV format, only supporting the RGB format, or simultaneously supporting the YUV format and the RGB format, etc.

[0050] It should be noted that in practical applications, the number of overlay layers included in different terminal devices and the capability information of the overlay layers are different. Whether the computing processor 201 can select the first overlay layer from at least one overlay layer depends on the capability information of the overlay layers included in the terminal device. In some application scenarios, there may be no overlay layer in the overlay layers included in the terminal device that simultaneously supports at least two types of cloud layer data, and the computing processor 201 will not be able to select the first overlay layer from them. In view of this situation, in this embodiment, the computer processor 201 is further configured to: in the case where the first overlay layer cannot be selected from at least one overlay layer, obtain at least one second overlay layer from at least one overlay layer, and each second overlay layer supports one type of cloud layer data. Among them, the number of second overlay layers can be one or more, which can be the same as the number of cloud layer data, or less than the number of cloud layer data. When the number of second overlay layers is the same as the number of cloud layer data, it means that each cloud layer data can be sent to an independent second overlay layer; when the number of second overlay layers is less than the number of cloud layer data, it means that some cloud layer data cannot be directly sent to the second overlay layer, and for this part of the cloud layer data, the GPU needs to be called for rendering.

[0051] Based on the above, in the case where the first overlay layer cannot be selected from at least one overlay layer, but at least one second overlay layer is selected, each time the computer processor 201 receives the target cloud layer data provided by the cloud server, it determines whether there is a second overlay layer corresponding to the target cloud layer data according to the type of the target cloud layer data; if there is a second overlay layer corresponding to the target cloud layer data, the target cloud layer data is decoded to obtain the fourth intermediate layer data, and the fourth intermediate layer data is directly sent to the corresponding second overlay layer without GPU rendering; on the other hand, the computer processor 201 also loads at least one type of local layer data, calls the GPU to render at least one type of local layer data to obtain the fifth intermediate layer data, and sends the fifth layer data to the main layer. Correspondingly, the display controller 205 is further configured to: obtain the fourth intermediate layer data from the second overlay layer, and obtain the fifth intermediate layer data from the main layer; perform hardware synthesis according to the fourth intermediate layer data and the fifth intermediate layer data to obtain the second data to be displayed layer data, and display the second data to be displayed layer data to obtain a cloud desktop. It should be noted that in this embodiment, the process of the cloud server providing the target cloud layer data to the terminal device is the same as that in the foregoing embodiment. In addition, the process of calling the GPU to render at least one type of local layer data to obtain the fifth intermediate layer data is the same as the process of obtaining the second intermediate layer data, and will not be elaborated herein.

[0052] Further optionally, if there is no second overlay layer corresponding to the target cloud layer data, the computer processor is further configured to decode the target cloud layer data to obtain the eighth intermediate layer data, call the GPU to perform rendering synthesis on the eighth intermediate layer data and at least one type of local layer data to obtain the sixth intermediate layer data; send the sixth intermediate layer data into the main layer as the third data to be displayed layer data; correspondingly, the display is further configured to: obtain the third data to be displayed layer data from the main layer, and display the third data to be displayed layer data to obtain a cloud desktop.

[0053] Continuing with the above embodiment, taking the cloud desktop including the first initial layer data (such as the hidden menu UI layer data), the third initial layer data (such as the Stream layer data), and the fourth initial layer data (such as the QXL layer data) as an example, a software framework for a terminal device to display a cloud desktop is as Figure 4aAs shown, in this software framework, the terminal device includes two second overlay layers and a main layer, and the two second overlay layers support different data formats. Among them, the main layer corresponds to the first initial layer data (such as the hidden menu UI layer data) and is used to display the fifth intermediate layer data rendered from the first initial layer data. One second overlay layer corresponds to the third initial layer data, supports the data format of the third initial layer data, and is used to display the fourth intermediate layer data decoded from the third initial layer data (such as the Stream layer data). The other second overlay layer corresponds to the fourth initial layer data, supports the data format of the fourth initial layer data, and is used to display the fourth intermediate layer data decoded from the fourth initial layer data (such as the QXL layer data).

[0054] Continuing with the above embodiment, taking the cloud desktop including the first initial layer data (such as the hidden menu UI layer data), the third initial layer data (such as the Stream layer data), and the fourth initial layer data (such as the QXL layer data) as an example, as Figure 4b shown, this is another software framework for a terminal device to display a cloud desktop. In this software framework, the terminal device includes a second overlay layer and a main layer. Among them, the second overlay layer corresponds to the third initial layer data and is used to display the fourth intermediate layer data decoded from the third initial layer data (such as the Stream layer data). Regarding the fourth initial layer data, it is necessary to decode it to obtain the eighth intermediate layer data, and then call the GPU to render and synthesize the eighth intermediate layer data and the first initial layer to obtain the sixth intermediate layer data. That is to say, the main layer corresponds to the first initial layer data and the fourth initial layer data and is used to display the sixth intermediate layer data rendered and synthesized from the first initial layer data and the fourth initial layer data.

[0055] Further, the computer processor is further configured to: in the case where the first overlay layer and any second overlay layer cannot be selected from at least one overlay layer, each time the target cloud layer data provided by the cloud server is received, decode the target cloud data to obtain the ninth intermediate layer data, call the GPU to render and synthesize the ninth intermediate layer data and at least one type of local layer data to obtain the seventh intermediate layer data; send the seventh intermediate layer data as the fourth layer data to be displayed to the main layer, and display the fourth layer data to be displayed to obtain the cloud desktop. When the computer processor loads the local layer data (such as the hidden menu UI layer data) and receives the target cloud layer data (such as the Stream layer data / QXL layer data) sent by the cloud server, it will initialize the rendering state of these layer data, pack each layer data with its rendering state, and then call the GPU for rendering to obtain the seventh intermediate layer; send the seventh intermediate layer data as the fourth layer data to be displayed to the main layer, and display the fourth layer data to be displayed to obtain the cloud desktop. It should be noted that, Figure 4a - Figure 4b The architecture of the graphical interface client interacting with the graphical interface server is still taken as an example for illustration. For the relevant implementation logic, reference can be made to the foregoing embodiments, and details are not described herein again.

[0056] It should be noted that, Figure 2 Only some components of the terminal device are shown in. Optionally, the terminal device may further include other components such as a communication component, a display, a power supply component, and an audio component.

[0057] Figure 5a It is a schematic flowchart of a cloud desktop display method provided by an embodiment of the present application. This method is applied to a terminal device, and the terminal device is used to provide the second part of the layer data required for the cloud desktop and display the cloud desktop. For the implementation structure of the terminal device, reference can be made to the foregoing embodiments, and details are not described herein again. As Figure 5a shown, the method includes:

[0058] 51a. Receive the first part of the layer data required for the cloud desktop sent by the cloud server according to the dynamic display requirements of the cloud desktop;

[0059] 52a. Decode the first part of the layer data to obtain the first intermediate layer data, and send the first intermediate layer data to the first overlay layer in at least one overlay layer included in the terminal device;

[0060] 53a. Load the second part of the layer data, call the GPU to render the second part of the layer data to obtain the second intermediate layer data, and send the second intermediate layer data to the main layer included in the terminal device;

[0061] 54a. Hardware - synthesize the first intermediate layer data and the second intermediate layer data to obtain the first layer data to be displayed, and display the first layer data to be displayed to obtain a cloud desktop.

[0062] In an alternative embodiment, sending the first intermediate layer data to the first overlay layer among at least one overlay layer included in the terminal device includes: directly sending the first intermediate layer data to the first overlay layer without GPU rendering.

[0063] In an alternative embodiment, the first part of the layer data includes at least two types of cloud - layer data, and the first overlay layer is an overlay layer among at least one overlay layer that supports at least two types of cloud - layer data simultaneously. Correspondingly, receiving the first part of the layer data required for the cloud desktop sent by the cloud server according to the dynamic display requirements of the cloud desktop includes: each time receiving the target cloud - layer data adapted to the dynamic display requirements sent by the cloud server, where the target cloud - layer data is one of at least two types of cloud - layer data; decoding the first part of the layer data to obtain the first intermediate layer data includes: decoding the target cloud - layer data to obtain the first intermediate layer data.

[0064] In an alternative embodiment, sending the first intermediate layer data to the first overlay layer among at least one overlay layer included in the terminal device includes: directly writing the first intermediate layer into the cache of the DMA without GPU rendering, and providing the first intermediate layer data to the first overlay layer through the DMA.

[0065] In an alternative embodiment, the second part of the layer data includes at least one type of local layer data; loading the second part of the layer data and calling the GPU to render the second part of the layer data to obtain the second intermediate layer data includes: simultaneously loading at least one type of local layer data and calling the GPU to simultaneously render at least one type of local layer data to obtain the second intermediate layer data.

[0066] In an alternative embodiment, at least one type of local layer data includes the first initial layer data, where the first initial layer data refers to the window menu content data required for the cloud desktop. Correspondingly, calling the GPU to simultaneously render at least one type of local layer data to obtain the second intermediate layer data includes: initializing the rendering state of the first initial layer data, encapsulating the first initial layer data and its rendering state into a first data packet, calling the GPU to render the first data packet to obtain the third intermediate layer data; calling the GPU to render and synthesize the second initial layer data provided by the window manager and the third intermediate layer data to obtain the second intermediate layer data, where the second initial layer data refers to the window menu style data required for the cloud desktop.

[0067] In an alternative embodiment, at least two types of cloud layer data include third initial layer data and fourth initial layer data. The third initial layer data refers to video stream data required for a cloud desktop, and the fourth initial layer data refers to graphic and text data required for a cloud desktop. Sending the first intermediate layer data to the first overlay layer among at least one overlay layer included in the above-mentioned sending to a terminal device includes: calling a graphics interface server to send the first intermediate layer data to a corresponding position of the first overlay layer according to the position information of the first intermediate layer data in the cloud desktop provided by a window manager.

[0068] In an alternative embodiment, the method of this embodiment further includes: selecting, as the first overlay layer, an overlay layer that simultaneously supports at least two types of cloud layer data from at least one overlay layer according to the capability information of at least one overlay coating. The capability information of the overlay layer includes the data formats supported by the overlay layer.

[0069] In an alternative embodiment, the method of this embodiment further includes: in a case where the first overlay layer cannot be selected from at least one overlay layer, obtaining at least one second overlay layer from at least one overlay layer, and each second overlay layer supports one type of cloud layer data.

[0070] Based on the above, the embodiments of the present application further provide another cloud desktop display method, wherein, Figure 5b is a schematic flowchart of another cloud desktop display method provided by the embodiments of the present application. This method is applied to a terminal device, such as Figure 5b , and this method includes:

[0071] 51b. Each time target cloud layer data provided by a cloud server is received, determining whether there is a second overlay layer corresponding to the target cloud layer data according to the type of the target cloud layer data;

[0072] 52b. If there is a second overlay layer corresponding to the target cloud layer data, decoding the target cloud layer data to obtain fourth intermediate layer data, and directly sending the fourth intermediate layer data to the corresponding second overlay layer without GPU rendering;

[0073] 53b. Loading at least one type of local layer data, and calling a GPU to render at least one type of local layer data to obtain fifth intermediate layer data, and sending the fifth layer data to a main layer;

[0074] 54b. Performing hardware synthesis on the fourth intermediate layer data and the fifth intermediate layer data to obtain second data to be displayed, and displaying the second data to be displayed to obtain a cloud desktop.

[0075] Further optionally, the method of this embodiment further includes: if there is no second overlay layer corresponding to the target cloud layer data, decoding the target cloud layer data to obtain eighth intermediate layer data, and invoking the GPU to render and synthesize the eighth intermediate layer data and the at least one type of local layer data to obtain sixth intermediate layer data; sending the sixth intermediate layer data as third data to be displayed into the main layer; and displaying the third data to be displayed to obtain a cloud desktop.

[0076] Further optionally, the method of this embodiment further includes: in the case where the first overlay layer and any second overlay layer cannot be selected from at least one overlay layer, each time the target cloud layer data provided by the cloud server is received, decoding the target cloud layer data to obtain ninth intermediate layer data, and invoking the GPU to render and synthesize the ninth intermediate layer data and the at least one type of local layer data to obtain seventh intermediate layer data; sending the seventh intermediate layer data as fourth data to be displayed into the main layer, and displaying the fourth data to be displayed to obtain a cloud desktop.

[0077] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can also be executed by different devices. For example, the execution subject of steps 51a to 51d can be device A; for another example, the execution subject of steps 51a and 51b can be device A, and the execution subject of step 53c can be device B; and so on.

[0078] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appear in a specific order, but it should be clearly understood that these operations can be executed not in the order in which they appear in this article or in parallel. The operation numbers such as 51a and 52a are only used to distinguish different operations, and the numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and these operations can be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit that "first" and "second" are of different types.

[0079] Figure 6 This is a schematic structural diagram of a cloud desktop display device provided by an embodiment of the present application. The device can be applied to a terminal device, and the terminal device is used to provide the second part of the layer data required for the cloud desktop and display the cloud desktop, such as Figure 6 As shown, the device includes: a receiving module 61, a decoding module 62, a first sending and displaying module 63, a loading module 64, a rendering module 65, a second sending and displaying module 66, and a display control module 67.

[0080] A receiving module 61, configured to receive a first part of layer data required for a cloud desktop sent by a cloud server according to the dynamic display requirements of the cloud desktop.

[0081] A decoding module 62, configured to decode the first part of layer data to obtain first intermediate layer data.

[0082] A first display sending module 63, configured to send and display the first intermediate layer data to a first overlay layer among at least one overlay layer included in a terminal device.

[0083] A loading module 64, configured to load a second part of layer data.

[0084] A rendering module 65, configured to call a GPU to render the second part of layer data to obtain second intermediate layer data.

[0085] A second display sending module 66, configured to send and display the second intermediate layer data to a main layer included in a terminal device.

[0086] A display control module 67, configured to perform hardware synthesis on the first intermediate layer data and the second intermediate layer data to obtain first data to be displayed, and display the first data to be displayed to obtain a cloud desktop.

[0087] In an optional embodiment, the first display sending module 63 is specifically configured to: directly send and display the first intermediate layer data to the first overlay layer without GPU rendering.

[0088] In an optional embodiment, the first part of layer data includes at least two types of cloud layer data, and the first overlay layer is an overlay layer that supports at least two types of cloud layer data among at least one overlay layer. Accordingly, the receiving module 61 is specifically configured to: each time receive target cloud layer data adapted to the dynamic display requirements sent by the cloud server, where the target cloud layer data is one of at least two types of cloud layer data; decode the first part of layer data to obtain first intermediate layer data, including: decoding the target cloud layer data to obtain first intermediate layer data.

[0089] In an optional embodiment, the first display sending module 63 is specifically configured to: directly write the first intermediate layer into a cache of a DMA without GPU rendering, and provide the first intermediate layer data to the first overlay layer through the DMA.

[0090] In an optional embodiment, the second part of layer data includes at least one type of local layer data; the loading module 64 is specifically configured to: simultaneously load at least one type of local layer data, and call a GPU to simultaneously render at least one type of local layer data to obtain second intermediate layer data.

[0091] In an alternative embodiment, at least one type of local layer data includes first initial layer data, which refers to the window menu content data required for the cloud desktop. Accordingly, the rendering module 65 is specifically configured to: initialize the rendering state of the first initial layer data, encapsulate the first initial layer data and its rendering state into a first data packet, and call the GPU to render the first data packet to obtain third intermediate layer data; call the GPU to render and synthesize the second initial layer data provided by the window manager and the third intermediate layer data to obtain second intermediate layer data, where the second initial layer data refers to the window menu style data required for the cloud desktop.

[0092] In an alternative embodiment, at least two types of cloud layer data include third initial layer data and fourth initial layer data, where the third initial layer data refers to the video stream data required for the cloud desktop, and the fourth initial layer data refers to the graphic and text data required for the cloud desktop. The first display sending module 63 is specifically configured to: according to the position information of the first intermediate layer data provided by the window manager in the cloud desktop, call the graphics interface server to send and display the first intermediate layer data to the corresponding position of the first overlay layer.

[0093] In an alternative embodiment, the device of this embodiment further includes: a selection module, configured to select, according to the capability information of at least one overlay layer, an overlay layer that simultaneously supports at least two types of cloud layer data from at least one overlay layer as the first overlay layer, where the capability information of the overlay layer includes the data formats supported by the overlay layer.

[0094] In an alternative embodiment, the selection module is further configured to: in the case where the first overlay layer cannot be selected from at least one overlay layer, obtain at least one second overlay layer from at least one overlay layer, and each second overlay layer supports one type of cloud layer data.

[0095] Based on the above, the device of this embodiment further includes: a judgment module, configured to, each time receiving the target cloud layer data provided by the cloud server, judge whether there is a second superimposed layer corresponding to the target cloud layer data according to the type of the target cloud layer data. The decoding module 62 is further configured to, when the judgment module judges that there is a second superimposed layer corresponding to the target cloud layer data, decode the target cloud layer data to obtain fourth intermediate layer data; the first display sending module 63 is further configured to directly send and display the fourth intermediate layer data to the corresponding second superimposed layer without GPU rendering. Correspondingly, the loading module 64 is further configured to: load at least one type of local layer data, the rendering module 65 is further configured to: call the GPU to render at least one type of local layer data to obtain fifth intermediate layer data, and the second display sending module 66 is further configured to: send and display the fifth layer data to the main layer. The display control module 67 is further configured to perform hardware synthesis on the fourth intermediate layer data and the fifth intermediate layer data to obtain second data to be displayed, and display the second data to be displayed to obtain a cloud desktop.

[0096] The device provided in this embodiment can be used to execute the steps in the above method embodiment. For the detailed function implementation of each functional module, reference can be made to the detailed description in the foregoing method embodiment or system embodiment, which will not be elaborated here.

[0097] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to be able to implement the steps executable by the terminal device in the above method embodiment.

[0098] The communication component in the above embodiment is configured to facilitate communication between the device where the communication component is located and other devices in a wired or wireless manner. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0099] The display in the above embodiments includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operation.

[0100] The power supply component in the above embodiments provides power for various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device where the power supply component is located.

[0101] The audio component in the above embodiments can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), which is configured to receive external audio signals when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in a memory or transmitted via a communication component. In some embodiments, the audio component further includes a speaker for outputting audio signals.

[0102] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0103] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0104] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device that implements the functions specified in one or more of the processes Figure 1 one or more of the processes and / or blocks Figure 1 specified in one or more of the blocks or blocks.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, such that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more of the processes Figure 1 one or more of the processes and / or blocks Figure 1 specified in one or more of the blocks or blocks.

[0106] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0107] Memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

[0108] Computer-readable media includes both permanent and non-permanent, removable and non-removable media implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0109] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising said element.

[0110] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A cloud desktop system, characterized in that, it includes: a cloud server for providing the first part of the layer data required for the cloud desktop, and a terminal device for providing the second part of the layer data required for the cloud desktop and displaying the cloud desktop, the terminal device at least includes a plurality of hardware layers, and the plurality of hardware layers include a main layer and at least one overlay layer; the cloud server is configured to send the first part of the layer data to the terminal device according to the dynamic display requirements of the cloud desktop; the terminal device is configured to decode the first part of the layer data sent by the cloud server to obtain first intermediate layer data, and send the first intermediate layer data to the first overlay layer in the at least one overlay layer; and load the second part of the layer data, call the graphics processing unit (GPU) to render the second part of the layer data to obtain second intermediate layer data, and send the second intermediate layer data to the main layer; perform hardware synthesis on the first intermediate layer data and the second intermediate layer data to obtain first data to be displayed layer data, and display the first data to be displayed layer data to obtain the cloud desktop.

2. A terminal device, characterized in that, the terminal device is used to provide the second part of the layer data required for the cloud desktop and display the cloud desktop, and the terminal device includes: a computing processor, a memory, a graphics processing unit (GPU), a plurality of hardware layers, and a display controller, and the plurality of hardware layers include a main layer and at least one overlay layer; the memory is configured to store a computer program corresponding to the cloud desktop client, and the computing processor is coupled to the memory and configured to execute the computer program to: receive the first part of the layer data required for the cloud desktop sent by the cloud server according to the dynamic display requirements of the cloud desktop; decode the first part of the layer data to obtain first intermediate layer data, and send the first intermediate layer data to the first overlay layer in the at least one overlay layer; and load the second part of the layer data, call the GPU to render the second part of the layer data to obtain second intermediate layer data, and send the second intermediate layer data to the main layer; the display controller is configured to perform hardware synthesis on the first intermediate layer data in the first overlay layer and the second intermediate layer data in the main layer to obtain first data to be displayed layer data, and display the first data to be displayed layer data to obtain the cloud desktop.

3. The terminal device according to claim 2, characterized in that, the first part of the layer data includes at least two types of cloud layer data; The computing processor is specifically configured to: each time receive the target cloud layer data adapted to the dynamic display requirement sent by the cloud server, where the target cloud layer data is one of the at least two types of cloud layer data; decode the target cloud layer data to obtain first intermediate layer data, and directly send the first intermediate layer data to the first overlay layer without GPU rendering, where the first overlay layer is the overlay layer among the at least one overlay layer that supports the at least two types of cloud layer data at the same time.

4. A cloud desktop display method Characterized in that It is applied to a terminal device, and the terminal device is used to provide the second part of the layer data required for the cloud desktop and display the cloud desktop. The method includes: Receiving the first part of the layer data required for the cloud desktop sent by the cloud server according to the dynamic display requirement of the cloud desktop; Decoding the first part of the layer data to obtain first intermediate layer data, and sending the first intermediate layer data to the first overlay layer in at least one overlay layer included in the terminal device; Loading the second part of the layer data, calling the GPU to render the second part of the layer data to obtain second intermediate layer data, and sending the second intermediate layer data to the main layer included in the terminal device; Performing hardware synthesis on the first intermediate layer data and the second intermediate layer data to obtain first data to be displayed, and displaying the first data to be displayed to obtain the cloud desktop.

5. The method according to claim 4, Characterized in that Sending the first intermediate layer data to the first overlay layer in at least one overlay layer included in the terminal device includes: directly sending the first intermediate layer data to the first overlay layer without GPU rendering.

6. The method according to claim 4, Characterized in that The first part of the layer data includes at least two types of cloud layer data, and the first overlay layer is the overlay layer among the at least one overlay layer that supports the at least two types of cloud layer data at the same time; Receiving the first part of the layer data required for the cloud desktop sent by the cloud server according to the dynamic display requirement of the cloud desktop includes: each time receiving the target cloud layer data adapted to the dynamic display requirement sent by the cloud server, where the target cloud layer data is one of the at least two types of cloud layer data; Decoding the first part of the layer data to obtain first intermediate layer data includes: decoding the target cloud layer data to obtain first intermediate layer data.

7. The method according to claim 6, Characterized in that Sending the first intermediate layer data to the first overlay layer in at least one overlay layer included in the terminal device includes: Directly writing the first intermediate layer into the cache of the direct memory access (DMA) without GPU rendering, and providing the first intermediate layer data to the first overlay layer through the DMA.

8. The method according to claim 6, Characterized in that The second part of the layer data includes at least one type of local layer data; Loading the second part of the layer data and invoking the GPU to render the second part of the layer data to obtain second intermediate layer data includes: simultaneously loading the at least one type of local layer data and invoking the GPU to simultaneously render the at least one type of local layer data to obtain second intermediate layer data.

9. The method according to claim 8, wherein, the at least one type of local layer data includes first initial layer data, and the first initial layer data refers to the window menu content data required by the cloud desktop; Invoking the GPU to simultaneously render the at least one type of local layer data to obtain second intermediate layer data includes: Initializing the rendering state of the first initial layer data, encapsulating the first initial layer data and its rendering state into a first data packet, and invoking the GPU to render the first data packet to obtain third intermediate layer data; Invoking the GPU to render and synthesize the second initial layer data provided by the window manager and the third intermediate layer data to obtain the second intermediate layer data, where the second initial layer data refers to the window menu style data required by the cloud desktop.

10. The method according to claim 9, wherein, the at least two types of cloud layer data include third initial layer data and fourth initial layer data, the third initial layer data refers to the video stream data required by the cloud desktop, and the fourth initial layer data refers to the graphic and text data required by the cloud desktop; Sending the first intermediate layer data to the first overlay layer among at least one overlay layer included in the terminal device includes: According to the position information of the first intermediate layer data in the cloud desktop provided by the window manager, invoking the graphics interface server to send the first intermediate layer data to the corresponding position in the first overlay layer.

11. The method according to any one of claims 8-10, wherein, further includes: According to the capability information of the at least one overlay layer, selecting an overlay layer that simultaneously supports the at least two types of cloud layer data from the at least one overlay layer as the first overlay layer, and the capability information of the overlay layer includes the data formats supported by the overlay layer.

12. The method according to claim 11, wherein, further includes: In the case where the first overlay layer cannot be selected from the at least one overlay layer, obtaining at least one second overlay layer from the at least one overlay layer, and each second overlay layer supports one type of cloud layer data; and Each time the target cloud layer data provided by the cloud server is received, determining whether there is a second overlay layer corresponding to the target cloud layer data according to the type of the target cloud layer data; If there is a second overlay layer corresponding to the target cloud layer data, decode the target cloud layer data to obtain fourth intermediate layer data, and send the fourth intermediate layer data to the corresponding second overlay layer for display; Call the GPU to render the at least one type of local layer data to obtain fifth intermediate layer data, and send the fifth layer data to the main layer for display; Perform hardware synthesis on the fourth intermediate layer data and the fifth intermediate layer data to obtain second data to be displayed, and display the second data to be displayed to obtain the cloud desktop.

13. The method according to claim 12, wherein, further comprising: If there is no second overlay layer corresponding to the target cloud layer data, decode the target cloud layer data to obtain eighth intermediate layer data, and call the GPU to perform rendering synthesis on the eighth intermediate layer data and the at least one type of local layer data to obtain sixth intermediate layer data; Send the sixth intermediate layer data as third data to be displayed into the main layer; display the third data to be displayed to obtain the cloud desktop.

14. A computer-readable storage medium storing a computer program, wherein, when the computer program is executed by a processor, the processor is caused to be able to implement the steps in the method according to any one of claims 4-13.

Images