Display method of cloud desktop, image display method, device and storage medium

Abstract

The present invention provides a display method for a cloud desktop, an image display method, a device, and a storage medium. The method includes: determining an updated area in the screen content of the cloud desktop at a first moment. If it is determined that the area of the updated area is greater than a set area threshold, the screen content is encoded using a lossy encoding method to obtain first encoded data. The first encoded data and the position information of the updated area are sent to the client, so that the client determines first decoded data corresponding to the position information according to the first encoded data. If the content of the updated area does not undergo content updates again within a set duration after the first moment, the content in the updated area at the first moment is encoded using a lossless encoding method to obtain second encoded data; the second encoded data is sent to the client, so that the client determines second decoded data corresponding to the second encoded data. The present invention can take into account the picture display quality, image compression efficiency, and bandwidth while encoding the screen content of the cloud desktop.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a cloud desktop display method, an image display method, a device and a storage medium. Background Art

[0002] A virtual desktop, also known as a cloud desktop, is usually installed by a user on a local terminal as a cloud desktop transmission protocol client (referred to as a cloud desktop client) to communicate with the cloud desktop server in the cloud. The cloud desktop client and the cloud desktop server transmit data in a streaming manner through the remote desktop protocol. The cloud desktop server encodes the screen content of the cloud desktop and transmits it to the cloud desktop client. The cloud desktop client decodes and presents it to the user through the local display system.

[0003] When encoding the screen content of a cloud desktop, lossy encoding algorithms or lossless encoding algorithms are generally used. Among them, although the lossy encoding algorithm has a relatively high compression ratio, it is completed at the expense of image quality. The screen display quality is poor, and some information in the screen content, such as text, is prone to being blurred. Although the lossless encoding algorithm can guarantee image quality, it has low compression efficiency, occupies a large bandwidth, and has poor real-time data transmission. Summary of the invention

[0004] The embodiments of the present invention provide a cloud desktop display method, an image display method, a device and a storage medium, which can take into account both the image display quality and the bandwidth when encoding the cloud desktop screen content.

[0005] In a first aspect, an embodiment of the present invention provides a method for displaying a cloud desktop, which is applied to a server running a cloud desktop, and the method includes:

[0006] Determine the update area in the screen content of the cloud desktop at the first moment;

[0007] If it is determined that the area of ​​the update region is greater than the set area threshold, the following encoding process is performed:

[0008] Encoding the screen content in a lossy encoding manner to obtain first encoded data;

[0009] Sending the first coded data and the location information of the update area to a client, so that the client determines the first decoded data corresponding to the location information according to the first coded data;

[0010] If no content update occurs in the update area within a set time period after the first moment, encoding the content in the update area at the first moment in a lossless encoding manner to obtain second encoded data;

[0011] Send the second encoded data to the client so that the client can determine the second decoded data corresponding to the second encoded data.

[0012] In a second aspect, an embodiment of the present invention provides a display device for a cloud desktop, which is applied to a server running the cloud desktop. The device includes:

[0013] A determination module, configured to determine an updated area in the screen content of the cloud desktop at a first moment;

[0014] An execution module, configured to, if it is determined that the area of the updated area is greater than a set area threshold, perform the following encoding process:

[0015] Encode the screen content using a lossy encoding method to obtain first encoded data;

[0016] Send the first encoded data and the position information of the updated area to the client so that the client can determine the first decoded data corresponding to the position information according to the first encoded data;

[0017] If the content of the updated area does not change again within a set duration after the first moment, encode the content in the updated area at the first moment using a lossless encoding method to obtain second encoded data;

[0018] Send the second encoded data to the client so that the client can determine the second decoded data corresponding to the second encoded data.

[0019] In a third aspect, an embodiment of the present invention provides a display method for a cloud desktop, which is applied to a client corresponding to the cloud desktop. The method includes:

[0020] Receive the first encoded data and the position information of the updated area sent by the server; wherein, the first encoded data is obtained by the server encoding the screen content using a lossy encoding method when it is determined that the updated area in the screen content of the cloud desktop at a first moment is greater than a set area threshold;

[0021] Decode the first encoded data to obtain first decoded data;

[0022] Extract the decoded data corresponding to the position information from the first decoded data and display the extracted decoded data;

[0023] Receive the second encoded data sent by the server; wherein, the second encoded data is obtained by the server encoding the content in the updated area at the first moment using a lossless encoding method when it is determined that the content of the updated area does not change again within a set duration after the first moment;

[0024] The second decoded data obtained by decoding the second encoded data, and display the second decoded data.

[0025] Fourthly, an embodiment of the present invention provides a display device for a cloud desktop, which is applied to a client corresponding to the cloud desktop. The device includes:

[0026] A receiving module, configured to receive the first encoded data and the position information of the updated area sent by the server; wherein, the first encoded data is obtained by encoding the screen content of the cloud desktop at the first moment using a lossy encoding method when the updated area in the screen content is determined to be larger than a set area threshold by the server;

[0027] A decoding module, configured to decode the first encoded data to obtain the first decoded data;

[0028] A display module, configured to extract the decoded data corresponding to the position information from the first decoded data and display the extracted decoded data;

[0029] The receiving module is further configured to receive the second encoded data sent by the server; wherein, the second encoded data is obtained by encoding the content in the updated area at the first moment using a lossless encoding method when it is determined by the server that the content in the updated area has not been updated again within a set duration after the first moment;

[0030] The display module is further configured to obtain the second decoded data by decoding the second encoded data, and display the second decoded data.

[0031] Fifthly, an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a communication interface; wherein, an executable code is stored on the memory, and when the executable code is executed by the processor, the processor is caused to execute the display method of the cloud desktop as described in the first aspect or the third aspect.

[0032] Sixthly, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which an executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the display method of the cloud desktop as described in the first aspect or the third aspect.

[0033] Seventhly, an embodiment of the present invention provides an image display method, and the method includes:

[0034] Determine an image update area in a video frame at a target moment;

[0035] If it is determined that the area of the image update area is larger than a set area threshold, then perform the following encoding process:

[0036] Encode the video frame using a lossy encoding method to obtain first encoded data;

[0037] Send the first encoded data and the position information of the image update area to a display device, so that the display device determines first decoded data corresponding to the position information according to the first encoded data and displays it;

[0038] If the content of the image update area does not change again within a set duration after the target moment, encode the content in the image update area using a lossless encoding method to obtain second encoded data;

[0039] Send the second encoded data to the display device, so that the display device determines second decoded data corresponding to the second encoded data and displays it.

[0040] The display method of the cloud desktop provided by the embodiment of the present invention determines an update area in the screen content of the cloud desktop at a first moment by applying it to a server running the cloud desktop. If it is determined that the area of the update area is greater than a set area threshold, it means that there are more changed parts in the screen content. At this time, encoding the screen content using a lossy encoding method to obtain first encoded data can improve the image compression efficiency of the screen content and reduce the bandwidth occupancy. Then, by sending the first encoded data and the position information of the update area to the client, the server can enable the client to extract the decoded data corresponding to the position information from the first decoded data obtained by decoding the first encoded data and display the extracted decoded data to the user. If it is determined that the content of the update area does not change again within a set duration after the first moment, it means that the update area has not changed for a long time and may be in a relatively static situation. At this time, encoding the content in the update area at the first moment using a lossless encoding method to obtain second encoded data improves the image quality of the update area, and then sending the second encoded data to the client can enable the client to display the second decoded data with better image quality obtained by decoding the second encoded data. In summary, the present invention can take both the screen display quality and bandwidth into account when encoding the screen content of the cloud desktop. Description of the Drawings

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0042] Figure 1 It is a schematic structural diagram of a cloud desktop system provided by an embodiment of the present invention;

[0043] Figure 2 Schematic flow chart of a display method for a cloud desktop provided by an embodiment of the present invention;

[0044] Figure 3 Schematic flow chart of a display method for a cloud desktop provided by an embodiment of the present invention;

[0045] Figure 4 Schematic flow chart of a method for judging the update of the updated area provided by an embodiment of the present invention;

[0046] Figure 5 Specific example diagram of a display method for a cloud desktop provided by an embodiment of the present invention;

[0047] Figure 6 Schematic flow chart of a display method for a cloud desktop provided by an embodiment of the present invention;

[0048] Figure 7a Schematic diagram of the structure of the first layer provided by an embodiment of the present invention;

[0049] Figure 7b Schematic diagram of the structure of the second layer provided by an embodiment of the present invention;

[0050] Figure 8 Specific example diagram of a display method for a cloud desktop provided by an embodiment of the present invention;

[0051] Figure 9 Schematic diagram of the structure of a display device for a cloud desktop provided by an embodiment of the present invention;

[0052] Figure 10 Schematic diagram of the structure of an electronic device provided by this embodiment;

[0053] Figure 11 Schematic diagram of the structure of a display device for a cloud desktop provided by an embodiment of the present invention;

[0054] Figure 12 Schematic diagram of the structure of an electronic device provided by this embodiment;

[0055] Figure 13 Schematic flow chart of an image display method provided by an embodiment of the present invention. Detailed implementation manners

[0056] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. In addition, the sequence of steps in the following method embodiments is only an example and is not strictly limited.

[0057] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data that have been authorized by the user or fully authorized by all parties. And the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0058] A cloud desktop is a cloud-based desktop service. It uses Desktop as a Service (DaaS for short) to provide users with an easy-to-use, secure and efficient cloud-based desktop office system. As Figure 1 shown in the structural schematic diagram of the cloud desktop system, the cloud desktop runs on the server 10. For example, several virtual machines are built on the server 10, and different cloud desktops are set up in different virtual machines. Therefore, the cloud desktop can also be called a cloud computer and is allocated to different users for use. The user uses the client 20 installed with the client program corresponding to the cloud desktop to connect to the cloud desktop through the public network or dedicated line, so as to realize remote access to the cloud desktop.

[0059] Among them, the server 10 refers to the server device that provides the cloud desktop service. It can manage the cloud desktop and respond to the cloud desktop service requests of the client 20. Not only cloud desktops can be deployed in the server 10, but also relevant communication protocols, control services, etc. can be deployed.

[0060] The client 20 refers to a device or software functional module that can access the server 10 through a wireless network, such as a mobile phone, notebook, tablet computer, POS machine, thin host, etc., in which the client program corresponding to the cloud desktop is deployed.

[0061] It should be understood that in order to realize the remote access of the client 20 to the cloud desktop, in actual applications, generally, the server 10 will obtain the screen content of the cloud desktop, encode it, and send the encoded data to the client 20. After the client 20 receives the encoded data and decodes it, the decoded content can be presented to the user through the local display system.

[0062] However, when encoding the screen content of a cloud desktop, lossy encoding algorithms or lossless encoding algorithms are generally used currently. Among them, lossy encoding algorithms such as JPEG and H264 have a relatively high compression ratio, but they are achieved at the cost of sacrificing image quality, and the screen display quality is poor. While lossless encoding algorithms such as RLE and LZ can ensure the quality of the image, but the compression efficiency is low and the bandwidth occupancy is large. It can be seen that the current screen content encoding strategy for cloud desktops cannot balance the screen display quality and bandwidth.

[0063] In view of this, the embodiments of the present invention provide a display method for a cloud desktop, which realizes the balance of picture quality and network bandwidth.

[0064] The schematic flow diagram thereof can be referred to Figure 2 , this method is applied to the server running the cloud desktop, such as Figure 2 shown, this method includes:

[0065] 201. Determine the updated area in the screen content of the cloud desktop at the first moment.

[0066] 202. If it is determined that the area of the updated area is greater than the set area threshold, then use the lossy encoding method to encode the screen content to obtain the first encoded data, and send the first encoded data and the position information of the updated area to the client, so that the client can determine the first decoded data corresponding to the position information according to the first encoded data.

[0067] 203. If the content of the updated area does not change again within the set duration after the first moment, then use the lossless encoding method to encode the content in the updated area at the first moment to obtain the second encoded data, and send the second encoded data to the client, so that the client can determine the second decoded data corresponding to the second encoded data.

[0068] In practical applications, the server calls an application program interface (Application Program Interface, abbreviated as API) provided by the operating system through the CPU. Through this API, the updated area in the screen content of the cloud desktop at the first moment can be determined. Specifically, this API will send a notification when it discovers that the screen content of the cloud desktop has been updated. At this time, the server can determine the updated area in the screen content of the cloud desktop at the first moment based on this notification. The first moment refers to the moment when an updated area is determined. In fact, after the cloud desktop is started and run, it is possible to determine in real time whether there is an updated area at different moments. Simply put, if the current frame of screen content changes compared with the previous frame of screen content, then the changed area is the updated area determined at the current moment.

[0069] It should be noted that the updated area in the screen content may be one or more. In specific implementation, if there is only one updated area in the current screen content, directly compare the area of this updated area with the set area threshold. If the area of this updated area is greater than the set area threshold, mark this updated area as a lossy area and encode the current screen content using a lossy encoding method (such as video encoding). On the contrary, if the area of this updated area is less than the set area threshold, encode this updated area using a lossless encoding method, and this lossless encoding method includes the Run Length Encoding (RLE); Lempel Zip encoding, abbreviated as LZ; Zstandard, abbreviated as ZSTD, etc.

[0070] If there are multiple updated areas in the current screen content, for example, two updated areas, calculate the total area of the two updated areas and compare this total area with the set area threshold. If this total area is greater than the set area threshold, mark both updated areas as lossy areas and encode the current screen content using a lossy encoding method. Here, it should be noted that when the total area of multiple updated areas is greater than the set area threshold, the encoding process of steps 203 - 206 needs to be executed for each updated area respectively. On the contrary, if this total area is less than the set area threshold, encode each updated area using a lossless encoding method respectively.

[0071] Optionally, the area threshold can be determined according to the scene characteristics corresponding to the screen content. For example, in a scene with high requirements for image quality (such as in the scene of writing a document, where the client user needs particularly clear image quality), the area threshold can be set larger, while in a scene with high requirements for real-time performance (such as in video scenes like watching a game or a movie, where the client user is more sensitive to real-time performance and bandwidth), the area threshold can be set smaller. As an example, the area threshold can be set to a size of 256×256, where the two 256 are respectively the number of pixel points of the length and width of the image.

[0072] Continuing from the above, taking the case where the area of the updated region is greater than the set area threshold, using a lossy coding method, and this lossy coding method is video coding as an example for subsequent description. Video coding refers to: using a video encoder to perform video compression coding on the complete screen content to obtain a coding result, that is, obtaining the first coded data. Then, the first coded data and the position information of the updated region are sent to the client. After receiving the first coded data, the client will determine the coding type of the first coded data, that is, determine whether the coding method of the first coded data is a lossy coding method or a lossless coding method. After determining that the coding method of the first coded data is a lossy coding method (i.e., encoded using a video encoder), the client will use the corresponding video decoder of the video encoder to decode it to obtain the complete screen content (i.e., the first decoded data), and combine the received position information of the updated region to extract the decoded data corresponding to the updated region from the first decoded data for rendering and display on the client. Among them, the video encoder includes but is not limited to: encoders such as H.264 and H.265. Among them, in the data sent from the server to the client, in addition to including the above-mentioned first coded data and the position information of the updated region, it will also include information indicating the coding type corresponding to the first coded data, based on which the client determines what decoding method should be used to decode the first coded data.

[0073] It should be noted that for the first decoded data, the client only extracts the partial decoded data corresponding to the updated region from it for rendering and display, that is, "incremental rendering": for example, before the client decodes the first decoded data, the displayed picture is picture X. Locate the above-mentioned updated region in picture X, keep the content of the non-updated region in picture X unchanged, and only render and display the first decoded data corresponding to the updated region within this updated region.

[0074] In addition, in the embodiment of the present invention, for the updated region whose area is greater than the set area threshold determined at the first moment, on the one hand, perform lossy coding on the current complete screen content including this updated region, send the obtained first coded data and the position information of the updated region to the client for decoding and display. On the other hand, observe whether the content of this updated region continues to be updated in the next period of time. If it is updated, no other processing is performed on the content of this updated region. If it is not updated, perform lossless coding on the content in this updated region.

[0075] Specifically, it is determined that the updated area in the screen content of the cloud desktop occurs at the first moment. If there is no further content update in the updated area within a set duration (such as 50 ms, which can be determined according to the actual situation) after the first moment, it indicates that the updated area may be in a relatively static situation at this time, that is, the content in the updated area has not changed during this period. At this time, a lossless encoding method can be used to encode the content in the updated area at the first moment to obtain the second encoded data, so as to ensure the clarity of the content in the updated area, improve the picture quality, and enable the picture to achieve the effect of changing from blurred to clear. It can also be understood as gradually sharpening the picture from a lossy picture quality to a lossless picture quality.

[0076] After that, the server can send the second encoded data to the client. After receiving the second encoded data, the client will judge the encoding type of the second encoded data. After determining that the encoding method of the second encoded data is a lossless encoding method, the client will use the corresponding lossless decoding method for the lossless encoding method to decode it, obtain a clearer updated area, and display it on the client.

[0077] Based on the cloud desktop display method provided in the above embodiments of the present invention, for the updated area in the screen content of the cloud desktop at the first moment, if it is determined that the area of the updated area is greater than the set area threshold, it indicates that there are more changed parts in the screen content. At this time, using a lossy encoding method to encode the screen content to obtain the first encoded data can improve the image compression efficiency of the screen content and reduce the bandwidth occupancy. If it is determined that there is no further content update in the updated area within the set duration after the first moment, it indicates that the updated area has not changed for a long time. At this time, using a lossless encoding method to encode the content in the updated area at the first moment to obtain the second encoded data improves the picture quality of the updated area, and then sending the second encoded data to the client can enable the client to display the second decoded data with better picture quality obtained by decoding the second encoded data. In this way, the requirements for both bandwidth and picture quality can be taken into account.

[0078] Figure 3 It is a schematic flowchart of a cloud desktop display method provided by an embodiment of the present invention, as Figure 3 shown, the method includes:

[0079] 301. Determine the updated area in the screen content of the cloud desktop at the first moment.

[0080] 302. Determine whether the area of the updated area is greater than the set area threshold. If so, execute steps 303-305; otherwise, execute step 306.

[0081] 303. Encoding the screen content using a lossy encoding method to obtain first encoded data, and sending the first encoded data and the position information of the updated area to the client, so that the client determines the first decoded data corresponding to the position information according to the first encoded data.

[0082] 304. Determine whether there is any content update in the updated area between the first moment and the second moment after a set time interval. If so, execute step 305; otherwise, end.

[0083] 305. Encoding the content in the updated area at the first moment using a lossless encoding method to obtain second encoded data, and sending the second encoded data to the client, so that the client displays the second decoded data obtained by decoding the second encoded data.

[0084] 306. Encoding the updated area using a lossless encoding method to obtain second encoded data, so that the client displays the second decoded data obtained by decoding the second encoded data.

[0085] In practical applications, after the server determines the updated area in the screen content of the cloud desktop at the first moment, if it is determined that the area of the updated area is greater than the set area threshold, the encoding process can refer to the above embodiments and will not be elaborated here. If it is determined that the area of the updated area is not greater than the set area threshold, it means that there are not many changed parts in the screen content. At this time, encoding the updated area using a lossless encoding method can not only ensure the screen display quality, but also does not require lossless encoding of the entire screen content. Only the updated area with a small area needs to be encoded, and the amount of data transmitted and the occupied bandwidth are small. In summary, the present invention can adopt a flexible encoding method according to the area size of the updated area, and can take into account the screen display quality, image compression efficiency and bandwidth at the same time.

[0086] Figure 4 It is a schematic flowchart of a method for judging content update in an updated area provided by an embodiment of the present invention. As Figure 4 shown, the method includes:

[0087] 401. Sending the first encoded data and the position information of the updated area to the client.

[0088] 402. Generating record information corresponding to the updated area, where the record information includes the first moment, the position information of the updated area, and a set lossy encoding flag.

[0089] Among them, the lossy encoding flag is a flag indicating the adopted lossy encoding method.

[0090] 403. Obtaining the record information when the current timing of the timer reaches the second moment.

[0091] 404. Determine whether the time interval between the first moment and the second moment in the recorded information is greater than a set duration.

[0092] 405. If it is greater than the set duration, determine that no content update has occurred in the updated area between the first moment and the second moment.

[0093] In practical applications, when the cloud desktop is started, a set timer (hereinafter referred to as the timer) can be started. The timer has a preset timing duration for determining whether the content in the updated area has changed. Specifically, when the timing duration is assumed to be 50 ms, a judgment is made every 50 ms to determine whether there is an updated area marked as lossy in the screen content. Here, marking lossy means that the updated area meets the condition of being greater than the set area threshold mentioned above. If at the first moment t 1 It is determined that there is an updated area A in the screen content X of the cloud desktop, and the area of the updated area A is greater than the set area threshold, then record information corresponding to the updated area is generated, such as "t 1 Sending moment, location information of the updated area A, lossy", and the screen content X is video compression encoded, and the first encoded data obtained after video compression encoding and the location information of the updated area A are sent to the client.

[0094] After that, at the second moment t when the current timing of the timer arrives 2 , the recorded information is obtained, and it is judged whether the time interval between the first moment and the second moment in the recorded information is greater than the set duration (such as 100 ms). If it is greater than the set duration, it is determined that no content update has occurred in the updated area A between the first moment and the second moment, and the updated area A is encoded using a lossless encoding method, and the obtained second encoded data is sent to the client. It should be noted that after the second encoded data is sent to the client, the recorded information will be deleted. Thus, for the screen content X containing the updated area A, the client receives the complete screen content encoded using a lossy encoding method and the location information of the updated area A at the first moment t 1 , and receives the encoding result of the content of the updated area A encoded using a lossless encoding method at the second moment t 2 . Among them, the set duration can be determined according to the network quality between the server and the client. Specifically, if the network quality is good (such as more available bandwidth), the set duration can be set slightly shorter, which is convenient for lossless encoding. On the contrary, if the network quality is poor, the set duration can be set slightly longer.

[0095] Furthermore, if at the third moment t between the first moment t 1 and the second moment t 2 the third moment t 3If it is determined that the content in the update area A has been updated, then update the first moment t in the record information 1 to the third moment t 3 ; at the second moment t when the current timing of the timer arrives 2 , after obtaining the record information, if it is determined that the third moment t in the record information 3 and the second moment t 2 have a time interval not greater than the set duration, then reset the timer.

[0096] Specifically, assume that at the first moment t 1 the update area A in the screen content X of the cloud desktop is determined, and the area of the update area A is greater than the set area threshold, then the screen content X is encoded using a lossy coding method, and at this time a record information is generated. Here, assume that the first moment is the moment when the timer starts to start a set duration of timing. After that, at the third moment t 1 after the first moment t 3 (assuming that this timing has not ended yet), if it is determined that the content in the update area A has been updated, then continue to encode the screen content X using a lossy coding method. At this time, update the first moment t in the record information 1 to the third moment t 3 to obtain the updated record information. After that, at the second moment t when the current timing of the timer arrives 2 (assuming that this timing ends), determine whether the time interval between the third moment t 3 in the record information and the second moment t 2 is greater than the set duration. If it is greater than the set duration, then it is determined that the content in the update area A has not been updated again between the third moment t 3 and the second moment t 2 , and the content in the update area A is encoded using a lossless coding method, and the obtained second coding data is sent to the client. If it is determined that the time interval between the third moment t 3 in the record information and the second moment t 2 is not greater than the set duration, then reset the timer.

[0097] For the sake of easy understanding, simply put, in the screen content X, if the content in the update area A is static or changes infrequently, then at two moments separated by the timing duration, the content in the update area A has not changed. At this time, using a lossless coding method to encode the content in the update area A can ensure the picture quality and achieve picture quality enhancement. Since only the update area A is encoded at this time and the complete screen content is not encoded, the amount of data transmitted is small and the bandwidth occupied is small. If the change frequency of the update area A is high, then no picture quality enhancement processing is performed at this time, and a lossy coding method is used to encode the corresponding complete screen content, which will not occupy too much bandwidth and has good real-time performance.

[0098] For ease of understanding, a complete example is provided below to specifically illustrate the application of the method of the present invention to the server running a cloud desktop: Figure 5 After the screen content of the cloud desktop is updated, the server determines the updated area in the screen content of the cloud desktop at the first moment, and determines whether the area of the updated area is greater than a set area threshold. If the area of the updated area is greater than the set area threshold, the screen content is encoded using a lossy encoding method to obtain first encoded data, and the position information of the updated area is filled based on the first encoded data, and the first encoded data and the position information of the updated area are sent to the client.

[0099] If the area of the updated area is not greater than the set area threshold, the updated area is encoded using a lossless encoding method to obtain second encoded data, and the second encoded data is sent to the client.

[0100] During this process, the timer makes a judgment operation based on a set duration to determine whether there is an updated area with an area greater than the set area threshold, that is, a lossy area. If not, the work ends; if so, it continues to determine whether the time interval between the first moment and the second moment is greater than the set duration. If so, it is determined that no content update has occurred in the updated area between the first moment and the second moment, and the updated area is encoded using a lossless encoding method, and the obtained second encoded data is sent to the client. If not, the work ends.

[0101] As shown in the flowchart of a display method for a cloud desktop provided by an embodiment of the present invention, this method is applied to the client corresponding to the cloud desktop, such as

[0102] Figure 6 shown, the method includes: Figure 6 shown, the method includes:

[0103] 601. Receive the first encoded data and the position information of the updated area sent by the server; wherein, the first encoded data is obtained by encoding the screen content using a lossy encoding method when the server determines that the updated area in the screen content of the cloud desktop at the first moment is greater than the set area threshold.

[0104] 602. Decode the first encoded data to obtain first decoded data.

[0105] 603. Extract the decoded data corresponding to the position information from the first decoded data and display the extracted decoded data.

[0106] 604. Receive the second encoded data sent by the server; wherein, the second encoded data is obtained by encoding the content in the updated area at the first moment using a lossless encoding method when the server determines that no content update has occurred in the updated area within the set duration after the first moment.

[0107] 605. Decode the second encoded data to obtain the second decoded data, and display the second decoded data.

[0108] For the above execution process, reference can be made to the relevant descriptions in the foregoing other embodiments, which will not be elaborated herein.

[0109] In the embodiment of the present invention, extracting the decoded data corresponding to the position information from the first decoded data and displaying the extracted decoded data includes: rendering the extracted decoded data on the first layer to display the first layer. Based on this, displaying the second decoded data includes: rendering the second decoded data on the second layer to display the second layer on the first layer.

[0110] In practical applications, the client will determine whether it uses a lossy encoding method or a lossless encoding method according to the encoding types of the first encoded data and the second decoded data received from the server. That is to say, the encoding type information corresponding to the first encoded data is also sent to the client together with the first encoded data and the position information of the updated area. Similarly, the encoding type information corresponding to the second encoded data is also sent to the client together with the second encoded data.

[0111] For the first encoded data encoded by a lossy encoding method (such as video encoding): use a video decoder corresponding to the video encoder used in the video encoding to decode to obtain the complete screen content, and according to the position information of the updated area, locate the content of the updated area from the complete screen content, and perform "incremental rendering" on it, that is, keep the content of the non-updated area in the previously displayed content unchanged, and only incrementally render the content in the updated area.

[0112] For the second encoded data encoded by a lossless encoding method: use a lossless decoding method corresponding to this lossless encoding method to decode the updated area to obtain the content of the updated area, and then perform rendering.

[0113] After the above processing process, for the updated area, a "sharp progressive" display effect can be obtained, that is: at the beginning, a lossy encoding method is used for encoding, and then decoded, and the content displayed by the first decoded data is blurred. Then a lossless encoding method is used for encoding, and then decoded, and the content displayed by the second decoded data is clear, so it is an effect from blurred to clear.

[0114] It should be noted that for the two renderings and displays of the updated area: the decoded data extracted from the first decoded data is rendered and displayed on the first layer, and when displaying the second decoded data, a new layer, that is, the second layer, will be generated again. The second layer will cover the first layer, and the second decoded data is rendered and displayed on the second layer. For details, please refer to Figure 7a andFigure 7b , Figure 7a is a schematic structural diagram of the first layer. In Figure 7a , the updated area A is the decoded data extracted from the first decoded data, which is rendered on the first layer. Figure 7b is a schematic structural diagram of the second layer. In Figure 7b , the updated area A * is the second decoded data, which is rendered on the second layer, and the second layer covers the first layer.

[0115] For ease of understanding, the following provides a complete example in conjunction with Figure 8 to specifically illustrate the application of the method of the present invention to the client corresponding to the cloud desktop:

[0116] Receive the first encoded data, the second encoded data, and the position information of the updated area sent by the server, and determine whether the first encoded data and the second encoded data are lossy encoded data according to the encoding type.

[0117] If it is lossy encoded data, then decode it using the decoding method corresponding to the lossy encoding method, and extract the decoded data according to the position information of the updated area. That is, if it is determined that the first encoded data is lossy encoded data, then decode it using the decoding method corresponding to the lossy encoding method, and extract the decoded data according to the position information of the updated area, and then copy the updated area to the RGB layer buffer of the first layer and render and display it in the first layer.

[0118] If it is not lossy encoded data, then decode it using the decoding method corresponding to the lossless encoding method, and copy the decoded second decoded data to the RGB layer buffer of the second layer and render and display it in the second layer. It should be noted that the second layer covers the first layer.

[0119] The following will describe in detail the display device of the cloud desktop of one or more embodiments of the present invention. Those skilled in the art can understand that these devices can all be configured by using commercially available hardware components through the steps taught by this solution.

[0120] Figure 9 is a schematic structural diagram of a display device of a cloud desktop provided by an embodiment of the present invention. This device is applied to the server running the cloud desktop. As Figure 9 shown, this device includes: a determination module 11 and an execution module 12.

[0121] The determination module 11 is used to determine the updated area in the screen content of the cloud desktop at the first moment.

[0122] The execution module 12 is configured to, when determining that the area of the update region is greater than a set area threshold, perform the following encoding process: encode the screen content in a lossy encoding manner to obtain first encoded data; send the first encoded data and the position information of the update region to the client, so that the client determines first decoded data corresponding to the position information according to the first encoded data; if the content of the update region does not change again within a set duration after the first moment, encode the content in the update region at the first moment in a lossless encoding manner to obtain second encoded data; send the second encoded data to the client, so that the client determines second decoded data corresponding to the second encoded data.

[0123] Optionally, the device further includes: an encoding module, configured to, if determining that the area of the update region is less than or equal to the set area threshold, encode the update region in a lossless encoding manner to obtain second encoded data.

[0124] Optionally, the screen content includes multiple update regions; the execution module 12 is specifically configured to: if determining that the total area of the multiple update regions is greater than the set area threshold, perform the encoding process for each update region.

[0125] Optionally, the device further includes: an update processing module.

[0126] The update processing module is configured to generate record information corresponding to the update region, where the record information includes the first moment, the position information of the update region, and a set lossy encoding flag; when the current timing of the timer reaches the second moment, obtain the record information; determine whether the time interval between the first moment and the second moment in the record information is greater than the set duration; if it is greater than the set duration, determine that the content of the update region does not change again between the first moment and the second moment.

[0127] Optionally, the update processing module is further configured to: after sending the second encoded data to the client, delete the record information.

[0128] Optionally, the update processing module is further configured to: if at a third moment between the first moment and the second moment, determine that the content of the update region changes, update the first moment in the record information to the third moment; if determining that the time interval between the third moment and the second moment in the record information is not greater than the set duration, reset the timer.

[0129] Optionally, the device further includes: a duration determination module, configured to determine the set duration according to the network quality between the server and the client.

[0130] Optionally, the device further includes: a threshold determination module, configured to determine the area threshold according to the scene characteristics corresponding to the screen content.

[0131] Figure 9The device shown above can execute the steps performed by the server in the foregoing embodiments. For the detailed execution process and technical effects, refer to the descriptions in the foregoing embodiments and will not be elaborated herein.

[0132] In a possible design, the structure of the display device of the above Figure 9 shown cloud desktop can be implemented as an electronic device. As Figure 10 shown, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. Among them, executable code is stored on the memory 22. When the executable code is executed by the processor 21, the processor 21 can at least implement the cloud desktop display method performed by the server provided in the foregoing embodiments. The electronic device may be a virtual machine in the cloud.

[0133] Figure 11 FIG. is a schematic structural diagram of a display device for a cloud desktop provided by an embodiment of the present invention. The device is applied to a client corresponding to the cloud desktop. As Figure 11 shown, the device includes: a receiving module 31, a decoding module 32, and a display module 33.

[0134] The receiving module 31 is configured to receive the first encoded data and the position information of the updated area sent by the server; wherein, the first encoded data is obtained by the server encoding the screen content in a lossy encoding manner when it is determined that the updated area in the screen content of the cloud desktop at the first moment is larger than the set area threshold.

[0135] The decoding module 32 is configured to decode the first encoded data to obtain first decoded data.

[0136] The display module 33 is configured to extract the decoded data corresponding to the position information from the first decoded data and display the extracted decoded data.

[0137] The receiving module 31 is further configured to receive the second encoded data sent by the server; wherein, the second encoded data is obtained by the server encoding the content in the updated area at the first moment in a lossless encoding manner when it is determined that no content update has occurred in the updated area within a set duration after the first moment.

[0138] The display module 33 is further configured to decode the second decoded data obtained from the second encoded data and display the second decoded data.

[0139] Optionally, the display module 33 is specifically configured to: render the extracted decoded data on a first layer to display the first layer; render the second decoded data on a second layer to display the second layer on the first layer.

[0140] Figure 11The device shown can perform the steps executed by the client in the foregoing embodiments. For the detailed execution process and technical effects, refer to the descriptions in the foregoing embodiments, which will not be elaborated herein.

[0141] In a possible design, the structure of the display device of the cloud desktop shown above Figure 11 can be implemented as an electronic device. As Figure 12 shown, the electronic device may include: a processor 41, a memory 42, and a communication interface 43. Among them, executable code is stored on the memory 42. When the executable code is executed by the processor 41, the processor 41 can at least implement the cloud desktop display method executed by the client provided in the foregoing embodiments. The electronic device may be a user terminal.

[0142] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by the processor of the electronic device, the processor can at least implement the cloud desktop display method provided in the foregoing embodiments.

[0143] Figure 13 It is a schematic flowchart of an image display method provided by an embodiment of the present invention. As Figure 13 shown, the method includes:

[0144] 1301. Determine the image update area in the video frame at the target time.

[0145] 1302. If it is determined that the area of the image update area is greater than the set area threshold, encode the video frame using a lossy encoding method to obtain first encoded data, and send the first encoded data and the position information of the image update area to the display device, so that the display device determines the first decoded data corresponding to the position information according to the first encoded data and displays it.

[0146] 1303. If the content of the image update area does not change again within the set duration after the target time, encode the content in the image update area using a lossless encoding method to obtain second encoded data, and send the second encoded data to the display device, so that the display device determines the second decoded data corresponding to the second encoded data and displays it.

[0147] It should be understood that in practical applications, this image display method can be applied to video playback scenarios, such as live broadcast scenarios, video-on-demand scenarios, etc. Taking the live broadcast scenario as an example, the first thing to determine is the image update area in the live broadcast frame corresponding to the target time, and then perform subsequent encoding, decoding, and display operations based on this image update area. For the specific execution process of the subsequent steps, reference can be made to the relevant descriptions in the foregoing other embodiments, which will not be elaborated herein.

[0148] The device embodiments described above are merely illustrative, where the units described as separate components may or may not be physically separated. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0149] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of adding a necessary general hardware platform, and of course, it can also be implemented by a combination of hardware and software. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a computer product. The present invention can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments or equivalently replace some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A display method for a cloud desktop, characterized in that, applied to a server running a cloud desktop, the method comprising: determining an updated area in the screen content of the cloud desktop at a first moment; if it is determined that the area of the updated area is greater than a set area threshold, then perform the following encoding process: encoding the screen content by using a lossy encoding method to obtain first encoded data; sending the first encoded data and the position information of the updated area to a client, so that the client determines first decoded data corresponding to the position information according to the first encoded data; if the content of the updated area does not undergo content update again within a set duration after the first moment, then encoding the content in the updated area at the first moment by using a lossless encoding method to obtain second encoded data; sending the second encoded data to the client, so that the client determines second decoded data corresponding to the second encoded data.

2. The method according to claim 1, characterized in that, the method further comprises: if it is determined that the area of the updated area is less than or equal to the set area threshold, then encoding the updated area by using a lossless encoding method to obtain the second encoded data.

3. The method according to claim 1, characterized in that, the screen content includes a plurality of updated areas; the if it is determined that the area of the updated area is greater than the set area threshold, then perform the following encoding process, comprising: if it is determined that the total area of the plurality of updated areas is greater than the set area threshold, then perform the encoding process for each updated area.

4. The method according to claim 1, characterized in that, after sending the first encoded data and the position information of the updated area to the client, further comprising: generating record information corresponding to the updated area, where the record information includes the first moment, the position information of the updated area, and a set lossy encoding identifier; acquiring the record information when the current timing of a timer reaches a second moment; determining whether the time interval between the first moment in the record information and the second moment is greater than the set duration; if it is greater than the set duration, then determining that the content of the updated area does not undergo content update again between the first moment and the second moment.

5. The method according to claim 4, characterized in that, the method further comprises: after sending the second encoded data to the client, deleting the record information.

6. The method according to claim 4, characterized in that, the method further comprises: if at a third moment between the first moment and the second moment, it is determined that the content of the updated area undergoes content update, then updating the first moment in the record information to the third moment; after acquiring the record information when the current timing of the timer reaches the second moment, further comprising: if it is determined that the time interval between the third moment in the record information and the second moment is not greater than the set duration, then resetting the timer.

7. The method according to any one of claims 1 to 6, characterized in that, the method further comprises: Determine the set duration according to the network quality between the server and the client.

8. The method according to any one of claims 1 to 6, wherein, the method further includes: Determine the area threshold according to the scene characteristics corresponding to the screen content.

9. A display method for a cloud desktop, wherein, applied to the client corresponding to the cloud desktop, the method includes: Receiving first encoded data and location information of an updated area sent by the server; wherein, the first encoded data is obtained by the server encoding the screen content of the cloud desktop at a first moment using a lossy encoding method when the updated area in the screen content is greater than a set area threshold; Decoding the first encoded data to obtain first decoded data; Extracting the decoded data corresponding to the location information from the first decoded data and displaying the extracted decoded data; Receiving second encoded data sent by the server; wherein, the second encoded data is obtained by the server encoding the content in the updated area at the first moment using a lossless encoding method when the updated area does not undergo content update within a set duration after the first moment; Decoding the second encoded data to obtain second decoded data and displaying the second decoded data.

10. The method according to claim 9, wherein, the displaying the extracted decoded data includes: Rendering the extracted decoded data on a first layer to display the first layer; the displaying the second decoded data includes: Rendering the second decoded data on a second layer to display the second layer on the first layer.

11. An electronic device, wherein, includes: A memory, a processor, and a communication interface; wherein, an executable code is stored on the memory, and when the executable code is executed by the processor, the processor executes the display method of the cloud desktop according to any one of claims 1 to 8, or executes the display method of the cloud desktop according to claim 9 or 10.

12. A non-transitory machine-readable storage medium, wherein, an executable code is stored on the non-transitory machine-readable storage medium, and when the executable code is executed by a processor of an electronic device, the processor executes the display method of the cloud desktop according to any one of claims 1 to 8, or executes the display method of the cloud desktop according to claim 9 or 10.

13. An image display method, wherein, includes: Determine an image update area in a video frame at a target moment; If it is determined that the area of the image update area is greater than a set area threshold, then perform the following encoding process: Encoding the video frame using a lossy encoding method to obtain first encoded data; Sending the first encoded data and the location information of the image update area to a display device, so that the display device determines first decoded data corresponding to the location information according to the first encoded data and displays it; If there is no further content update in the image update area within the set duration after the target moment, the content in the image update area is encoded using a lossless encoding method to obtain second encoded data; The second encoded data is sent to the display device so that the display device determines second decoded data corresponding to the second encoded data and displays it.

Images