Cloud desktop coding method, reconstruction method and display system

Abstract

The invention discloses a cloud desktop coding method, a reconstruction method and a display system. The coding method comprises the following steps: detecting a to-be-enhanced region contained in a first format cloud desktop; encoding the second-format cloud desktop to obtain first encoding information and a second-format reconstructed cloud desktop; obtaining a chromaticity residual error between the first format cloud desktop and the second format reconstruction cloud desktop in the to-be-enhanced area; performing coding according to the chroma residual error of the to-be-enhanced region to obtain second coding information; and sending the first coding information and the second coding information to the client. Therefore, the client decodes to obtain the reconstructed chrominance residual error, and the reconstructed chrominance residual error is superposed on the chrominance component of the to-be-enhanced area in the reconstructed cloud desktop in the second format obtained by decoding so as to improve the display effect and the visual effect. Due to the fact that only the to-be-enhanced area of the cloud desktop is selected for chromaticity residual error coding transmission, and the residual error value is a small numerical value, the required transmission code rate is low, and the influence on the network bandwidth is small.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a cloud desktop encoding method, reconstruction method and display system. Background Art

[0002] Cloud desktop is a cloud computing method that places computing and storage in the cloud through a transmission protocol to provide virtual desktop services. Users can access the cloud desktop for work anytime and anywhere. The cloud uses a video / image encoding transmission solution to transmit cloud desktop data to users. Since the human eye is less sensitive to chromaticity changes, in order to reduce the amount of video encoding data, the cloud desktop is usually chromaticity downsampled to obtain YUV4:2:0 data for encoding and transmission.

[0003] However, the cloud desktop will contain content generated by computer rendering (such as text, icons, etc.). Since this computer-generated content has the characteristics of high contrast and sudden color changes, after chroma downsampling, the encoding and transmission to the user side will cause relatively serious color distortion, color cast and other problems, thus affecting the user experience. Summary of the invention

[0004] The purpose of this application is to propose a cloud desktop encoding method, reconstruction method, display system, electronic device and storage medium to address the deficiencies of the above-mentioned prior art, and this purpose is achieved through the following technical solutions.

[0005] The first aspect of the present application proposes a cloud desktop encoding method, the method comprising:

[0006] Detecting the area to be enhanced contained in the first format cloud desktop;

[0007] Encode the second format cloud desktop to obtain first encoding information and reconstruct the second format cloud desktop; wherein the second format cloud desktop is obtained by chroma downsampling the first format cloud desktop;

[0008] Obtaining a chroma residual in the to-be-enhanced area between the first-format cloud desktop and the second-format reconstructed cloud desktop;

[0009] Encoding according to the chrominance residual of the area to be enhanced to obtain second encoding information;

[0010] The first encoding information and the second encoding information are sent to a client.

[0011] A second aspect of the present application provides a cloud desktop reconstruction method, the method comprising:

[0012] Receive first encoding information and second encoding information; wherein the first encoding information is obtained by encoding the second format cloud desktop by the cloud end, and the second encoding information is obtained by encoding the chroma residual in the to-be-enhanced area between the first format cloud desktop and the second format reconstructed cloud desktop by the cloud end;

[0013] Decoding the first coded information to obtain a second format to reconstruct the cloud desktop;

[0014] Decoding the second coded information to obtain a reconstructed chroma residual of the area to be enhanced;

[0015] The reconstructed chroma residual is superimposed on the chroma component of the area to be enhanced in the second format reconstructed cloud desktop to obtain the first format reconstructed cloud desktop.

[0016] The third aspect of the present application proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect and / or the second aspect above.

[0017] The fourth aspect of the present application proposes a cloud desktop display system, which includes: a cloud and a client; wherein the cloud includes an electronic device for executing the above-mentioned cloud desktop encoding method; the client includes an electronic device for executing the above-mentioned cloud desktop reconstruction method.

[0018] The fifth aspect of the present application proposes a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the steps of the method described in the first aspect and / or the second aspect above.

[0019] Based on the above-mentioned cloud desktop encoding method, reconstruction method and display system, the present application has at least the following beneficial effects or advantages:

[0020] The cloud detects the area to be enhanced of the first format cloud desktop to adaptively select the area to be enhanced, and encodes the second format cloud desktop to obtain the first encoding information and the second format reconstructed cloud desktop. The second format cloud desktop is obtained by chroma downsampling the first format cloud desktop, and then obtains the chroma residual between the first format cloud desktop and the second format reconstructed cloud desktop in the area to be enhanced, and encodes the second encoding information according to the chroma residual of the area to be enhanced, and sends the first encoding information and the second encoding information to the client. Thus, the client decodes to obtain the reconstructed chroma residual, and superimposes the reconstructed chroma residual on the chroma component of the area to be enhanced in the second format reconstructed cloud desktop obtained by decoding, so as to improve the display effect and visual effect of the area to be enhanced, and improve the user experience. The area to be enhanced can be a content area on the cloud desktop for displaying text, icons, etc. Since the present application only selects the area to be enhanced of the cloud desktop for chroma residual encoding and transmission, and the residual value itself is also a relatively small value, the transmission bit rate required for the encoding code stream is relatively low, and the impact on the network transmission bandwidth is relatively small.

[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 The present invention is a flow chart of an embodiment of a cloud desktop encoding method according to an exemplary embodiment;

[0024] Figure 2 The figure is a schematic diagram of a region to be enhanced included in a cloud desktop according to an exemplary embodiment;

[0025] Figure 3 The present invention is a flow chart of an embodiment of a cloud desktop reconstruction method according to an exemplary embodiment;

[0026] Figure 4 The present invention is a flowchart showing an overall implementation of cloud desktop transmission according to an exemplary embodiment;

[0027] Figure 5 is a schematic diagram of a hardware structure of an electronic device according to an exemplary embodiment;

[0028] Figure 6The figure is a schematic diagram of the structure of a storage medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0030] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0031] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0032] In traditional video / image encoding, the original RGB domain of the image needs to be converted to the YUV domain. Considering that the human eye is less sensitive to chromaticity changes, in order to reduce the amount of encoded data, the encoder often performs chromaticity downsampling on the YUV4:4:4 format data to obtain YUV4:2:0 format data for encoding and transmission. This can maintain good image quality for encoding and transmitting images / videos captured by image sensors in YUV4:2:0 format. However, in the cloud desktop transmission scenario, the cloud desktop may contain both image sensor captured content and computer generated content (such as text, icons). However, for computer-generated content areas with high contrast and color mutations, chromaticity downsampling will cause serious color distortion problems, thereby affecting the user experience.

[0033] In view of this, the present application proposes a solution for realizing cloud desktop encoding and reconstruction. On the basis of encoding and decoding using the existing codec, the cloud detects the area to be enhanced of the first format cloud desktop, adaptively selects the area to be enhanced, and encodes the second format cloud desktop to obtain the first encoding information and the second format reconstructed cloud desktop. The second format cloud desktop is obtained by chroma downsampling the first format cloud desktop, and then the chroma residual between the first format cloud desktop and the second format reconstructed cloud desktop in the area to be enhanced is obtained, and the second encoding information is encoded according to the chroma residual of the area to be enhanced, and the first encoding information and the second encoding information are sent to the client. Thus, the client decodes to obtain the reconstructed chroma residual, and superimposes the reconstructed chroma residual on the chroma component of the area to be enhanced in the second format reconstructed cloud desktop obtained by decoding, so as to improve the display effect and visual effect of the area to be enhanced, and improve the user experience. The area to be enhanced can be a content area on the cloud desktop for displaying text, icons, etc. Since this application only selects the area to be enhanced of the cloud desktop for chroma residual encoding and transmission, and the residual value itself is also a relatively small value, the transmission bit rate required for the encoded code stream is relatively low, and the impact on the network transmission bandwidth is relatively small.

[0034] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0035] Embodiment 1:

[0036] Figure 1 The following is a flow chart of an embodiment of a cloud desktop encoding method according to an exemplary embodiment. The method can be applied to the cloud, and includes the following steps:

[0037] Step 101: Detecting a region to be enhanced contained in a first-format cloud desktop.

[0038] In this step, in order to ensure the overall computing efficiency, only the area that needs to be enhanced can be detected. The area to be enhanced refers to the chromaticity-sensitive area with high contrast and color mutation on the cloud desktop, which can be identified in the form of coordinates. Generally speaking, the cloud desktop may contain both camera-captured content and computer-generated content. The camera-captured content is the traditional natural scene video or image, which can be called the video area. The computer-generated content can be called the non-video area, such as text or icons, with drastic color changes, especially the area with colored text, which is greatly affected by the existing encoding scheme.

[0039] It can be seen that the overall computing efficiency can be ensured by detecting the non-video area on the cloud desktop as the area to be enhanced.

[0040] In an optional embodiment, the cloud desktop in the second format is obtained by performing chroma downsampling on the cloud desktop in the first format, and then the area to be enhanced is located based on the chroma distortion of the cloud desktop in the second format relative to the cloud desktop in the first format.

[0041] The first format may be a YUV4:4:4 format, the second format may be a YUV4:2:0 format, and chroma downsampling refers to downsampling the chroma components U and V of the YUV4:4:4 cloud desktop to reduce the resolution of the chroma components to 1 / 4 of the original to obtain a YUV4:2:0 cloud desktop. That is, when chroma downsampling is performed, four pixels share one chroma value.

[0042] It is worth noting that after chroma downsampling, the chroma of the second format cloud desktop is distorted relative to the chroma of the corresponding position of the first format cloud desktop, but the chroma distortion is most serious in the area with color mutation. Therefore, the chroma distortion can be used to accurately locate the area that needs to be enhanced.

[0043] See also Figure 2 The cloud desktop shown in the figure, except for the black area, is the area to be enhanced. Figure 2 It can be seen that the area to be enhanced is the computer-generated text content area contained in the cloud desktop. Due to the color mutation between the text and the text background, the text content area has serious chromaticity distortion after chromaticity downsampling. For the image area that is not completely marked in black at the bottom right of the cloud desktop, because the background color in the image is pure color, the color mutation between the object and the background is obvious, so it is regarded as the area to be enhanced.

[0044] In a specific implementation, the process of locating the area to be enhanced based on the chromaticity distortion of the second format cloud desktop relative to the first format cloud desktop includes:

[0045] First, the first format cloud desktop is divided into a plurality of image blocks, wherein by dividing the image blocks, each image block can be processed in parallel, thereby improving the computing efficiency.

[0046] Then, the average chroma distortion of each image block is determined according to the chroma component of the first format cloud desktop and the chroma component of the second format cloud desktop.

[0047] Among them, since the chroma components include two: one is the U component and the other is the V component, the average chroma distortion of the image block also includes two items: one is the average chroma distortion of the U component and the other is the average chroma distortion of the V component. And the calculation principle of the average chroma distortion of these two items is the same, and the calculation principle of the average chroma distortion is explained below.

[0048] Optionally, each image block can be divided into multiple pixel units according to the number of pixels sharing a chroma value during chroma downsampling, and the chroma distortion value of each pixel unit can be determined according to the chroma component of the first format cloud desktop and the chroma component of the second format cloud desktop, and then the average chroma distortion of each image block can be determined according to the chroma distortion value of each pixel unit.

[0049] As mentioned above, when chroma downsampling, the number of pixels that share a chroma value is 4, so the size of each pixel unit is 2*2. Assuming that the image block size is 32*32, an image block can be divided into 256 2*2 pixel units. The chroma distortion value calculation formula for each pixel unit is: Among them, Pix C,i represents the four chrominance component values ​​of the pixel unit in the corresponding image block in the first format cloud desktop, Represents the chrominance component value shared by four pixel points of the pixel unit in the corresponding image block in the second format cloud desktop, i=1, 2, 3, 4.

[0050] Furthermore, when determining the average chromaticity distortion of each image block based on the chromaticity distortion value of each pixel unit, pixel units having chromaticity distortion values ​​greater than a preset threshold value can be extracted from each pixel unit, and the average chromaticity distortion of each image block can be determined using the chromaticity distortion values ​​of the extracted pixel units and the image block size.

[0051] Among them, by preliminarily screening the chromaticity distortion values ​​of the pixel units, pixels with high contrast can be extracted. The preset threshold can be set according to practical experience. The average chromaticity distortion calculation formula of the image block is: Among them, Σσ ψ2×2 It represents the accumulated value of chroma distortion of each pixel unit, and size represents the image block size.

[0052] Finally, after the average chromaticity distortion of each image block is obtained, image blocks whose average chromaticity distortion meets a preset condition are screened out from each image block, and a region formed by the screened image blocks is determined as a region to be enhanced.

[0053] As mentioned above, the average chromaticity distortion includes the average chromaticity distortion of the U component and the average chromaticity distortion of the V component. Therefore, the preset condition is that the average chromaticity distortion of the U component is greater than or equal to the U component threshold or the average chromaticity distortion of the V component is greater than or equal to the V component threshold.

[0054] That is, if one of the average chromaticity distortion of the U component and the average chromaticity distortion of the V component meets the corresponding threshold condition, the image block is considered to belong to the area to be enhanced.

[0055] Step 102: Encode the second format cloud desktop to obtain first encoding information and reconstruct the second format cloud desktop, where the second format cloud desktop is obtained by chroma downsampling the first format cloud desktop.

[0056] In this step, a preset encoder can be used to compress and encode the second format cloud desktop to obtain a binary code stream suitable for network transmission. The encoder can be AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), JPEG (an international image compression standard), JPEG2000 (an international image compression standard), HEIF (High Efficiency Image Format), etc.

[0057] It is worth noting that when the encoder encodes the input image / video, it also obtains the corresponding reconstructed image / video, and the reconstructed image / video is consistent with the decoder output. This feature is often used to verify the development effect of the encoder and decoder. Therefore, when encoding the second format cloud desktop, in addition to obtaining the first encoding information, the second format reconstructed cloud desktop can also be obtained.

[0058] Step 103: Obtain the chroma residual in the area to be enhanced between the first format cloud desktop and the second format reconstructed cloud desktop, and encode according to the chroma residual in the area to be enhanced to obtain second encoding information.

[0059] In specific implementation, the chromaticity residual between the first format cloud desktop and the second format reconstructed cloud desktop can be calculated first, and then the chromaticity residual of the area to be enhanced is kept unchanged, and the chromaticity residuals except the area to be enhanced are modified to 0.

[0060] In step 103, the chrominance residual includes a residual of a U component and a residual of a V component.

[0061] It is understandable that an existing encoder may be used as a residual encoder to encode the chrominance residual, and of course a custom encoder may also be used, which is not limited in this application.

[0062] It should be noted that, since only the area to be enhanced has chrominance residual, other areas are all 0, and the chrominance residual itself has a very small value, the amount of data of the second encoded information obtained by encoding is relatively small, and the impact on network transmission is small.

[0063] Step 104: Send the first encoding information and the second encoding information to the client.

[0064] At this point, the above Figure 1The cloud desktop encoding process shown detects the area to be enhanced of the first format cloud desktop to adaptively select the area to be enhanced, and encodes the second format cloud desktop to obtain the first encoding information and the second format reconstructed cloud desktop. The second format cloud desktop is obtained by chroma downsampling the first format cloud desktop, and then the chroma residual between the first format cloud desktop and the second format reconstructed cloud desktop in the area to be enhanced is encoded to obtain the second encoding information, and the first encoding information and the second encoding information are sent to the client. Thus, the client decodes to obtain the reconstructed chroma residual, and superimposes the reconstructed chroma residual on the chroma component of the area to be enhanced in the second format reconstructed cloud desktop obtained by decoding, so as to improve the display effect and visual effect of the cloud desktop and improve the user experience. The area to be enhanced can be the content area on the cloud desktop for displaying text, icons, etc. Since the present application only selects the area to be enhanced of the cloud desktop for chroma residual encoding and transmission, and the residual value itself is also a relatively small value, the transmission bit rate required for the encoding code stream is relatively low, and the impact on the network transmission bandwidth is relatively small.

[0065] Embodiment 2:

[0066] Based on the above Figure 1 to Figure 2 The embodiments shown are all described from the perspective of the cloud, and the following description will be given from the perspective of the client. Figure 3 The following is a flow chart of an embodiment of a cloud desktop reconstruction method according to an exemplary embodiment. The method is applied to a client and includes the following steps:

[0067] Step 301: Receive first encoding information and second encoding information, the first encoding information is obtained by encoding the second format cloud desktop by the cloud end, and the second encoding information is obtained by encoding the chroma residual in the area to be enhanced between the first format cloud desktop and the second format reconstructed cloud desktop by the cloud end.

[0068] In this step, the client communicates with the cloud and receives the first encoding information and the second encoding information sent by the cloud. While transmitting the encoded code stream of the cloud desktop in the second format, the cloud also detects the area to be enhanced contained on the cloud desktop and transmits the encoded code stream of the chroma residual of the area to be enhanced, so that the client can enhance the chroma of the area to be enhanced on the cloud desktop, thereby improving the display effect and visual effect of the cloud desktop.

[0069] Step 302: Decode the first coded information to obtain a second format to reconstruct the cloud desktop.

[0070] In this step, a decoder corresponding to the encoder may be used for decoding.

[0071] In the embodiment of the present application, it can be considered that the second format reconstructed cloud desktop output by the encoder in the above step 102 is consistent with the second format reconstructed cloud desktop output by the decoder in this step.

[0072] Step 303: Decode the second coded information to obtain a reconstructed chroma residual of the area to be enhanced.

[0073] In this step, a residual decoder corresponding to the residual encoder used above may be used to decode the second encoded information.

[0074] It is worth noting that, since the second coding information only records the coding information of the chroma residual of the area to be enhanced, the decoded information also only obtains the reconstructed chroma residual of the area to be enhanced, and the chroma residuals of other areas are set to 0.

[0075] Step 304: superimpose the reconstructed chroma residual onto the chroma component of the area to be enhanced in the second format reconstructed cloud desktop to obtain the first format reconstructed cloud desktop.

[0076] In this step, when the cloud encodes the second format cloud desktop, it will lose some high-frequency information that is less sensitive to the human eye, including some edge and texture information. Therefore, the second format reconstructed cloud desktop obtained by decoding by the client is missing some edge and texture information, and the reconstructed chrominance residual can just restore most of the edge and texture information in the chrominance component.

[0077] In the specific implementation, since the chromaticity residual is obtained by subtracting the chromaticity components between the first format cloud desktop and the second format reconstructed cloud desktop, after superimposing the reconstructed chromaticity residual on the chromaticity components of the area to be enhanced in the second format reconstructed cloud desktop, it is equivalent to restoring the area to be enhanced in the second format to the area to be enhanced in the first format. For the area that does not need to be enhanced in the second format reconstructed cloud desktop, it can be directly restored to the first format through upsampling, so as to obtain a complete first format reconstructed cloud desktop, and the chromaticity of the area to be enhanced on the first format reconstructed cloud desktop is also enhanced, and both the display effect and the visual effect are improved.

[0078] It should be noted that after obtaining the first format reconstructed cloud desktop, the luminance component of the first format reconstructed cloud desktop can be further enhanced based on the reconstructed chrominance residual to obtain a first format reconstructed cloud desktop with enhanced luminance to eliminate the coding distortion of the luminance component.

[0079] Optionally, brightness enhancement can be achieved using a lightweight and easy-to-deploy chroma-guided brightness enhancement neural network. The implementation process is: input the reconstructed chroma residual, the brightness component and the chroma component of the first format reconstructed cloud desktop into a preset neural network model, and the neural network model outputs the brightness enhanced first format reconstructed cloud desktop based on the reconstructed chroma residual, the brightness component and the chroma component of the first format reconstructed cloud desktop.

[0080] At this point, the above Figure 3In the cloud desktop reconstruction process shown, the client decodes the first encoded information to obtain a second-format reconstructed cloud desktop, and decodes the second encoded information to obtain a reconstructed chromaticity residual of the area to be enhanced, and then superimposes the reconstructed chromaticity residual on the chromaticity component of the area to be enhanced in the second-format reconstructed cloud desktop to enhance the display effect and visual effect of the area to be enhanced. The area to be enhanced can be a content area on the cloud desktop for displaying text, icons, etc., thereby improving the display effect of the text or icon area on the cloud desktop and improving the user experience.

[0081] Based on the above Figures 1 to 3 Based on the embodiment shown, taking the first format as YUV4:4:4 and the second format as YUV4:2:0 as an example, see Figure 4 The overall implementation flow chart of cloud desktop transmission is shown, which shows the processing performed on the cloud and client sides respectively. In the figure, the left side is the cloud side processing the original cloud desktop image, and the right side is the client reconstructing the cloud desktop based on the received code stream.

[0082] On the cloud side, first convert the original cloud desktop from the RGB domain to the YUV domain, that is, obtain a YUV4:4:4 format image, then perform chroma downsampling on the YUV4:4:4 format image x to obtain a YUV4:2:0 format image x', and encode the YUV4:2:0 format image x' through the image / video encoder to output a binary code stream b BL Transmit over the network and output the reconstructed image X in YUV4:2:0 format BL rec The chroma residual encoder inputs the chroma components of the YUV4:2:0 format image x' and the chroma components of the YUV4:4:4 format image x into the image pre-analysis module, and the image pre-analysis module locates the area to be enhanced on the cloud desktop according to the chroma components of the YUV4:2:0 format image x' and the chroma components of the YUV4:4:4 format image x, and obtains a mask image. The area to be enhanced on the mask image is represented by 1, and other areas are represented by 0. The chroma residual encoder first obtains the chroma components of the YUV4:4:4 format image x and the YUV4:2:0 format reconstructed image X according to the input mask image. BL rec The chroma residual image between the chroma components of the chroma residual image, the area to be enhanced on the chroma residual image is represented by the chroma residual, and the other areas are represented by 0. Then the chroma residual encoder encodes the chroma residual image to obtain a binary code stream b EL Perform network transmission.

[0083] On the client side, the binary stream b received by the image / video encoder BL Decode and obtain the reconstructed image X in YUV4:2:0 formatBL rec , the chroma residual decoder receives the binary code stream b EL Decode and obtain the reconstructed chrominance residual e EL , and the chroma enhancement module reconstructs the chroma residual e EL Overlay to YUV4:2:0 format to reconstruct image X BL rec The chrominance components of the chrominance residual area are upsampled to obtain the YUV4:4:4 format reconstructed image e EL rec , and then reconstruct the image in YUV4:4:4 format EL rec and the reconstructed chrominance residual e EL Input brightness enhancement module, which is based on the reconstructed chroma residual e EL Reconstruct the image in YUV4:4:4 format EL rec The brightness component is enhanced and the reconstructed image X' is output in YUV4:4:4 format. Finally, the reconstructed image X' in YUV4:4:4 format is converted from YUV domain to RGB domain to obtain a reconstructed cloud desktop image that meets the visual and display effects.

[0084] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0085] Corresponding to the embodiments of the aforementioned cloud desktop encoding method, the present application also provides an embodiment of a cloud desktop encoding device, which is used to execute the cloud desktop encoding method provided in any of the above embodiments, and the cloud desktop encoding device includes:

[0086] A region detection module, used to detect the region to be enhanced contained in the first format cloud desktop;

[0087] A first encoding module is used to encode the second format cloud desktop to obtain first encoding information and reconstruct the second format cloud desktop; wherein the second format cloud desktop is obtained by chroma downsampling the first format cloud desktop;

[0088] A second encoding module is used to obtain a chroma residual in the area to be enhanced between the first format cloud desktop and the second format reconstructed cloud desktop, and encode according to the chroma residual in the area to be enhanced to obtain second encoding information;

[0089] A sending module is used to send the first coding information and the second coding information to a client.

[0090] The detecting the area to be enhanced contained in the first format cloud desktop includes:

[0091] Performing chroma downsampling on the first format cloud desktop to obtain a second format cloud desktop;

[0092] The area to be enhanced is located based on the chromaticity distortion of the second format cloud desktop relative to the first format cloud desktop.

[0093] Optionally, the region detection module is specifically used to divide the first format cloud desktop into multiple image blocks; determine the average chromaticity distortion of each image block according to the chromaticity component of the first format cloud desktop and the chromaticity component of the second format cloud desktop; screen image blocks whose average chromaticity distortion meets preset conditions in each image block; and determine the region composed of the screened image blocks as the region to be enhanced.

[0094] Optionally, the area detection module is specifically used to divide each image block into multiple pixel units according to the number of pixels sharing a chroma value during chroma downsampling in the process of determining the average chroma distortion of each image block according to the chroma component of the first format cloud desktop and the chroma component of the second format cloud desktop; determine the chroma distortion value of each pixel unit according to the chroma component of the first format cloud desktop and the chroma component of the second format cloud desktop; and determine the average chroma distortion of each image block according to the chroma distortion value of each pixel unit.

[0095] Optionally, the area detection module is specifically used to extract pixel units whose chromaticity distortion values ​​are greater than a preset threshold from each pixel unit in the process of determining the average chromaticity distortion of each image block based on the chromaticity distortion values ​​of each pixel unit; and determine the average chromaticity distortion of each image block using the extracted chromaticity distortion values ​​of the pixel units and the image block size.

[0096] Optionally, the average chromaticity distortion includes an average chromaticity distortion of a U component and an average chromaticity distortion of a V component; the preset condition is that the average chromaticity distortion of the U component is greater than or equal to a U component threshold or the average chromaticity distortion of the V component is greater than or equal to a V component threshold.

[0097] The implementation process of the functions and effects of each unit in the above-mentioned device or system is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0098] Corresponding to the embodiments of the aforementioned cloud desktop reconstruction method, the present application also provides an embodiment of a cloud desktop reconstruction device, which is used to execute the cloud desktop reconstruction method provided in any of the aforementioned embodiments, and the cloud desktop reconstruction device includes:

[0099] A receiving module, configured to receive first coded information and second coded information; wherein the first coded information is obtained by encoding the second format cloud desktop by the cloud end, and the second coded information is obtained by encoding the chroma residual in the to-be-enhanced area between the first format cloud desktop and the second format reconstructed cloud desktop by the cloud end;

[0100] A first decoding module, used to decode the first coded information to obtain a second format to reconstruct the cloud desktop;

[0101] A second decoding module, used for decoding the second coded information to obtain a reconstructed chroma residual of the area to be enhanced;

[0102] The chroma enhancement module is used to superimpose the reconstructed chroma residual on the chroma component of the area to be enhanced in the second format reconstructed cloud desktop to obtain the first format reconstructed cloud desktop.

[0103] Optionally, the device further comprises:

[0104] The brightness enhancement module is used to enhance the brightness component of the first-format reconstructed cloud desktop based on the reconstructed chroma residual after the chroma enhancement module obtains the first-format reconstructed cloud desktop, so as to obtain the brightness-enhanced first-format reconstructed cloud desktop.

[0105] Optionally, the brightness enhancement module is specifically used to input the reconstructed chromaticity residual, the brightness component and the chromaticity component of the first format reconstructed cloud desktop into a preset neural network model; and output the brightness enhanced first format reconstructed cloud desktop according to the reconstructed chromaticity residual, the brightness component and the chromaticity component of the first format reconstructed cloud desktop through the neural network model.

[0106] The implementation process of the functions and effects of each unit in the above-mentioned device or system is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0107] An embodiment of the present application also provides a cloud desktop display system, including a cloud and a client; the cloud includes an electronic device for executing the above-mentioned cloud desktop encoding method; the client includes an electronic device for executing the above-mentioned cloud desktop reconstruction method.

[0108] The embodiment of the present application also provides an electronic device to execute any one of the above-mentioned cloud desktop encoding method and reconstruction method.

[0109] Figure 5The hardware structure diagram of an electronic device according to an exemplary embodiment is shown, and the electronic device includes: a communication interface 601, a processor 602, a memory 603 and a bus 604; wherein the communication interface 601, the processor 602 and the memory 603 complete mutual communication through the bus 604. The processor 602 can execute the cloud desktop encoding method and reconstruction method described above by reading and executing the machine executable instructions corresponding to the control logic of the cloud desktop encoding method and the reconstruction method in the memory 603. The specific content of the method is referred to the above embodiment, and will not be repeated here.

[0110] The memory 603 mentioned in this application can be any electronic, magnetic, optical or other physical storage device, and can contain storage information, such as executable instructions, data, etc. Specifically, the memory 603 can be RAM (Random Access Memory), flash memory, storage drive (such as hard disk drive), any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 601 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0111] The bus 604 may be an ISA bus, a PCI bus or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 603 is used to store programs, and the processor 602 executes the programs after receiving execution instructions.

[0112] Processor 602 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in processor 602 or an instruction in software form. The above-mentioned processor 602 can be a general-purpose processor, including a network processor (Network Processor, referred to as NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute.

[0113] The electronic device provided in the embodiment of the present application and the cloud desktop encoding method and reconstruction method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.

[0114] The present application also provides a computer-readable storage medium corresponding to the cloud desktop encoding method and reconstruction method provided in the above-mentioned embodiment. Figure 6 As shown, the computer-readable storage medium shown is a CD 30, on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, it will execute the cloud desktop encoding method and reconstruction method provided by any of the aforementioned implementations.

[0115] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0116] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the cloud desktop encoding method and reconstruction method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0117] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0118] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0119] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A cloud desktop encoding method, characterized in that: The method comprises: Detecting the area to be enhanced contained in the first format cloud desktop; Encode the second format cloud desktop to obtain first encoding information and reconstruct the second format cloud desktop; wherein the second format cloud desktop is obtained by chroma downsampling the first format cloud desktop; Obtaining a chroma residual in the area to be enhanced between the first-format cloud desktop and the second-format reconstructed cloud desktop; Encoding according to the chrominance residual of the area to be enhanced to obtain second encoding information; The first encoding information and the second encoding information are sent to a client.

2. The method according to claim 1, characterized in that The detecting the area to be enhanced contained in the first format cloud desktop includes: Performing chroma downsampling on the first format cloud desktop to obtain a second format cloud desktop; The area to be enhanced is located based on the chromaticity distortion of the second format cloud desktop relative to the first format cloud desktop.

3. The method according to claim 2, characterized in that The locating the area to be enhanced based on the chromaticity distortion of the second format cloud desktop relative to the first format cloud desktop includes: Dividing the first format cloud desktop into a plurality of image blocks; Determine an average chroma distortion of each image block according to the chroma component of the first format cloud desktop and the chroma component of the second format cloud desktop; Selecting image blocks whose average chromaticity distortion meets preset conditions from among the image blocks; The area composed of the filtered image blocks is determined as the area to be enhanced.

4. The method according to claim 3, characterized in that The determining the average chroma distortion of each image block according to the chroma component of the first format cloud desktop and the chroma component of the second format cloud desktop includes: Dividing each image block into a plurality of pixel units according to the number of pixels that share a chroma value when chroma downsampling; Determine a chromaticity distortion value of each pixel unit according to the chromaticity component of the cloud desktop in the first format and the chromaticity component of the cloud desktop in the second format; According to the chromaticity distortion value of each pixel unit, the average chromaticity distortion of each image block is determined.

5. The method according to claim 4, characterized in that Determining the average chromaticity distortion of each image block according to the chromaticity distortion value of each pixel unit includes: Extracting pixel units whose chromaticity distortion values ​​are greater than a preset threshold from each pixel unit; The average chromaticity distortion of each image block is determined by using the extracted chromaticity distortion value of the pixel unit and the image block size.

6. The method according to claim 3, characterized in that The average chromaticity distortion includes the average chromaticity distortion of the U component and the average chromaticity distortion of the V component; The preset condition is that the average chromaticity distortion of the U component is greater than or equal to the U component threshold or the average chromaticity distortion of the V component is greater than or equal to the V component threshold.

7. A cloud desktop reconstruction method, characterized in that: The method comprises: Receive first encoding information and second encoding information; wherein the first encoding information is obtained by encoding the second format cloud desktop by the cloud end, and the second encoding information is obtained by encoding the chroma residual in the to-be-enhanced area between the first format cloud desktop and the second format reconstructed cloud desktop by the cloud end; Decoding the first coded information to obtain a second format to reconstruct the cloud desktop; Decoding the second coded information to obtain a reconstructed chroma residual of the area to be enhanced; The reconstructed chroma residual is superimposed on the chroma component of the area to be enhanced in the second format reconstructed cloud desktop to obtain the first format reconstructed cloud desktop.

8. The method according to claim 7, characterized in that After obtaining the first format to reconstruct the cloud desktop, the method further includes: The brightness component of the first-format reconstructed cloud desktop is enhanced based on the reconstructed chroma residual to obtain a first-format reconstructed cloud desktop with enhanced brightness.

9. The method according to claim 8, characterized in that The step of enhancing the brightness component of the cloud desktop reconstructed in the first format based on the reconstructed chroma residual includes: Inputting the reconstructed chroma residual, the brightness component and the chroma component of the cloud desktop reconstructed in the first format into a preset neural network model; The neural network model is used to reconstruct the cloud desktop in the first format based on the reconstructed chroma residual, the brightness component of the cloud desktop reconstructed in the first format, and the chroma component output brightness enhancement.

10. A cloud desktop display system, characterized in that: The system comprises: a cloud and a client; Wherein, the cloud comprises an electronic device for executing the cloud desktop encoding method according to any one of claims 1 to 6; The client includes an electronic device for executing the cloud desktop reconstruction method described in any one of claims 7-9.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the program to implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the steps of the method according to any one of claims 1 to 9.

Images