Image coding and decoding display method and system

Abstract

The invention discloses an image coding and decoding display method and system, relates to the technical field of electronic information, and can determine an image needing to additionally transmit a quality enhanced data packet by additionally considering the operation logic of a user on a cloud desktop terminal side and the refreshing frequency of the image in a cloud desktop image coding process, thereby ensuring the user experience. According to the specific scheme, a server side encodes a to-be-displayed image in an H264 lossy encoding mode and issues the to-be-displayed image to a cloud desktop terminal to be decoded and displayed. Meanwhile, a time threshold value T is set, and under the condition that the cloud desktop image is not refreshed after the time T, the server side constructs an image quality enhancement data packet of lossless coding and sends the image quality enhancement data packet to the cloud desktop terminal to be decoded and displayed. The server side sets a time threshold T according to the user input quantity uploaded by the cloud desktop terminal, and the larger the user input quantity is, the smaller the T is.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electronic information technology, in particular to an image encoding and decoding display method and system. BACKGROUND

[0002] In recent years, with the development of Internet technology, cloud desktop technology has gradually become popular. The technology that users access user virtual machines through cloud desktop terminals is usually called a remote display protocol, such as Remote Desktop Protocol (RDP), Virtual Network Computing (VNC), etc. The remote display protocol obtains user input from the cloud desktop terminal and transmits the screen content of the server-side cloud desktop to the cloud desktop terminal for display through the network. Specifically, the screen content displayed and output by the user virtual machine is captured, encoded, and then transmitted to the cloud desktop terminal through the network. The cloud desktop terminal displays the screen content on the screen after decoding the screen content. At the same time, the cloud desktop terminal captures user input and USB redirection data, analyzes the input events, and transmits them to the user virtual machine through the network. The user virtual machine executes the input logic after receiving the input.

[0003] The technology of encoding and transmitting cloud desktop images to the terminal for decoding and display is a key link of cloud desktop technology and plays a decisive role in user experience. The core goal of this technology is to reduce the size of the encoded data and the delay of the terminal decoding and display as much as possible while ensuring image quality. Traditional cloud desktop image encoding methods often use uniform parameters to process the entire image or only consider the differences in the image content itself, resulting in a difficult balance between encoding efficiency and user experience.

[0004] Therefore, it can be seen that traditional cloud desktop image encoding methods often use uniform parameters to process the entire image or only consider the differences in the image content itself, resulting in a difficult balance between encoding efficiency and user experience. SUMMARY

[0005] Therefore, the present application provides an image encoding and decoding display method and system, which can additionally consider the operation logic of the user on the cloud desktop terminal side and the refresh frequency of the image during the cloud desktop image encoding process, determine the images that need to be additionally transmitted quality enhancement data packets, and thus ensure user experience.

[0006] To achieve the above purpose, the technical scheme of the present application is an image encoding and decoding display method, which comprises the following steps: The server side uses H264 lossy encoding to encode the image to be displayed and transmits it to the cloud desktop terminal for decoding and display.

[0007] Meanwhile, a time threshold T is set, and in the case that the cloud desktop image is not refreshed for more than T time, the server end constructs a lossless encoded image quality enhancement data packet and sends it to the cloud desktop terminal for decoding and display.

[0008] The server end sets the time threshold T according to the number of user inputs uploaded by the cloud desktop terminal, wherein the greater the number of user inputs, the smaller T is.

[0009] Further, the time threshold T is specifically T=T0+T1 / (1+Y), wherein Y is the number of user inputs in the last second, i.e., the number of keyboard and mouse events captured by the cloud desktop terminal; T0 is a preset initial threshold, and T1 is a preset threshold step.

[0010] Further, the cloud desktop terminal captures and counts the number of user operations in one second and sends it to the server end, and for the user input events generated by the mouse sliding operation and the number of which exceeds a set value, the number of generated input events is added to a weight statistic into the number of user inputs.

[0011] Further, the server end uses the H264 lossy encoding mode to encode the to-be-displayed image and sends it to the cloud desktop terminal for decoding and display, and the specific steps are as follows: For the to-be-displayed image A, in the H264 lossy compression process, it is first converted to YUV space to obtain image B, and then H264 lossy encoding is used for encoding, the cloud desktop terminal decodes to obtain the YUV space image B, and then converts to RGB space to obtain image C for display.

[0012] Further, the server end constructs a lossless encoded image quality enhancement data packet, and the specific steps are as follows: first, simulate the two color space conversions of the server end and the cloud desktop terminal, i.e., first convert the to-be-displayed image A to YUV image B and then to RGB image C, and finally use the lossless encoding of image C as the image quality enhancement data packet.

[0013] Further, the to-be-displayed image is a single image block obtained by block processing of the cloud desktop image.

[0014] Another embodiment of the application further provides an image encoding and decoding display system, a server end module arranged at the server end and a cloud desktop terminal module arranged at the cloud desktop terminal.

[0015] The server-side module encodes the image to be displayed in a manner of H264 lossy coding, and sends to the cloud desktop terminal for decoding and display; meanwhile, a time threshold T is set, and in the case that the cloud desktop image is not refreshed for more than T time, the server-side module constructs an image quality enhancement data packet of lossless coding, and sends to the cloud desktop terminal for decoding and display; the server-side module sets the time threshold T according to the number of user inputs uploaded by the cloud desktop terminal, wherein the greater the number of user inputs, the smaller T is.

[0016] The cloud desktop terminal module captures and counts the number of user operations in one second and sends to the server, and for the user input events generated by the mouse sliding operation and the number of which exceeds the set value, the number of generated input events is added to the weight statistics into the number of user inputs.

[0017] Further, the server-side module specifically performs the following steps: for the image to be displayed A, in the H264 lossy compression process, the image A is first converted to YUV space to obtain image B, and then encoded using H264 lossy coding, the cloud desktop terminal decodes to obtain the YUV space image B, and then converts to RGB space to obtain image C for display; meanwhile, a time threshold T is set, and in the case that the cloud desktop image is not refreshed for more than T time, the server-side module constructs an image quality enhancement data packet of lossless coding, and the specific steps are as follows: first, simulate the two color space conversions of the server-side and the cloud desktop terminal, i.e. first convert the image to be displayed A to YUV image B and then to RGB image C, and finally take the lossless coding of image C as the image quality enhancement data packet.

[0018] Further, the image to be displayed is a single image block obtained by block processing of the cloud desktop image.

[0019] Further, the time threshold T is specifically T=T0+T1 / (1+Y), wherein Y is the number of user inputs of the last second, i.e. the number of keyboard and mouse events captured by the cloud desktop terminal; T0 is a preset initial threshold, and T1 is a preset threshold step.

[0020] Beneficial effects: 1. The image encoding and decoding display method and system provided by the application additionally consider the operation logic of the user on the cloud desktop terminal side and the refresh frequency of the image in the cloud desktop image encoding method, determine which image quality is more important for user experience based on these additional information, and additionally transmit quality enhancement data packets for these images to ensure user experience.

[0021] 2: The application provides an image coding and decoding display method and system based on a color space conversion scheme, which solves the screen brushing problem caused by the lossless image enhancement package in the desktop cloud scenario. Considering that if a completely lossless image quality enhancement package is simply constructed, the difference caused by color space conversion will cause display screen brushing problems, which greatly affects user experience. In order to solve this problem, when constructing the image enhancement package on the server side, first simulate the two color space conversions of the server and the cloud desktop terminal, that is, first convert the initial RGB picture A to YUV image B and then to RGB image C, and finally issue the lossless encoding of C image. The image enhancement package. In this way, the display screen brushing problem can be solved.

[0022] 3: The image coding and decoding display method and system provided by the application can perform block processing on the cloud desktop image, and if an image block does not refresh for more than a threshold time, a quality enhancement package is issued. In this way, the image quality enhancement package of most areas will not be affected by the continuous refreshing of areas such as flickering cursors, pointers, and stopwatches. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A flowchart of the principle of the image coding and decoding display method provided by the application. DETAILED DESCRIPTION

[0024] The application will be described in detail below with reference to the drawings and examples.

[0025] The application provides an image coding and decoding display method, the main principle of which is that the server side uses H264 lossy coding to encode the cloud desktop image and issue it to the cloud desktop terminal for decoding and display, which can bring very good operation flow and greatly reduce the network data transmission pressure. At the same time, a time threshold T is set, and in the case that the cloud desktop image is not refreshed for more than T time, the server side issues a lossless coding image quality enhancement data package. The time threshold T is affected by the user operation input of the cloud desktop terminal, and in the case that the user input frequency is high, the threshold is low, and in the case that the input frequency is low, the threshold is high. The specific algorithm used is T=T0+T1 / (1+Y), where Y is the number of user inputs in the last second (i.e. the number of keyboard and mouse events captured by the terminal); T0 is the initial threshold value, and T1 is the threshold step. It is not difficult to find that when Y is very large, the threshold is the lowest and close to T0, and when Y is 0, the threshold is the highest and T0+T1.

[0026] The cloud desktop terminal captures and counts the number of user operations in one second and sends it to the server side. For user input events generated by mouse sliding operations that exceed the set value, the number of input events generated is added to a weight statistic to the user input number.

[0027] The present application considers how to ensure that the action of issuing an image quality enhancement package does not affect the user experience. H264 lossy image compression will first perform color space conversion from RGB space to YUV space, and the software implementation of the server side and the hardware implementation of the cloud desktop terminal are different, which will cause slight differences between the YUV converted back to RGB space data of the final cloud desktop terminal and the original RGB space data. Specifically, for the initial image to be displayed A, in the H264 lossy compression process, it will be first converted to image B in YUV space and then encoded, the cloud terminal decodes to obtain image B in YUV space, and then converts to RGB space through hardware to obtain image C for display. The difference between the software and hardware RGB and YUV space conversion will cause color difference between image A and image C. Therefore, if a completely lossless image quality enhancement package is simply constructed, the difference caused by color space conversion will cause display screen flickering, which greatly affects the user experience. In order to solve this problem, when constructing the image enhancement package on the server side, first simulate the two color space conversions of the server and the cloud desktop terminal, that is, first convert the image to be displayed A to YUV image B and then to RGB image C using software, and finally issue the lossless encoding of C image to issue the image enhancement package. In this way, the display screen flickering problem can be solved. The overall solution process of the first two problems is shown in Figure 1

[0028] In the embodiment of the present application, the image to be displayed is a single image block after the cloud desktop image is processed by block. This design is to process the cloud desktop image by block, and if an image block does not refresh for more than a threshold time, a quality enhancement package is issued. In this way, the areas such as flickering cursors, pointers, and chronographs that are continuously refreshed will not affect the image quality enhancement package issued in most areas.

[0029] Embodiment 2 The present embodiment provides an image encoding and decoding display system for executing the method provided in the above-mentioned embodiment 1, which includes a server side module arranged at the server side and a cloud desktop terminal module arranged at the cloud desktop terminal.

[0030] The server side module is used for encoding the image to be displayed using the H264 lossy encoding mode and issuing it to the cloud desktop terminal for decoding and display. At the same time, a time threshold T is set, and in the case that the cloud desktop image is not refreshed for more than T time, the server side module constructs a lossless encoded image quality enhancement data package and sends it to the cloud desktop terminal for decoding and display. The server side module sets the time threshold T according to the number of user inputs uploaded by the cloud desktop terminal, wherein the greater the number of user inputs, the smaller T is. ​The cloud desktop terminal module captures and counts the number of user operations in one second and sends to the server side, and for the user input events generated by the mouse sliding operation, the number of generated input events is added to a weight statistic into the user input number.

[0031] In the embodiment of the application, the server side module specifically performs the following steps: For the image A to be displayed, in the H264 lossy compression process, the image A is first converted to YUV space to obtain image B, and then encoded using H264 lossy encoding, the cloud desktop terminal decodes to obtain the image B in YUV space, and then converts to RGB space to obtain image C for display. Meanwhile, a time threshold T is set, and in the case that the cloud desktop image is not refreshed for more than T time, the server side module constructs the image quality enhancement data packet of lossless encoding, and the specific steps are as follows: first, simulate the two color space conversions of the server side and the cloud desktop terminal, that is, first convert the image A to be displayed to YUV image B and then to RGB image C, and finally take the lossless encoding of the image C as the image quality enhancement data packet.

[0032] In the embodiment of the application, the image to be displayed is a single image block after block processing of the cloud desktop image.

[0033] In the embodiment of the application, the time threshold T is specifically T=T0+T1 / (1+Y), wherein Y is the number of user inputs of the last second, that is, the number of keyboard and mouse events captured by the cloud desktop terminal; T0 is a preset initial threshold, and T1 is a preset threshold step.

[0034] To sum up, the above is only a preferred embodiment of the application, and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. An image coding and decoding display method, characterized by, The method comprises the following steps: The server end encodes the image to be displayed using H264 lossy coding, and sends the image to the cloud desktop terminal for decoding and display. A time threshold T is set, and if the cloud desktop image is not refreshed for more than T time, the server end constructs a lossless coding image quality enhancement data packet and sends the data packet to the cloud desktop terminal for decoding and display. The server end sets the time threshold T according to the number of user inputs uploaded by the cloud desktop terminal, wherein the greater the number of user inputs, the smaller the T.

2. The image coding and decoding display method of claim 1, wherein, The time threshold T is specifically T=T0+T1 / (1+Y), wherein Y is the number of user inputs in the last second, i.e., the number of keyboard and mouse events captured by the cloud desktop terminal; T0 is a preset initial threshold, and T1 is a preset threshold step.

3. The image coding and decoding display method of claim 1, wherein, The cloud desktop terminal captures and counts the number of user operations in one second and sends the number to the server end. For user input events generated by mouse sliding operations and exceeding a set value, the number of generated input events is added to a weight statistic to the number of user inputs.

4. The image coding and decoding display method of claim 1, wherein, The server end encodes the image to be displayed using H264 lossy coding, and sends the image to the cloud desktop terminal for decoding and display. The specific steps are as follows: For the image to be displayed A, in the H264 lossy compression process, the image A is first converted to a YUV image B, and then encoded using H264 lossy coding. The cloud desktop terminal decodes the YUV image B to obtain an image C in the RGB space, and displays the image C.

5. The image coding and decoding display method of claim 1, wherein, The server end constructs a lossless coding image quality enhancement data packet. The specific steps are as follows: first, simulate the two color space conversions of the server end and the cloud desktop terminal, i.e., convert the image to be displayed A to a YUV image B and then to an RGB image C, and finally use the lossless coding of the image C as the image quality enhancement data packet.

6. The image coding and decoding display method of claim 1, wherein, The image to be displayed is a single image block obtained by block processing of a cloud desktop image.

7. An image coding and decoding display system, characterized by, A server end module is arranged on the server end, and a cloud desktop terminal module is arranged on the cloud desktop terminal. The server end module encodes the image to be displayed using H264 lossy coding, and sends the image to the cloud desktop terminal for decoding and display. Meanwhile, a time threshold T is set, and if the cloud desktop image is not refreshed for more than T time, the server end module constructs a lossless coding image quality enhancement data packet and sends the data packet to the cloud desktop terminal for decoding and display. The server end module sets the time threshold T according to the number of user inputs uploaded by the cloud desktop terminal, wherein the greater the number of user inputs, the smaller the T. The cloud desktop terminal module captures and counts the number of user operations in one second and sends the number to the server end. For user input events generated by mouse sliding operations and exceeding a set value, the number of generated input events is added to a weight statistic to the number of user inputs.

8. An image coding and decoding display system as claimed in claim 7, characterized in that The server end module specifically performs the following steps: For the image to be displayed A, in the H264 lossy compression process, the image A is first converted to a YUV image B, and then encoded using H264 lossy coding. The cloud desktop terminal decodes the YUV image B to obtain an image C in the RGB space, and displays the image C. Meanwhile, a time threshold T is set, and in a case where the cloud desktop image is not refreshed for more than T time, the server-side module constructs an image quality enhancement data packet of lossless coding, and the specific steps are as follows: firstly, simulating twice color space conversion of the server side and the cloud desktop terminal, that is, firstly converting the image to be displayed A to YUV image B and then to RGB image C, and finally taking the lossless coding of the image C as the image quality enhancement data packet.

9. The image coding and decoding display system of claim 1, wherein, The image to be displayed is a single image block obtained by block processing of the cloud desktop image.

10. The image coding and decoding display system of claim 1, wherein, The time threshold T is specifically T=T0+T1 / (1+Y), wherein Y is the number of user inputs in the last second, that is, the number of keyboard and mouse events captured by the cloud desktop terminal; T0 is a preset initial threshold, and T1 is a preset threshold step.

Images