A cloud desktop image transmission method and device

Abstract

The present invention provides a cloud desktop image transmission method and device. Used to solve the technical problem of low transmission efficiency of text-based cloud desktop images, the present invention divides text-based desktop images into two levels of blocks, performs a hit judgment on the two-level blocks of the desktop image to be transmitted, and replaces the image data with the block cache identifier of the hit block. The second-level blocks are divided into horizontal blocks to distinguish between text blocks and non-text blocks, making full use of the characteristics of the text interface, reducing the bandwidth occupancy of the text desktop scene in the cloud desktop transmission protocol, and increasing the fluency of the cloud desktop in a weak network environment.

Description

Technical Field

[0001] The present invention relates to the technical field of communications and cloud computing, and in particular to a cloud desktop image transmission method and device. Background Art

[0002] In the actual deployment of the current Virtual Desktop Infrastructure (VDI) protocol, a cloud desktop transmission protocol client (referred to as cloud desktop client) is usually installed on a thin terminal. The cloud desktop client and the cloud desktop server interact with each other through the remote desktop protocol. The cloud desktop server compresses the cloud desktop image and transmits it to the cloud desktop client. The cloud desktop client decompresses it and presents it to the user through the local display system.

[0003] The efficiency of cloud desktop data transmission has always been the focus of developers and users. On the one hand, bandwidth is a user resource and involves cost investment. The high bandwidth occupied by a single cloud desktop connection will lead to too high total bandwidth when everyone uses the cloud desktop at the same time, which will affect the network equipment investment and operator bandwidth leasing costs when users build cloud desktop environments. On the other hand, high bandwidth usage will affect the user experience under poor link conditions (such as WAN remote connection).

[0004] Currently, in the VDI infrastructure, when the virtual machine on the server refreshes the desktop, it can use interface cache deduplication to reduce the amount of data transmitted, and only transmit the image of the desktop change area to the cloud desktop client. However, in the scenario where the cloud desktop mainly displays a text interface, the server usually needs to use lossless compression, and the desktop data transmission efficiency of the cloud desktop is not high. Summary of the invention

[0005] In view of this, the present invention provides a cloud desktop text image transmission method, device and storage medium, which are used to solve the technical problem of low transmission efficiency of text-type cloud desktop image data.

[0006] According to one aspect of an embodiment of the present invention, a cloud desktop image transmission method is provided, which is applied to a cloud desktop server, and includes:

[0007] Determine whether the desktop image to be transmitted is a text interface;

[0008] When it is determined to be a text interface, all first-level blocks intersecting with the desktop refresh area are obtained to form a first-level refresh block set; the first-level block is a screen display area obtained by segmenting the cloud desktop based on a preset first-level block size;

[0009] Scan and split each primary block in the primary refresh block set into secondary blocks, and calculate the hash value of each secondary block;

[0010] Calculate the hash values ​​of all the first-level blocks of the desktop image to be transmitted;

[0011] For the first-level blocks in the first-level unrefreshed block set, the block cache identifier of the block of the first-level block that is hit is replaced; for each first-level block in the first-level refreshed block set, the block cache identifier of the block of the second-level block that is hit is replaced;

[0012] The desktop image data to be transmitted that has completed the block cache identifier replacement process is packaged and compressed and then sent to the cloud desktop client.

[0013] Furthermore, the method for determining whether the desktop image to be transmitted is a text interface is as follows:

[0014] By comparing the rate of change of the color change of the desktop image, it is determined whether the desktop image is a text interface. When the rate of change is greater than a preset rate of change threshold, it is determined to be a text interface, otherwise it is determined to be a non-text interface; or,

[0015] A trained deep learning neural network model is used to determine whether the input desktop image is a text interface.

[0016] Furthermore, the method of scanning and dividing each primary block in the primary refresh block set into secondary blocks is:

[0017] The image data of the first-level block is scanned at the pixel level, and the area containing text and the blank area are divided into multiple second-level blocks.

[0018] Furthermore, the method for calculating the hash value of the first-level block is:

[0019] The hash values ​​of all the secondary blocks of the primary block are combined, the hash value of the combined data is calculated, and the hash value of the combined data is used as the hash value of the image data of the primary block.

[0020] Furthermore, the method for replacing the block cache identifier of the hit secondary block for each primary block in the primary refresh block set is:

[0021] Use the hash value of the secondary block to match the secondary block cache list in the local cache pool to determine whether it hits;

[0022] If the secondary block cache data structure in the secondary block cache list is hit, the image data of the secondary block is replaced by the block cache identifier of the secondary block;

[0023] If the secondary block cache data structure in the secondary block cache list is not hit, the cache data structure of the secondary block is recorded in the local cache pool;

[0024] The secondary block cache data structure includes but is not limited to the block start coordinates, block length and width, block cache identifier, and block hash value.

[0025] Furthermore, the method also includes an aging and cleaning step of the cache data structure of the primary block and the secondary block, including:

[0026] Add an aging time field to each primary block and secondary block to record the cache duration;

[0027] Every time a block is hit, the cache time field of the block is refreshed;

[0028] The aging cleanup timer periodically determines whether the cache duration of each block exceeds the preset threshold. If so, the cache space of the corresponding block is released.

[0029] According to another aspect of the embodiment of the present invention, a cloud desktop image transmission device is also provided, which is applied to a cloud desktop server, including:

[0030] A text interface determination module, used to determine whether the desktop image to be transmitted is a text interface;

[0031] A refresh block acquisition module is used to acquire all the first-level blocks that intersect with the desktop refresh area to form a first-level refresh block set when it is determined to be a text-type interface; the first-level block is a screen display area obtained by segmenting the cloud desktop based on a preset first-level block size;

[0032] The secondary block segmentation module is used to scan and segment each primary block in the primary refresh block set into secondary blocks, and calculate the hash value of each secondary block;

[0033] A primary block hash module, used to calculate the hash values ​​of all primary blocks of the desktop image currently to be transmitted;

[0034] The block cache identifier replacement module is used to replace the block cache identifier of the block of the first-level block that hits the first-level block in the first-level unrefreshed block set; and replace the block cache identifier of the block of the second-level block that hits each first-level block in the first-level refreshed block set;

[0035] The compression transmission module is used to package and compress the desktop image data to be transmitted after completing the block cache identifier replacement process and then send it to the cloud desktop client.

[0036] Furthermore, the text interface judgment module judges whether the desktop image is a text interface by comparing the rate of change of the color change of the desktop image. When the rate of change is greater than a preset rate of change threshold, it is judged to be a text interface, otherwise it is judged to be a non-text interface; or a trained deep learning neural network model is used to judge whether the input desktop image is a text interface.

[0037] Furthermore, the secondary block segmentation module performs pixel-level line scanning on the image data of the primary block, and segments the area containing text and the blank area into a plurality of secondary blocks.

[0038] Furthermore, the device also includes:

[0039] The aging processing module is used to add an aging time field to each primary block and secondary block to record the cache duration; each time a block is hit, the cache time field of the block is refreshed; and the aging cleaning timer is used to periodically determine whether the cache duration of each block exceeds the preset threshold. If so, the cache space of the corresponding block is released.

[0040] According to another aspect of the present invention, a cloud desktop image transmission device is provided. The device is applied to a cloud desktop client and includes:

[0041] The decompression module is used to receive the compressed data packet of the text interface sent by the cloud desktop server and parse the compressed data packet;

[0042] A primary block processing module, for directly reading the image data of the corresponding primary block from the local cache through the block cache identifier carried in the data packet when the hit type identifier of the block in the data packet indicates a hit of the primary block; and for reading the image data of the primary block from the data packet and caching the image data of the corresponding primary block in the local cache pool when the hit type identifier of the block in the data packet indicates a miss of the primary block and the secondary block;

[0043] A secondary block processing module is used to parse the data structure of the secondary block included in the data packet and read the data structure of the secondary block in sequence when the hit type identifier of the block in the data packet indicates that the secondary block is hit; if the hit type field in the data structure of the secondary block indicates that the current secondary block is hit, the client directly searches and reads the image data of the corresponding secondary block in the local cache pool according to the block cache identifier in the data structure of the secondary block; if the hit type field in the data structure of the secondary block indicates that the current secondary block is not hit, the client reads the image data of the secondary block from the data structure of the current secondary block and caches it in the local cache pool;

[0044] The block combination module is used to assemble the image data of all the first-level blocks read from the compressed data packet into complete desktop image data, and output it to the display driver of the client.

[0045] The present invention divides text-based desktop images into two levels of blocks, performs hit judgment on the two-level blocks of the desktop image to be transmitted, and replaces the image data with the block cache identifier of the hit block. The segmentation of the secondary blocks adopts horizontal block segmentation to distinguish text blocks from non-text blocks, making full use of the characteristics of the text-based interface, reducing the bandwidth occupancy of the text desktop scene in the cloud desktop transmission protocol, and increasing the fluency of the cloud desktop in a weak network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments of the present invention or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present invention.

[0047] Figure 1 A schematic diagram of a text interface and a desktop image block division displayed on a cloud desktop in one embodiment of the present invention;

[0048] Figure 2 This is an example diagram of dividing a text interface into secondary blocks based on the primary blocks according to an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the steps of a cloud desktop image transmission method according to an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of the structure of a server-side cloud desktop image transmission device provided by an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of the structure of a client cloud desktop image transmission device provided by an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, rather than limiting the embodiments of the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings. The term "and / or" used in the present invention refers to any or all possible combinations of one or more associated listed items.

[0054] It should be understood that although the terms first, second, third, etc. may be used to describe various information in embodiments of the present invention, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "at the time of" or "when" or "in response to determining".

[0055] In an office scenario, the cloud desktop may display a text interface most of the time. For example, when users use software such as Word, Excel, Notepad, Outlook, PDF, etc. to view and edit documents or emails, or use web pages or reading software to read articles, most of the areas presented on the desktop may be lines of text, tables, etc.

[0056] In order to improve the efficiency of cloud desktop data transmission, a cloud desktop block cache mechanism can be used, that is, the cloud desktop image is divided into blocks according to a preset size, for example, the desktop is divided into blocks according to pixel block area sizes such as 16×16, 64×64, 128×128 or 256×256. The cloud desktop server packages the divided cloud desktop image data and sends it to the cloud desktop client running on the terminal. The cloud desktop server and the client both cache the sent desktop block image data. If a cached block image data is hit later, the server only sends the block cache identifier (Cache ID) of the block image data to the cloud desktop client, and the client directly obtains data from the local cache through the block cache identifier to fill the display.

[0057] After analysis, the inventors found that although the above-mentioned cloud desktop image block data caching transmission method has improved the transmission efficiency of cloud desktop data to a certain extent, for text-based interfaces, the block data caching method does not fully utilize the characteristics of text-based interfaces, resulting in a low hit probability of block image data and a low transmission efficiency.

[0058] Figure 1The figure is a schematic diagram of the text interface and desktop image block division displayed by the cloud desktop in one embodiment of the present invention. Multiple lines of text are displayed in the desktop image of the cloud desktop, and there is blank space between the lines of text. The entire desktop image is divided into N blocks, and the size of each block is L*W, where L is the length and W is the width, and the unit can be pixels. For example, a desktop image with a resolution of 256*256 pixels can be divided into N=16 blocks of L=64 pixels and W=64 pixels. As can be seen from the figure, there are a lot of blank spaces between the lines of text and around the document, and these blank areas are usually unchanged. If the large blocks in the figure are cached, as long as any text symbol or line position in the block is changed, the block data will change. The server needs to compress the entire data and transmit it to the cloud desktop client. In fact, many blank areas have not changed, and the area where the text changes may account for a small percentage of the entire block area. If the entire block data is compressed and transmitted, it will cause bandwidth waste.

[0059] In order to improve the desktop image transmission efficiency in the text desktop scene in the cloud desktop transmission protocol and increase the fluency of the cloud desktop displaying the text interface in an environment with insufficient network bandwidth resources, the present invention proposes a cloud desktop image transmission method, which involves processing steps at both ends of the cloud desktop server and the cloud desktop client, and requires coordination and cooperation between the two ends to achieve the purpose of the invention. The processing action of one party determines the execution timing and execution action of the corresponding action of the other party. For example, the server adopts a secondary interface image block caching method to transmit desktop image data to the client, and the processing method of the server determines that the client also adopts a corresponding secondary interface image block caching method to receive and process the cloud desktop image.

[0060] The basic idea of ​​the present invention is to identify the interface type for the cloud desktop image and divide the text interface into two levels of blocks. First, the desktop image is divided into a first-level block as a whole, and then a second-level block is divided inside the first-level block. The second-level block division adopts a non-fixed horizontal or vertical (determined according to the text typesetting format) block division method that conforms to the characteristics of text typesetting. The cloud desktop server and the client respectively establish a two-level block cache list locally. When the first-level block has not changed and needs to be transmitted repeatedly, the block cache identifier of the first-level block is directly transmitted to the cloud desktop client. When the first-level block cache does not hit, the judgment of the second-level block continues. When the second-level block cache hits, the block cache identifier of the second-level block is transmitted to the cloud desktop client. When the cloud desktop client receives the block cache identifier of the first-level block or the second-level block, it directly reads the cache block corresponding to the block cache identifier from the local for display.

[0061] Figure 2 This is an example diagram of dividing the text interface into secondary blocks based on the primary blocks in one embodiment of the present invention. The size of the secondary blocks may not be fixed, for example Figure 2 In the example, the i-th first-level block is divided into four second-level blocks according to the locations of the text and blank areas.

[0062] Figure 3 The figure is a schematic diagram of the steps of the cloud desktop image transmission method in one embodiment of the present invention, which illustrates the specific implementation process of the server transmitting the cloud desktop image, including:

[0063] Step 301. The cloud desktop server determines whether the desktop image to be transmitted is a text interface;

[0064] In the cloud desktop infrastructure, the server virtual graphics driver will capture the cloud desktop image, which will be processed and sent frame by frame. The cloud desktop server may select different transmission modes according to the scene configuration, such as streaming mode transmission in extreme speed mode and micro-frame algorithm for processing. In HD mode, a series of processing processes such as deduplication, text interface separation and compression are used before sending it to the cloud desktop client. In HD mode, the desktop image can be divided into two categories: text interface and non-text interface images, and processed separately. Different algorithms are used to process them to improve transmission efficiency. Finally, the processed desktop image is packaged, compressed using a preset compression algorithm, and sent to the client. Deduplication refers to removing the same parts of the previous and next frame images. The refresh area of ​​the desktop can be known through the deduplication step, and the repeated parts are not sent, thereby reducing the amount of data sent.

[0065] Step 311. The present invention does not elaborate on the processing method of the non-text interface. For example, algorithms such as nebula can be used.

[0066] In one embodiment of the present invention, the rate of change of the color change of the desktop image is compared to determine whether the desktop image is a text interface. Generally, the rate of change of the color of a natural image is relatively gentle, while the rate of change of a text interface image is generally steep. Therefore, a rate of change threshold can be set to determine whether it is a text interface. When the rate of change is greater than the preset rate of change threshold, it is determined to be a text interface, otherwise it is determined to be a non-text interface. For example, the rate of change is determined by calculating the hash value of the sum of all pixels in each pixel row, and then comparing the difference in hash values ​​between rows.

[0067] In another embodiment of the present invention, a deep learning neural network model is used to more accurately determine whether the current desktop image is a text interface, such as Vgg, googlenet, and Inceptionv4 neural network algorithm models. The neural network model is trained through samples to obtain a final applicable neural network model. The desktop image is input into the trained neural network model, and the neural network model can output a judgment result.

[0068] Step 302. When it is determined to be a text interface, all first-level blocks that have an intersection with the desktop refresh area are obtained to form a first-level refresh block set, and those without an intersection form a first-level non-refresh block set; wherein the first-level block is a screen display area obtained by segmenting the cloud desktop based on a preset first-level block size;

[0069] The purpose of this step is to determine which first-level blocks are involved in the desktop refresh area. For the unrefreshed first-level blocks, the cache data structure of the first-level block image data in the local cache pool will be hit, and the image data can be directly replaced with the block cache identifier of the first-level block; and for the refreshed first-level blocks, it is necessary to continue the hit judgment and processing of the second-level blocks.

[0070] When a text interface is transmitted for the first time, the entire desktop is the refresh area, and the server and client will establish a cache data structure of all first-level blocks locally. After the text interface is refreshed, the cloud desktop server can obtain the location and size of the desktop refresh area through the virtual graphics driver interface, and based on the distribution position of each first-level block on the screen, it can obtain all first-level blocks that intersect with the desktop refresh area.

[0071] In the present invention, the first-level blocks can be divided into fixed-size blocks, that is, the size of each first-level block is consistent and the position is fixed, such as Figure 1 Of course, according to the layout characteristics of the actual text interface of different languages, the division method of non-fixed-size primary blocks can also be adopted, as long as the cloud desktop server and the client are synchronized, and the present invention does not make specific limitations.

[0072] Step 303: Scan and split each primary block in the primary refresh block set into secondary blocks, and calculate the hash value of each secondary block.

[0073] In one embodiment of the present invention, the method for scanning and dividing the primary block is:

[0074] Step 3031. Perform pixel-level line scanning on the image data of the primary block, and divide the area containing text and the blank area into multiple secondary blocks;

[0075] refer to Figure 2For example, by scanning pixel rows, the i-th primary block can be horizontally divided into 4 secondary blocks. The specific division method can be: starting from the first row, scan row by row. If the current row includes non-white pixels (assuming that the color of the blank area between text lines is white), it means that the row is in the text area. Continue to scan downward until a row with a whole white row or the last row of the primary block is scanned, and the previous multiple continuous rows in the text area are divided into a group to form a secondary block. In the same way, if a row with a whole white row (blank area) is scanned, it means that the row is in the blank area. Continue to scan downward until a row with a whole white row or the last row of the primary block is scanned, and the previous multiple continuous rows in the blank area are divided into a group to form a secondary block.

[0076] In the above method, the first-level block can be divided into multiple second-level blocks. Considering that the background of some text interface documents is not necessarily pure white, the desktop image can be processed before dividing the second-level blocks, and the desktop image can be converted into a black and white image (to highlight the text and leave blank space), and then the second-level blocks can be divided.

[0077] Step 3032. Calculate the hash value of each secondary block, generate the secondary block data structure of each secondary block, and form the secondary block list of the current primary block.

[0078] The secondary block data structure may include the starting coordinates of the secondary block, the length and width of the block, the hash value of the block, the block cache identifier of the block, etc. The secondary block data structure constitutes the secondary block list of the current primary block, and the secondary block list can be temporarily stored in the memory; when the secondary block data structure and the secondary block list are cached in the local cache pool, they are called the secondary block cache data structure and the secondary block cache list.

[0079] Step 304. Calculate the hash values ​​of all primary blocks of the desktop image to be transmitted;

[0080] This step will calculate the hash values ​​of all the first-level blocks of the desktop image to be transmitted, including the first-level refresh blocks and the first-level non-refresh blocks. In one embodiment of the present invention, for the first-level blocks in the first-level refresh block set, the method of calculating their hash values ​​can be the following two methods:

[0081] Method 1: directly perform hash value calculation on the image data of the first-level block to obtain the hash value of the first-level block;

[0082] Method 2: Combine the hash values ​​of all the secondary blocks of the primary block, calculate the hash value of the combined data, and use the hash value of the combined data as the hash value of the image data of the primary block.

[0083] Step 305: For each primary block in the primary refresh block set, replace the block cache identifier of the hit secondary block.

[0084] The first-level blocks in the first-level refresh block set are all first-level blocks that have changed relative to the previously transmitted desktop image. Therefore, the first-level blocks in the first-level refresh block set will not hit the first-level block cache data structure in the local cache pool, so these first-level blocks need to be compressed and transmitted as second-level blocks. The basic idea of ​​block cache compression transmission of second-level blocks is: for the second-level blocks that hit the second-level block cache data structure in the local cache pool, replace the block image data with the block cache identifier of the second-level block.

[0085] In this embodiment, the local cache pool is used to cache the first-level block cache data structure and the second-level block cache data structure that have been successfully transmitted before, and the first-level and second-level block cache data structures can be managed through lists. For example, all the first-level block cache data structures form a first-level block cache list, and each first-level block cache data structure is connected to the second-level block cache list composed of all the second-level block cache data structures belonging to the first-level block through a pointer. The first-level and second-level block cache data structures may include but are not limited to the block starting coordinates, block length and width, block cache identifier (or block number), and hash value of the block. The block cache identifier of the first-level and second-level blocks can be the hash value of the block, or a unique identifier generated by the server and the client according to preset rules, which is used to uniquely identify a block at both ends. The present invention does not specifically limit the organizational management form of the first-level and second-level block cache data structures.

[0086] In this step, the following processing steps 3051 to 3053 need to be called cyclically to process all secondary blocks in the primary block one by one until all secondary blocks in the current primary block are processed.

[0087] Step 3051. Use the hash value of the secondary block to match the secondary block cache list in the local cache pool to determine whether it hits;

[0088] Step 3052: If the secondary block cache data structure in the secondary block cache list is hit, the image data of the block is replaced by the block cache identifier of the secondary block.

[0089] The first and second level blocks cached in the local cache pool will have the corresponding first and second level block image data and the corresponding first and second level block cache data structure cached in the cloud desktop client. When the cloud desktop client receives the block cache identifier of the first and second level blocks sent by the server, it can read the corresponding block image data from the local cache for display.

[0090] Step 3053. If the secondary block cache data structure in the secondary block cache list is not hit, the cache data structure of the secondary block is recorded in the local cache pool;

[0091] In the case of a miss in the secondary block, it is necessary to record the cache data structure of the secondary block in the secondary block cache list of the current primary block in the local cache pool.

[0092] Step 306: For the first-level blocks in the first-level unrefreshed block set, replace the block cache identifier of the hit first-level block;

[0093] The principle of the block cache compression transmission process of the first-level block is similar to that of the second-level block. For the first-level block that hits the first-level block cache data structure in the local cache pool, the block image data is identified by the block cache of the first-level block.

[0094] Step 307: Pack and compress the image data of the current desktop image to be transmitted and send it to the cloud desktop client.

[0095] In an embodiment of the present invention, the cloud desktop server can package and compress the image data to be transmitted in units of desktop image frames. The packaged desktop image data to be transmitted includes the block cache identifier of the hit block, the image data of the missed block, etc. The packaged data packet will organize these image data to be sent in a certain data structure. In the packaged data packet data structure, for each block, a hit type identifier indicating whether the first and second level blocks are hit can be carried.

[0096] In the embodiment of the present invention, although described in a sequential step manner, the order of performing block cache compression transmission of the first-level refreshed blocks and the first-level unrefreshed blocks is not limited. The refreshed blocks and the unrefreshed blocks may be arranged in sequence or alternately, for example Figure 1 The first-level block in the dotted box is a refresh block, and the one in the non-dotted box is an unrefreshed first-level block. The server may process the image data transmission of a first-level refresh block and then process the image data transmission of a first-level unrefreshed block, which is determined by the arrangement relationship of the blocks.

[0097] In one embodiment of the present invention, in order to avoid excessive consumption of the local cache and to promptly release the cached block data that has become invalid after refreshing, an aging cleaning step of the cache data structure of the primary block and the secondary block is also included, including:

[0098] (1) Add an aging time field to each primary block and secondary block to record the cache duration;

[0099] (2) Every time a block is hit, the cache time field of the block is refreshed;

[0100] (3) The aging cleanup timer periodically determines whether the cache duration of each block exceeds a preset threshold. If so, the cache space of the corresponding block is released.

[0101] In one embodiment of the present invention, a hit count field may be added to the cache data structure of the first and second level blocks, and the hit count is increased after each hit. During aging cleaning, cache data of blocks with low hit rates are cleaned first.

[0102] In the embodiment of the present invention, the cloud desktop client will divide the desktop image into blocks using the same primary block division method as the server. When the client receives the compressed data packet of the text interface sent by the cloud desktop server, it decompresses the data packet and performs the following processing:

[0103] If the hit type identifier of the block in the compressed data packet indicates that it hits a first-level block, the client directly reads the image data of the corresponding first-level block from the local cache through the block cache identifier carried in the compressed data packet;

[0104] If the hit type identifier of the block in the compressed data packet indicates that the primary block and the secondary block are not hit, the client reads the image data of the primary block from the compressed packet and caches the image data of the corresponding primary block in the local cache pool;

[0105] If the hit type identifier of the block in the compressed data packet indicates a hit on the secondary block, the client further parses the data structure of the secondary block included in the data packet and reads the data structure of the secondary block in sequence; if the hit type field in the data structure of the secondary block indicates a hit on the current secondary block, the client directly searches the cache data structure of the secondary block in the local cache pool according to the block cache identifier in the data structure of the secondary block, and then reads the image data of the corresponding secondary block; if the hit type field in the data structure of the secondary block indicates a miss on the current secondary block, the client reads the image data of the secondary block from the data structure of the current secondary block and caches it in the local cache pool (or overwrites the image data of the corresponding area in the current primary block).

[0106] After the client reads the image data of all the first-level blocks from the compressed data packet, it assembles the complete desktop image data and outputs it to the display driver of the client to display the desktop image.

[0107] Figure 4 This is a schematic diagram of the structure of a server-side cloud desktop image transmission device provided by an embodiment of the present invention. Each functional module in the device 400 can be implemented by software, hardware, or a combination of software and hardware. The device 400 includes:

[0108] The text interface determination module 401 is used to determine whether the desktop image to be transmitted is a text interface;

[0109] The refresh block acquisition module 402 is used to acquire all the first-level blocks that intersect with the desktop refresh area to form a first-level refresh block set when it is determined to be a text-type interface; the first-level block is a screen display area obtained by segmenting the cloud desktop based on a preset first-level block size;

[0110] The secondary block segmentation module 403 is used to scan and segment each primary block in the primary refresh block set into secondary blocks, and calculate the hash value of each secondary block;

[0111] The first-level block hash module 404 is used to calculate the hash values ​​of all the first-level blocks of the desktop image to be transmitted;

[0112] The block cache identifier replacement module 405 is used to replace the block cache identifier of the block of the first-level block that hits the first-level block in the first-level unrefreshed block set; and replace the block cache identifier of the block of the second-level block that hits each first-level block in the first-level refreshed block set;

[0113] The compression transmission module 406 is used to package and compress the desktop image data to be transmitted after the block cache identifier replacement process is completed and then send it to the cloud desktop client.

[0114] Furthermore, the text interface judgment module 401 judges whether the desktop image is a text interface by comparing the image change rate. When the change rate is greater than a preset change rate threshold, it is judged to be a text interface, otherwise it is judged to be a non-text interface; or a trained deep learning neural network model is used to judge whether the input desktop image is a text interface.

[0115] Furthermore, the secondary block segmentation module 403 performs pixel-level line scanning on the image data of the primary block, and segments the area containing text and the blank area into a plurality of secondary blocks.

[0116] Furthermore, the device 400 also includes an aging processing module, which is used to add an aging time field to each first-level block and second-level block to record the cache duration; each time a block is hit, the cache time field of the block is refreshed; and the aging cleanup timer is used to periodically determine whether the cache duration of each block exceeds a preset threshold, and if so, release the cache space of the corresponding block.

[0117] Figure 5 This is a schematic diagram of the structure of a client cloud desktop image transmission device provided by an embodiment of the present invention. Each functional module in the device 500 can be implemented by software, hardware, or a combination of software and hardware. The device 500 includes:

[0118] The decompression module 501 is used to receive the compressed data packet of the text interface sent by the cloud desktop server and parse the compressed data packet;

[0119] The primary block processing module 502 is used to directly read the image data of the corresponding primary block from the local cache through the block cache identifier carried in the data packet when the hit type identifier of the block in the data packet indicates that the primary block is hit; and when the hit type identifier of the block in the data packet indicates that the primary block and the secondary block are not hit, read the image data of the primary block from the data packet, and cache the image data of the corresponding primary block in the local cache pool;

[0120] The secondary block processing module 503 is used to parse the data structure of the secondary block included in the data packet and read the data structure of the secondary block in sequence when the hit type identifier of the block in the data packet indicates a hit on the secondary block; if the hit type field in the data structure of the secondary block indicates a hit on the current secondary block, the client directly searches and reads the image data of the corresponding secondary block in the local cache pool according to the block cache identifier in the data structure of the secondary block; if the hit type field in the data structure of the secondary block indicates a miss on the current secondary block, the client reads the image data of the secondary block from the data structure of the current secondary block and caches it in the local cache pool.

[0121] The block combination module 504 is used to assemble the image data of all the first-level blocks read from the compressed data packet into complete desktop image data, and output it to the display driver of the client.

[0122] Figure 6 A schematic diagram of the structure of an electronic device provided in one embodiment of the present invention, the device 600 includes: a processor 610 such as a central processing unit (CPU), a communication bus 620, a communication interface 640, and a storage medium 630. The processor 610 and the storage medium 630 can communicate with each other through the communication bus 620. The storage medium 630 stores a computer program for implementing the cloud desktop image transmission method provided by the present invention, or the storage medium 630 includes a software function module that can implement the functions of the device 400 or the device 500. When the computer program is executed by the processor 610 or the software function module is loaded and run, the functions of each step of the method provided by the present invention can be implemented.

[0123] Among them, the storage medium may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. In addition, the storage medium may also be at least one storage device located away from the aforementioned processor. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0124] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner. Each program may be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if necessary, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose. In addition, the operation of the process described in the present invention may be performed in any suitable order, unless the present invention otherwise indicates or otherwise clearly contradicts the context. The process described in the present invention (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0125] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described in the present invention includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0126] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud desktop image transmission method, characterized in that: The method is applied to the cloud desktop server, including: Determine whether the desktop image to be transmitted is a text interface; When it is determined to be a text interface, all first-level blocks intersecting with the desktop refresh area are obtained to form a first-level refresh block set; the first-level block is a screen display area obtained by segmenting the cloud desktop based on a preset first-level block size; Scan and split each primary block in the primary refresh block set into secondary blocks, and calculate the hash value of each secondary block; For each primary block of the desktop image to be transmitted, the hash values ​​of all the secondary blocks of the primary block are combined, the hash value of the combined data is calculated, and the hash value of the combined data is used as the hash value of the image data of the primary block; or, the hash value of the image data of the primary block is calculated to obtain the hash value of the primary block; For each first-level block in the first-level unrefreshed block set, the hash value of the first-level block is used to match the first-level block cache list in the local cache pool to determine whether it is a hit; if the first-level block cache data structure in the first-level block cache list is hit, the desktop image data of the block is replaced with the block cache identifier of the first-level block; the block cache identifier is used to uniquely identify a block on both the cloud desktop server and the cloud desktop client; For each primary block in the primary refresh block set, the hash value of the secondary block is used to match the secondary block cache list in the local cache pool to determine whether it is a hit; if the secondary block cache data structure in the secondary block cache list is hit, the desktop image data of the block is replaced with the block cache identifier of the secondary block; The desktop image data to be transmitted that has completed the block cache identifier replacement process is packaged and compressed and then sent to the cloud desktop client.

2. The method according to claim 1, characterized in that: The method for determining whether the desktop image to be transmitted is a text interface is as follows: By comparing the rate of change of the color change of the desktop image, it is determined whether the desktop image is a text interface. When the rate of change is greater than a preset rate of change threshold, it is determined to be a text interface, otherwise it is determined to be a non-text interface; or, A trained deep learning neural network model is used to determine whether the input desktop image is a text interface.

3. The method according to claim 1, characterized in that The method of scanning and dividing each primary block in the primary refresh block set into secondary blocks is: The image data of the first-level block is scanned at the pixel level, and the area containing text and the blank area are divided into multiple second-level blocks.

4. The method according to claim 1, characterized in that: The method further comprises: If the secondary block cache data structure in the secondary block cache list is not hit, the cache data structure of the secondary block is recorded in the local cache pool; The secondary block cache data structure includes but is not limited to the block start coordinates, block length and width, block cache identifier, and block hash value.

5. The method according to claim 1, characterized in that The method further includes an aging and cleaning step of the cache data structure of the primary block and the secondary block, including: Add an aging time field to each primary block and secondary block to record the cache duration; Every time a block is hit, the cache time field of the block is refreshed; The aging cleanup timer periodically determines whether the cache duration of each block exceeds the preset threshold. If so, the cache space of the corresponding block is released.

6. A cloud desktop image transmission device, characterized in that: The device is applied to the cloud desktop server, including: A text interface determination module, used to determine whether the desktop image to be transmitted is a text interface; A refresh block acquisition module is used to acquire all the first-level blocks that intersect with the desktop refresh area to form a first-level refresh block set when it is determined to be a text-type interface; the first-level block is a screen display area obtained by segmenting the cloud desktop based on a preset first-level block size; The secondary block segmentation module is used to scan and segment each primary block in the primary refresh block set into secondary blocks, and calculate the hash value of each secondary block; The first-level block hash module is used to combine the hash values ​​of all the second-level blocks of the first-level block with respect to the hash value of each first-level block of the desktop image to be transmitted, calculate the hash value of the combined data, and use the hash value of the combined data as the hash value of the image data of the first-level block; or, perform hash value calculation on the image data of the first-level block to obtain the hash value of the first-level block; The block cache identifier replacement module is used to match each first-level block in the first-level unrefreshed block set with the hash value of the first-level block in the first-level block cache list in the local cache pool to determine whether it is a hit; if the first-level block cache data structure in the first-level block cache list is hit, the desktop image data of the block is replaced with the block cache identifier of the first-level block; the block cache identifier is used to uniquely identify a block at both the cloud desktop server and the cloud desktop client; for each first-level block in the first-level refreshed block set, the hash value of the second-level block is used to match the second-level block cache list in the local cache pool to determine whether it is a hit; if the second-level block cache data structure in the second-level block cache list is hit, the desktop image data of the block is replaced with the block cache identifier of the second-level block; The compression transmission module is used to package and compress the desktop image data to be transmitted after completing the block cache identifier replacement process and then send it to the cloud desktop client.

7. The device according to claim 6, characterized in that The text interface judgment module judges whether the desktop image is a text interface by comparing the rate of change of the color change of the desktop image. When the rate of change is greater than a preset rate of change threshold, it is judged to be a text interface, otherwise it is judged to be a non-text interface; or a trained deep learning neural network model is used to judge whether the input desktop image is a text interface.

8. The device according to claim 6, characterized in that The secondary block segmentation module performs pixel-level line scanning on the image data of the primary block, and segments the area containing text and the blank area into a plurality of secondary blocks.

9. The device according to claim 6, characterized in that The device also includes: The aging processing module is used to add an aging time field to each primary block and secondary block to record the cache duration; each time a block is hit, the cache time field of the block is refreshed; and the aging cleaning timer is used to periodically determine whether the cache duration of each block exceeds the preset threshold. If so, the cache space of the corresponding block is released.

10. A cloud desktop image transmission device, characterized in that: The device is applied to the cloud desktop client, including: The decompression module is used to receive the compressed data packet of the text interface sent by the cloud desktop server and parse the compressed data packet; A primary block processing module, for directly reading the image data of the corresponding primary block from the local cache through the block cache identifier carried in the data packet when the hit type identifier of the block in the data packet indicates a hit of the primary block, wherein the primary block is a screen display area obtained by segmenting the cloud desktop based on a preset primary block size; and for reading the image data of the primary block from the data packet and caching the image data of the corresponding primary block in the local cache pool when the hit type identifier of the block in the data packet indicates a miss of the primary block and the secondary block; wherein the secondary block is a block obtained by scanning and segmenting the primary block; A secondary block processing module is used to parse the data structure of the secondary block included in the data packet and read the data structure of the secondary block in sequence when the hit type identifier of the block in the data packet indicates that the secondary block is hit; if the hit type field in the data structure of the secondary block indicates that the current secondary block is hit, the client directly searches and reads the image data of the corresponding secondary block in the local cache pool according to the block cache identifier in the data structure of the secondary block; if the hit type field in the data structure of the secondary block indicates that the current secondary block is not hit, the client reads the image data of the secondary block from the data structure of the current secondary block and caches it in the local cache pool; The block combination module is used to assemble the image data of all the first-level blocks read from the compressed data packet into complete desktop image data, and output it to the display driver of the client.

Images