A cloud desktop image processing method, system and related equipment

Abstract

The present invention discloses a cloud-based desktop image processing method, system, and related equipment. The method includes: dividing a current image into multiple image blocks; determining the unchanged and changed regions of the current image relative to the previous image based on the hash values ​​of each image block in the current image and the previous image frame; performing a motion search on the image blocks in the changed region in the previous image frame based on the hash values ​​of the image blocks in the changed region, determining the moved and newly added regions of the current image relative to the previous image frame, and determining the motion vectors of the image blocks in the moved region relative to the previous image frame; and determining the data to be transmitted for the current image based on the unchanged, moved, and newly added regions. The method can accurately analyze the unchanged and changed regions in the image and the types of the changed regions, and determine the data to be transmitted based on the analysis results to reduce the amount of data transmitted, increase the data transmission speed, and meet the real-time and low-latency requirements of cloud-based desktop image transmission.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a cloud desktop image processing method, system and related equipment. Background Art

[0002] Cloud desktops are a new model that uses cloud computing technology to replace traditional computers. A virtual computer host is created on a cloud server. Front-end devices typically use lightweight clients to connect to displays and input devices. After installing the client on the front-end device, it accesses the virtual computer host on the cloud server through a unique communication protocol for interactive operations, achieving the same user experience as a traditional computer host. Cloud applications are applications that interact between front-end devices and cloud servers. Applications run on the cloud server and are synchronized to the front-end device for display.

[0003] Using cloud desktops and cloud applications requires compressing and encoding the cloud desktop images on the cloud server virtual host and transmitting them to the local client on the front-end device for display. To improve image compression efficiency, the correlation between image frames can be exploited to analyze the content of each frame. Different compression methods are then applied to different image regions, reducing data transmission volume and bandwidth requirements.

[0004] There are two ways to perform image analysis in the prior art. One is to analyze image changes and motion information output by the operating system through screen capture, and the other is to perform image analysis using an open source computer vision framework (such as OpenCV). Summary of the Invention

[0005] The inventors of this application have discovered that the image analysis methods in the prior art cannot meet the low latency and real-time requirements of cloud desktops and cloud applications. Specifically, the use of operating system screen capture methods does not accurately analyze image motion and changes, and cannot accurately know the motion and changes of image frames, resulting in a large amount of redundancy, which increases the amount of data to be transmitted, cannot meet the requirements of fast data transmission, and the data transmission time is still relatively long. Using OpenCV for image analysis, although the analysis accuracy is improved to a certain extent, the analysis process is too time-consuming and cannot meet the low latency and real-time requirements of cloud desktops. In short, there is no better image analysis method in the prior art, which cannot accurately determine the unchanged area, changing area, motion situation, image type and other information between image frames, that is, it cannot analyze image change information with high precision and high accuracy. Therefore, it is impossible to quickly and accurately divide different types of image areas for targeted compression processing, and cannot meet the transmission latency requirements of cloud desktops and cloud applications.

[0006] In view of the above problems, the present invention is proposed to provide a cloud desktop image processing method, system and related devices that overcome the above problems or at least partially solve the above problems.

[0007] An embodiment of the present invention provides a cloud desktop image processing method, comprising:

[0008] Divide the current image into multiple image blocks;

[0009] Determine, based on the hash value of each image block in the current image and the hash value of each image block in the previous image, the unchanged area and the changed area of ​​the current image relative to the previous image;

[0010] Based on the hash values ​​of the image blocks in the changed region, a motion search is performed on the image blocks in the changed region in the previous frame image to determine the moved region of the current image relative to the previous frame image and the newly added region in the current image, as well as the motion vector of the image blocks in the moved region relative to the previous frame image;

[0011] Based on the unchanged area, moving area and newly added area of ​​the current image, the data that needs to be transmitted for the current image is determined.

[0012] In some optional embodiments, dividing the current image into a plurality of image blocks includes:

[0013] According to a preset image block size, the current image is divided into a plurality of image blocks of equal size; the image block size is determined according to the resolution and configuration information of the current image.

[0014] In some optional embodiments, determining the unchanged area and the changed area of ​​the current image relative to the previous image based on the hash value of each image block in the current image and the hash value of each image block in the previous image includes:

[0015] Calculate the hash value of each image block in the current image;

[0016] Compare the hash values ​​of each image block in the current image with the image block at the same position in the previous frame image;

[0017] Based on the image blocks with the same hash value, determine the unchanged area of ​​the current image relative to the previous frame image;

[0018] Based on image blocks with different hash values, the changed areas of the current image relative to the previous image are determined.

[0019] In some optional embodiments, based on hash values ​​of image blocks in the changed region, performing a motion search on the image blocks in the changed region in the previous image frame, determining a moved region of the current image relative to the previous image frame and a newly added region in the current image, and determining a motion vector of the image blocks in the moved region relative to the previous image frame include:

[0020] The image blocks in the changed area are used as the image blocks to be matched. For each image block to be matched, a motion search is performed in the previous frame image. If a matching image block with the same hash value is found in the previous frame image, the motion vector of the image block to be matched relative to the previous frame image is determined based on the position of the image block to be matched in the current image and the position of the matching image block in the previous frame image.

[0021] Based on the image blocks to be matched that have matching image blocks in the previous frame image, a moving area of ​​the current image relative to the previous frame image is obtained; and other areas in the changed area except the moving area are used as new areas.

[0022] In some optional embodiments, obtaining a moving area of ​​the current image relative to the previous frame image based on the image block to be matched having a matching image block in the previous frame image includes:

[0023] For a matching image block with a matching image block in the previous frame, if multiple adjacent image blocks have the same motion vector, the image blocks with the same motion vector are diffused outward from the matching image block to form an image block group with the same motion vector.

[0024] If the obtained image block group is unique, the obtained image block group is used as the moving area;

[0025] If the obtained image block group is not unique, the image block group with the largest area is used as the moving area.

[0026] In some optional embodiments, the method further includes: calculating a hash value of the image block using the following fast hash algorithm:

[0027] For each pixel coordinate y in the image, the pixel coordinate (x, x+N-1) corresponding to it is a pixel element meta, where N represents the size of the image block, which is N*N, and N is a positive integer;

[0028] Calculate the hash value meta_hash(x, x+N-1; y) of each meta according to the pixel values ​​of the N pixels included in each meta.

[0029] Perform hash calculation on the hash values ​​of N consecutive metas starting at pixel coordinate y to obtain the hash value of the image block starting at pixel coordinate (x, y), where the hash values ​​of N consecutive metas include meta_hash(x, x+N-1; y), meta_hash(x, x+N-1; y+1), ..., meta_hash(x, x+N-1; y, y+N-1).

[0030] In some optional embodiments, determining data that needs to be transmitted to the front-end device for the current image includes:

[0031] The motion vector of the image block in the moving area relative to the previous frame image and the content data of the image block in the non-moving area are used as the data that needs to be transmitted to the front-end device for the current image.

[0032] In some optional embodiments, after determining the moved area of ​​the current image relative to the previous frame image and the newly added area in the current image, the method further includes:

[0033] performing color analysis on the data blocks in the newly added area, and determining the content data type of each data block based on the color complexity, wherein the content data type includes at least one of text and images; accordingly, after determining the data to be transmitted for the current image, further comprising: selecting a compression algorithm based on the content data type of the image block, and compressing the content data using the selected compression algorithm; and / or

[0034] Correct the moving area and adjust it to a square or rectangular area.

[0035] An embodiment of the present invention also provides another cloud desktop image processing method, including:

[0036] Receive the current image data transmitted by the cloud server;

[0037] Parsing the received data to obtain the content data of the data block in the newly added area of ​​the current image and the motion vector of the image block in the moving area relative to the previous frame image;

[0038] Obtaining content data of image blocks in an unchanged area of ​​the current image from a previous frame of image, and obtaining content data of image blocks in a moving area from the previous frame of image based on motion vectors of image blocks in the moving area relative to the previous frame of image;

[0039] The current image is generated based on the parsed content data of the data blocks in the newly added area and the acquired content data of the image blocks in the moving area and the unchanged area.

[0040] In some optional embodiments, acquiring content data of the image block in the moving area from the previous frame image according to the motion vector of the image block in the moving area relative to the previous frame image includes:

[0041] The position of the image block in the moving area in the previous image is determined based on the position of the image block in the moving area in the current image and the motion vector relative to the previous image frame, and the content data of the corresponding image block is obtained based on the determined position.

[0042] An embodiment of the present invention further provides a cloud server, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned cloud desktop image processing method is implemented.

[0043] An embodiment of the present invention further provides a front-end device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned another cloud desktop image processing method when executing the program.

[0044] An embodiment of the present invention further provides a cloud desktop image transmission system, comprising: the aforementioned cloud server and the aforementioned front-end device;

[0045] The cloud server is configured to transmit the determined transmission data to the front-end device;

[0046] The front-end device is used to receive data provided by the cloud server and generate a current image based on the received data and data of a previous frame of image.

[0047] An embodiment of the present invention further provides a computer storage medium storing computer executable instructions. When the computer executable instructions are executed by a processor, the above-mentioned one cloud desktop image processing method or the above-mentioned another cloud desktop image processing method is implemented.

[0048] The beneficial effects of the above technical solutions provided by the embodiments of the present invention include at least:

[0049] The above-mentioned method provided by the embodiment of the present invention analyzes the image based on the hash value of each image block in the current image of the data to be transmitted and the hash value of each image block in the previous frame image, determines the unchanged area in the current image that has not changed and the changed area that has changed relative to the previous frame image, further analyzes the changed area to determine the moved area that has moved relative to the previous frame image and the newly added area in the current image. This method can quickly and accurately analyze the unchanged area, moved area, and newly added area in the image, and adopt different transmission processing methods for the content data of the unchanged area, moved area, and newly added area. For the unchanged area, the content data of the previous frame image can be directly used without data transmission. For the moving area, the motion vector relative to the previous frame image needs to be transmitted to determine which image block or blocks in the previous frame image have moved their content data. For the newly added area, the content data needs to be transmitted. This method can perform targeted data compression and transmission for different areas, minimize the data to be transmitted, reduce transmission delay, and meet the low-latency and high real-time transmission requirements of cloud desktops and cloud applications.

[0050] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0051] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0053] Figure 1 This is a flowchart of the cloud desktop image processing method in Example 1 of the present invention;

[0054] Figure 2 Schematic diagram of the principle of the cloud desktop image processing method in the first embodiment of the present invention;

[0055] Figure 3 This is one of the diagrams illustrating the principle of hash value calculation in the first embodiment of the present invention;

[0056] Figure 4 This is the second diagram illustrating the principle of hash value calculation in the first embodiment of the present invention;

[0057] Figure 5 This is a flowchart of a cloud desktop image processing method in Embodiment 2 of the present invention;

[0058] Figure 6 This is a schematic structural diagram of a cloud desktop image processing device according to an embodiment of the present invention;

[0059] Figure 7 This is a schematic structural diagram of another cloud desktop image processing device according to an embodiment of the present invention;

[0060] Figure 8 Schematic diagram of the structure of the cloud desktop image processing system in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0062] To address the existing problem of being unable to quickly and accurately divide different types of image regions for targeted transmission and processing during cloud desktop image transmission, an embodiment of the present invention provides a cloud desktop image processing method that can quickly and accurately analyze the unchanged regions, moving regions, and newly added regions in an image, and adopt different transmission processing methods for the content data of the unchanged regions, moving regions, and newly added regions, thereby meeting the low-latency and high-real-time transmission requirements of cloud desktops and cloud applications. This is described in detail below through specific embodiments.

[0063] Example 1

[0064] The first embodiment of the present invention provides a cloud desktop image processing method, the process of which is as follows: Figure 1 As shown, the following steps are included:

[0065] Step S101: Divide the current image into multiple image blocks.

[0066] For each frame of image data to be transmitted, the current image is divided into multiple image blocks of equal size according to a preset image block size; the image block size is determined according to the resolution and configuration information of the current image.

[0067] The cloud desktop image processing method provided in this embodiment has the following implementation principles: Figure 2 As shown, this step belongs to the preprocessing step of the current image, corresponding to Figure 2The block pre-processing step in the image processing process involves dividing the image into blocks. This can be done for each frame, dividing the entire image into several equal-sized blocks. The block size can be adjusted based on the resolution and configuration, or it can be set according to the actual needs of the cloud desktop or cloud application. For example, the block size can be set to 16*16 pixels, 32*32 pixels, and so on. The following examples will use 16*16 pixels as an example.

[0068] Step S102: Based on the hash value of each image block in the current image and the hash value of each image block in the previous image, determine the unchanged area and the changed area of ​​the current image relative to the previous image.

[0069] In this step, the hash value of each image block in the current image is calculated; the hash value of each image block in the current image is compared with the image block at the same position in the previous frame; the unchanged area of ​​the current image relative to the previous frame is determined based on the image blocks with the same hash value; and the changed area of ​​the current image relative to the previous frame is determined based on the image blocks with different hash values. In other words, the image blocks in the current image with the same hash value as the image block at the same position in the previous frame are determined to obtain the unchanged area; the image blocks in the current image with different hash values ​​as the image block at the same position in the previous frame are determined to obtain the changed area.

[0070] A fast hash algorithm can be used to calculate the hash value of each image block. For each frame of image, the hash value of each image block can be calculated after the image blocks are divided for subsequent use. Alternatively, the hash value of each image block in the current image and the previous frame of image to be compared can be calculated before comparison. Usually, the hash value of the image block in the previous image has been calculated before the data of the previous frame of image is transmitted. Therefore, when comparing, the hash value of each image block in the current image is calculated and the hash value of the image block in the previous frame of image that was calculated previously is obtained for comparison.

[0071] During the comparison, the hash value of the image block in the current image is compared with the hash value of the image block at the same position in the previous frame to determine whether the image block in the current image has changed. If the hash values ​​are the same, it is considered unchanged. If the hash values ​​are different, it is considered changed. This determines the unchanged and changed areas of the current image relative to the previous frame. The unchanged area includes the image blocks in the current image that have not changed compared to the previous frame, and the changed area includes the image blocks in the current image that have changed compared to the previous frame.

[0072] See also Figure 2The hash calculation and dirty detection parts compare the hash values ​​of each block in the current image with the block at the same position in the previous frame to determine whether there are any changes. Blocks with changes are recorded as changed (dirty) image blocks and form the changed region, while blocks without changes are recorded as unchanged (skip) image blocks and form the unchanged region. Motion estimation is performed on dirty image blocks, and skip image blocks are not processed.

[0073] Step S103: Based on the hash value of the image block in the changed area, a motion search is performed on the image block in the changed area in the previous frame image to determine the moving area of ​​the current image relative to the previous frame image and the newly added area in the current image, as well as the motion vector of the image block in the moving area relative to the previous frame image.

[0074] The image blocks in the changed area are used as image blocks to be matched. For each image block to be matched, a motion search is performed in the previous frame image. If a matching image block with the same hash value is found in the previous frame image, the motion vector of the image block to be matched relative to the previous frame image is determined based on the position of the image block to be matched in the current image and the position of the matching image block in the previous frame image. Based on the image blocks to be matched that have matching image blocks in the previous frame image, the moving area of ​​the current image relative to the previous frame image is obtained. The other areas in the changed area except the moving area are regarded as new areas.

[0075] When performing motion search in the previous frame image, the search can be performed row by row and column by column, that is, starting from the first row and first column until the last row and last column position that can form an image block is found. Taking an image block of N*N pixels as an example, where N is a positive integer, for each image block in the changed area, an image block with the same hash value is searched in the previous frame image, starting from the first row and first column, with each position point (x, y) as the starting point, to form an image block of the same size as the image block to be matched. For example, if the starting position is (x, y), then each image block includes pixels (x, x+N-1; y, y+N-1), that is, an N*N image block with (x, y) as the starting point. The image block to be matched in the changed area is matched one by one with each formed image block, and the image blocks with the same matched hash values ​​are the matching image blocks of the image block to be matched.

[0076] For an image block to be matched that has a matching image block in the previous frame image, if multiple adjacent image blocks have the same motion vector, the image blocks with the same motion vector are diffused outward from the image block to be matched as the center to form an image block group with the same motion vector; if the obtained image block group is unique, the obtained image block group is used as the moving area; if the obtained image block group is not unique, the image block group with the largest area is used as the moving area.

[0077] See also Figure 2 Motion estimation is used in the algorithm to perform motion search on dirty areas. For each dirty block in the current image frame, a block with the same hash value is matched in the previous image frame. The difference between the positions of these two blocks is recorded as a motion vector (MV). If the four adjacent blocks of the block have a unique matching block with the same motion vector, the block and the MV are used as seed blocks for diffusion search to find more blocks with the same MV. Blocks with the same MV form an image block group.

[0078] See also Figure 2 The motion vector filter (MV filter) in

[15] may obtain multiple image block groups with the same MV for a current image. However, due to image changes in cloud desktops and cloud applications, there can only be one motion direction at a time. Therefore, MV filtering is required. During filtering, the area of ​​the image block groups corresponding to each MV can be calculated, and the image block group with the largest area can be found. Its MV is used as the MV of the moving area in the current image, and all dirty blocks of this MV are recorded as image blocks in the moving area.

[0079] Optionally, the moving area is corrected to a square or rectangular shape. To facilitate processing at the application layer, the moving area consisting of all dirty blocks is corrected to a regular shape, such as a rectangle or square. The moving area only needs to transmit the motion vector. The client on the front-end device can generate the current image based on the previous frame image and the motion vector, without transmitting the specific image content data.

[0080] Step S104: determining data to be transmitted for the current image based on the unchanged area, the moving area, and the newly added area of ​​the current image.

[0081] Based on the determined unchanged area, moving area, and newly added area of ​​the current image, different transmission processing methods are used for processing. For the content data of the image blocks in the unchanged area, no data transmission is required. For the image blocks in the moving area, only the motion vector data can be transmitted. For the image blocks in the newly added area, the content data needs to be transmitted. Therefore, based on the unchanged area, moving area, and newly added area of ​​the current image, the motion vector of the image blocks in the moving area relative to the previous frame image and the content data of the image blocks in the non-moving area are used as the data that needs to be transmitted to the front-end device for the current image. Figure 2 The output part in .

[0082] In some optional embodiments, after determining the moving area of ​​the current image relative to the previous frame image and the newly added area in the current image, the method further includes: performing color analysis on the data blocks in the newly added area, and determining the content data type of each data block based on the color complexity, where the content data type includes at least one of text and image; see Figure 2 Correspondingly, after determining the data that needs to be transmitted for the current image, it also includes: selecting a compression algorithm according to the content data type of the image block, and compressing the content data using the selected compression algorithm.

[0083] After the cloud server determines the data that needs to be transmitted, it will encode the data when transmitting it. Encoding refers to the process of converting the original image into a bit video stream through compression technology. Using different compression algorithms for different areas such as text areas and picture areas can further improve compression efficiency.

[0084] In the above method, a fast hash algorithm can be used to calculate the hash value of each image block. Since hash calculation needs to be performed for each frame of image, the amount of calculation is very large, especially in the motion search step, it is necessary to search and match row by row and column by column, that is, the hash value calculation needs to be performed for each pixel block (x, x+N-1; y, y+N-1) in the previous frame of image, and the amount of calculation will be even greater. Therefore, the present invention provides a fast hash algorithm to reduce the overall calculation time to meet the real-time requirements.

[0085] During motion search, for an image block to be matched in the current image, it needs to be matched row by row and column by column in the previous frame. Therefore, multiple hash values ​​need to be calculated for each image block that can be divided into an image, or in other words, a hash value needs to be calculated for each image block that can be formed with each pixel (x, y) in the previous frame as the starting position.

[0086] See also Figure 3As shown in the figure, taking the image block size of 16*16 pixels and the vertical direction of the image block as an example, during motion search, a hash value (MB hash) must be calculated for each row in each image block. That is, if the pixel range of the image block is (x, x+15; y, y+15), then starting from the starting position (x, y) of the image block, 16 MB hashes need to be calculated.

[0087] MB_hash(x, x+15; y, y+15)=HASH{rgb(x, y), rgb(x+1, y),..., rgb(x+15, y); rgb(x, y+1), r gb(x+1,y+1),…,rgb(x+15,y+1);…;rgb(x,y+15),rgb(x+1,y+15),…,rgb(x+15,y+15)}

[0088] MB_hash(x, x+15, y+1, y+16)=HASH{rgb(x, y), rgb(x+1, y),..., rgb(x+15, y); rgb(x, y+1), rgb(x+1,y+1),…,rgb(x+15,y+1);…;rgb(x,y+15),rgb(x+1,y+15),…,rgb(x+15,y+15)}

[0089] MB_hash(x, x+15, y+2, y+17)=HASH{rgb(x, y), rgb(x+1, y),..., rgb(x+15, y); rgb(x, y+1), rgb(x+1,y+1),…,rgb(x+15,y+1);…;rgb(x,y+15),rgb(x+1,y+15),…,rgb(x+15,y+15)}

[0090] …

[0091] MB_hash(x, x+15, y+15, y+30)=HASH{rgb(x, y+15), rgb(x+1, y+15),…, rgb(x+15, y+15), rgb(x, y +16),rgb(x+1,y+16),…,rgb(x+15,y+16);…;rgb(x,y+30),rgb(x+1,y+30),…,rgb(x+15,y+30)}

[0092] The horizontal direction is similar, and each image block in the image needs to be calculated. Therefore, this method has a huge amount of calculation and is time-consuming. Using a fast hash algorithm can reduce the amount of calculation and greatly improve the calculation speed.

[0093] In the above method, the following fast hash algorithm can be used to calculate the hash value of each image block pre-divided in each frame image and each image block that may be used for matching during the motion search process:

[0094] For each pixel coordinate y in the image, the pixel coordinate (x, x+N-1) corresponding to the pixel coordinate is a pixel element meta, where N represents the size of the image block (N*N); N is a positive integer. Based on the pixel values ​​of the N pixels included in each meta, a hash value meta_hash(x, x+N-1; y) is calculated for each meta. The hash values ​​of N consecutive metas starting with pixel coordinate y are hashed to obtain the hash value of the image block starting with pixel coordinate (x, y). The hash values ​​of the N consecutive metas include meta_hash(x, x+N-1; y), meta_hash(x, x+N-1; y+1), ..., meta_hash(x, x+N-1; y, y+N-1). That is, for each image block, a hash value can be first calculated using each row of pixels as a pixel element, and then the hash values ​​of each row are hashed to obtain the hash value of the image block.

[0095] See also Figure 4 The principle of the fast hash algorithm shown in the figure is schematically illustrated. Taking an image block size of 16*16 pixels as an example, the (x, x+15) pixels corresponding to each y coordinate are first defined as a meta, meta_hash is calculated for each meta, and then the hash calculation is performed on 16 consecutive meta_hashes in the vertical direction to obtain MB_hash.

[0096] meta_hash(x,x+15;y)=HASH{rgb(x,y),rgb(x+1,y),...,rgb(x+15,y)}

[0097] meta_hash(x, x+15; y+1)=HASH{rgb(x, y+1), rgb(x+1, y+1),..., rgb(x+15, y+1)}

[0098] meta_hash(x, x+15, y+2)=HASH{rgb(x, y+2), rgb(x+1, y+2),..., rgb(x+15, y+2)}

[0099] …

[0100] meta_hash(x, x+15; y+30)=HASH{rgb(x, y+30), rgb(x+1, y+30),..., rgb(x+15, y+30)}

[0101] Then calculate MB_hash:

[0102] MB_hash(x, x+15, y, y+15)=HASH{meta_hash(x, x+15, y), meta_hash(x, x+15, y+1)..., meta_hash(x, x+15, y+15)}

[0103] MB_hash(x, x+15, y+1, y+16)=HASH{meta_hash(x, x+15; y+1), meta_hash(x, x+15; y+2)..., meta_hash(x, x+15; y+16)}

[0104] …

[0105] MB_hash(x, x+15, y+15, y+30)=HASH{meta_hash(x, x+15; y+15), meta_hash(x, x+15; y+16)..., meta_hash(x, x+15; y+30)}

[0106] The aforementioned fast hashing algorithm can use SSE / AVX instructions to accelerate the hashing algorithm and multi-threaded parallel computing for each meta-hash calculation, further significantly improving computing performance. SSE / AVX are hardware acceleration instructions provided by computer systems.

[0107] The above-mentioned method of this embodiment can be implemented in a cloud server, analyzing the image data to be transmitted and then transmitting the image data based on the analysis results. The method analyzes the image based on the hash values ​​of the image blocks in the current image of the data to be transmitted, and determines the unchanged areas in the current image that have not changed and the changed areas that have changed relative to the previous frame. The changed areas are further analyzed to determine the moved areas that have moved relative to the previous frame and the newly added areas in the current image. The method can quickly and accurately analyze the unchanged areas, moved areas, and newly added areas in the image, and adopt different transmission processing methods for the content data of the unchanged areas, moved areas, and newly added areas. For the unchanged areas, the content data of the previous frame can be directly used without data transmission. For the moved areas, the motion vector relative to the previous frame is required to be transmitted to determine which image block or blocks in the previous frame have moved their content data. For the newly added areas, the content data needs to be transmitted. The method can perform targeted data compression and transmission for different areas, minimizing the data required for transmission and reducing transmission latency, thereby meeting the low-latency and high-real-time transmission requirements of cloud desktops and cloud applications. Data processing speed and efficiency can be further improved and latency can be reduced through fast hashing algorithms, color analysis, and the use of different compression algorithms.

[0108] Example 2

[0109] The second embodiment of the present invention provides a cloud desktop image processing method, which is implemented on the front-end device and cooperates with the method provided in the first embodiment to receive and process the transmitted image data so that the cloud desktop and cloud applications on the cloud device can quickly and correctly display the cloud desktop image. The method flow is as follows: Figure 2 As shown, the following steps are included:

[0110] Step S201: Receive current image data transmitted from a cloud server.

[0111] The data transmitted by the cloud server mainly includes the content data of the data blocks in the newly added area of ​​the current image and the motion vectors of the image blocks in the moving area relative to the previous frame image. Optionally, the unchanged area can be specified, that is, which blocks are image blocks in the unchanged area. Of course, it can also be not specified. By default, the image blocks that do not transmit any data are image blocks in the unchanged area.

[0112] Step S202: parse the received data to obtain the content data of the data blocks in the newly added area of ​​the current image and the motion vectors of the image blocks in the moving area relative to the previous frame of image.

[0113] Cloud servers encode data when transmitting it, and front-end devices decode it after receiving it. Encoding is the process of converting raw images into bit streams through compression technology, while decoding is the reverse process.

[0114] Step S203: obtaining content data of the image blocks in the moving area from the previous frame image according to the motion vectors of the image blocks in the moving area relative to the previous frame image.

[0115] In this step, the position of the image block in the moving region in the previous image frame is determined based on the position of the image block in the moving region in the current image and the motion vector relative to the previous image frame, and the content data of the corresponding image block is obtained based on the determined position. In other words, for the moving region, the position of the image block in the previous image frame can be determined based on the motion vector and the position of the image block in the moving region in the current image frame, and the content data of the corresponding position in the previous image frame can be obtained.

[0116] Step S204: obtaining content data of the image block in the unchanged area of ​​the current image from the previous frame image.

[0117] For the unchanged area, the position of the image block in the current image is the same as its position in the previous frame image, and the content data of the corresponding position in the previous frame image can be used.

[0118] The above steps S203 and S204 can be executed at the same time without any particular order.

[0119] Step S205: generating a current image based on the parsed content data of the data blocks in the newly added area and the obtained content data of the image blocks in the moving area and the unchanged area.

[0120] Based on the position of the image block, the content data of the image block is rendered to generate the current image. Rendering refers to the process of generating and drawing an image through graphics processing.

[0121] Based on the same inventive concept, the embodiment of the present invention further provides a cloud desktop image processing device, which can be set in a cloud server. The structure of the device is as follows: Figure 6 As shown, including:

[0122] A division module 11 is used to divide the current image into multiple image blocks;

[0123] A calculation and comparison module 12 is used to determine the unchanged area and the changed area of ​​the current image relative to the previous frame image based on the hash value of the image block in the current image;

[0124] a motion estimation module 13 for performing a motion search for the image blocks in the changed region in the previous image frame based on the hash value of each image block in the changed region and the hash value of each image block in the previous image frame, determining a moved region of the current image relative to the previous image frame and a newly added region in the current image, and determining a motion vector of the image blocks in the moved region relative to the previous image frame;

[0125] The data processing module 14 is configured to determine the data that needs to be transmitted for the current image based on the unchanged area, the moving area, and the newly added area of ​​the current image.

[0126] Based on the same inventive concept, the embodiment of the present invention also provides another cloud desktop image processing device, which can be set in the front-end device. The structure of the device is as follows: Figure 7 Shown, including:

[0127] The data receiving module 21 is used to receive the data of the current image transmitted by the cloud server;

[0128] A data parsing module 22 is configured to parse the received data to obtain the content of the data blocks in the newly added area of ​​the current image and the motion vectors of the image blocks in the moving area relative to the previous frame of image;

[0129] a data acquisition module 23 configured to acquire content data of image blocks in an unchanged region of a current image from a previous frame of image, and to acquire content data of image blocks in a moving region from a previous frame of image based on motion vectors of image blocks in the moving region relative to the previous frame of image;

[0130] The image generation module 24 is configured to generate a current image based on the parsed content data of the data blocks in the newly added area and the acquired content data of the image blocks in the moving area and the unchanged area.

[0131] Based on the same inventive concept, an embodiment of the present invention further provides a cloud desktop image transmission system, comprising a cloud server 1 and a front-end device 2. There may be multiple cloud servers 1 and front-end devices 2, and the cloud server may adopt a server cluster and allow a large number of front-end devices to access.

[0132] In some optional embodiments, the cloud server 1 may include: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement Figure 1 The cloud desktop image processing method.

[0133] In some optional embodiments, the front-end device 2 may include: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement Figure 5The cloud desktop image processing method.

[0134] Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and when the computer executable instructions are executed by a processor, Figure 1 The cloud desktop image processing method or Figure 5 The cloud desktop image processing method.

[0135] Regarding the apparatus and system in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiment of the method and will not be elaborated here.

[0136] The above-mentioned method, system and device of the embodiment of the present invention divides each frame of image into blocks, performs image analysis based on the blocks, and calculates the hash value of each block using a fast hash algorithm. Then, by comparing the hashes, the areas with changes and areas without changes are found, and the areas with changes are further analyzed to try to search for motion vectors of the moving areas. Content analysis is performed on the non-moving changed areas (i.e., newly added areas) to distinguish between image areas and text areas. Block-based analysis can obtain highly accurate and high-precision image change information. The block size can be flexibly configured, and the image change status, motion status, content information, etc. can be obtained in a single operation, saving computing resources. The process uses each frame of image captured by the cloud desktop / cloud application as input and uses the analysis results of the current image frame as output. The analysis results include: unchanged areas and changed areas relative to the previous frame of image, moving areas and motion vectors in the changed areas, and newly added areas in the changed areas and content analysis results of the newly added areas. The content analysis results are used to distinguish between image areas and text areas.

[0137] The above method is a fast algorithm for image analysis. This algorithm can be used to accurately analyze changes in image frames, obtain good time-consuming performance, and reduce delays in data transmission. Delay refers to the total time it takes in a cloud desktop or cloud application system from the time a user operates or enters a command on the front-end device to the time the corresponding desktop image is received.

[0138] Unless otherwise specifically stated, terms such as process, calculate, compute, determine, display, and the like may refer to the actions and / or processes of one or more processing or computing systems, or similar devices, that manipulate and convert data represented as physical (e.g., electronic) quantities within registers or memories of a processing system into other data similarly represented as physical quantities within the memories, registers, or other such information storage, transmission, or display devices of the processing system. Information and signals may be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, light fields or particles, or any combination thereof.

[0139] It should be understood that the specific order or hierarchy of steps in the disclosed processes is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0140] In the foregoing detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that embodiments of the claimed subject matter require more features than are expressly recited in each claim. On the contrary, as reflected in the appended claims, the invention comprises less than all the features of any individual disclosed embodiment. The appended claims are therefore hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0141] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein may be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps described above are generally described around their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. A skilled person may implement the described functions in an adaptable manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of this disclosure.

[0142] The steps of the methods or algorithms described in conjunction with the embodiments herein may be directly embodied as hardware, software modules executed by a processor, or a combination thereof. The software module may be located in a RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and storage medium may also be present in a user terminal as discrete components.

[0143] For software implementation, the techniques described in this application can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or external to the processor. In the latter case, it is communicatively coupled to the processor via various means, which are well known in the art.

[0144] The foregoing description includes examples of one or more embodiments. Of course, it is not possible to describe all possible combinations of components or methods for the purposes of describing the above embodiments, but one of ordinary skill in the art will recognize that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to encompass all such changes, modifications and variations that fall within the scope of the appended claims. Furthermore, to the extent the term "comprising" is used in the specification or claims, the term is intended to be encompassed in a manner similar to the term "including," as explained in terms of "including," used as a transitional word in the claims. Furthermore, any use of the term "or" in the specification of the claims is intended to mean a "non-exclusive or."

Claims

1. A cloud desktop image processing method, characterized in that: include: Divide the current image into multiple image blocks; Determine, based on the hash value of each image block in the current image and the hash value of each image block in the previous image, the unchanged area and the changed area of ​​the current image relative to the previous image; The image blocks in the changed area are used as the image blocks to be matched. For each image block to be matched, a motion search is performed in the previous frame image. If a matching image block with the same hash value is found in the previous frame image, the motion vector of the image block to be matched relative to the previous frame image is determined based on the position of the image block to be matched in the current image and the position of the matching image block in the previous frame image. Based on the image blocks to be matched that have matching image blocks in the previous frame image, a moving area of ​​the current image relative to the previous frame image is obtained; the other areas in the changed area except the moving area are regarded as newly added areas; For a matching image block with a matching image block in the previous frame, if multiple adjacent image blocks have the same motion vector, the image blocks with the same motion vector are diffused outward from the matching image block to form an image block group with the same motion vector. If the obtained image block group is unique, the obtained image block group is used as the moving area; If the obtained image block group is not unique, the image block group with the largest area is used as the moving area; Based on the unchanged area, moving area and newly added area of ​​the current image, the data that needs to be transmitted for the current image is determined.

2. The method according to claim 1, wherein The step of dividing the current image into a plurality of image blocks comprises: According to a preset image block size, the current image is divided into a plurality of image blocks of equal size; the image block size is determined according to the resolution and configuration information of the current image.

3. The method according to claim 1, wherein The determining, based on the hash value of each image block in the current image and the hash value of each image block in the previous frame image, the unchanged area and the changed area of ​​the current image relative to the previous frame image includes: Calculate the hash value of each image block in the current image; Compare the hash values ​​of each image block in the current image with the image block at the same position in the previous frame image; Based on the image blocks with the same hash value, determine the unchanged area of ​​the current image relative to the previous frame image; Based on image blocks with different hash values, the changed areas of the current image relative to the previous image are determined.

4. The method according to claim 1, wherein The method further includes: calculating a hash value of the image block using the following fast hash algorithm: For each pixel coordinate y in the image, the pixel coordinate (x, x+N-1) corresponding to it is a pixel element meta, where N represents the size of the image block N. N, N is a positive integer; Calculate the hash value meta_hash(x, x+N-1; y) of each meta according to the pixel values ​​of the N pixels included in each meta. Hash the hash values ​​of N consecutive meta values ​​starting at pixel coordinate y to obtain the hash value of the image block starting at pixel coordinate (x, y), where the hash values ​​of N consecutive meta values ​​include meta_hash(x, x+N-1; y), meta_hash(x, x+N-1; y+1), ..., meta_hash(x, x+N-1; y, y+N-1).

5. The method according to any one of claims 1 to 4, characterized in that: The determining of data that needs to be transmitted to the front-end device for the current image includes: The motion vector of the image block in the moving area relative to the previous frame image and the content data of the image block in the non-moving area are used as the data that needs to be transmitted to the front-end device for the current image.

6. The method according to any one of claims 1 to 4, characterized in that: After determining the moving area of ​​the current image relative to the previous frame image and the newly added area in the current image, the method further includes: performing color analysis on the data blocks in the newly added area, and determining the content data type of each data block based on the color complexity, wherein the content data type includes at least one of text and images; accordingly, after determining the data to be transmitted for the current image, further comprising: selecting a compression algorithm based on the content data type of the image block, and compressing the content data using the selected compression algorithm; and / or Correct the moving area and adjust it to a square or rectangular area.

7. A cloud desktop image processing method, characterized in that: include: Receive the current image data transmitted by the cloud server; Parsing the received data to obtain the content data of the data block in the newly added area of ​​the current image and the motion vector of the image block in the moving area relative to the previous frame image; Obtaining content data of image blocks in an unchanged region of a current image from a previous frame of image, and obtaining content data of image blocks in a moving region from the previous frame of image based on motion vectors of image blocks in the moving region relative to the previous frame of image; the moving region is determined by the cloud server through the following steps: The image blocks in the changed area are used as the image blocks to be matched. For each image block to be matched, a motion search is performed in the previous frame image. If a matching image block with the same hash value is found in the previous frame image, the motion vector of the image block to be matched relative to the previous frame image is determined based on the position of the image block to be matched in the current image and the position of the matching image block in the previous frame image. Based on the image blocks to be matched that have matching image blocks in the previous frame image, a moving area of ​​the current image relative to the previous frame image is obtained; the other areas in the changed area except the moving area are regarded as newly added areas; For a matching image block with a matching image block in the previous frame, if multiple adjacent image blocks have the same motion vector, the image blocks with the same motion vector are diffused outward from the matching image block to form an image block group with the same motion vector. If the obtained image block group is unique, the obtained image block group is used as the moving area; If the obtained image block group is not unique, the image block group with the largest area is used as the moving area; The current image is generated based on the parsed content data of the data blocks in the newly added area and the acquired content data of the image blocks in the moving area and the unchanged area.

8. The method according to claim 7, wherein The acquiring content data of the image block in the moving area from the previous frame image according to the motion vector of the image block in the moving area relative to the previous frame image includes: The position of the image block in the moving area in the previous image is determined based on the position of the image block in the moving area in the current image and the motion vector relative to the previous image frame, and the content data of the corresponding image block is obtained based on the determined position.

9. A cloud server, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the cloud desktop image processing method according to any one of claims 1 to 6 is implemented.

10. A front-end device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the cloud desktop image processing method according to any one of claims 7 to 8 is implemented.

11. A cloud desktop image transmission system, characterized in that: include: The cloud server according to claim 9 and the front-end device according to claim 10; The cloud server is configured to transmit the determined transmission data to the front-end device; The front-end device is used to receive data provided by the cloud server and generate a current image based on the received data and data of a previous frame of image.

12. A computer storage medium, characterized in that The computer storage medium stores computer-executable instructions, which, when executed by a processor, implement the cloud desktop image processing method described in any one of claims 1-6 or the cloud desktop image processing method described in any one of claims 7-8.

Images