Image processing method, apparatus, device, medium and product

Abstract

This application relates to the field of data processing technology, and discloses an image processing method, apparatus, device, medium, and product. The method includes: acquiring a first interface image to be processed and a second interface image preceding the first interface image, when dragging does not display window content; dividing the first interface image and the second interface image into multiple image blocks; identifying the first image block in the first interface image that has changed; when a window dragging operation exists, encoding the first image block at the corresponding position according to the second image block in the second interface image to generate encoded image blocks; and sending the encoded image blocks corresponding to each first image block to a client. This application utilizes the characteristic that most pixels are the same between the first image block and the second image block in the previous second interface image, so that the generated encoded image block has a smaller data volume than the first image block, which can reduce bandwidth consumption in window dragging scenarios.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to image processing methods, apparatus, devices, media, and products. Background Technology

[0002] Cloud desktop technology, with its flexibility, scalability, and security, is gradually becoming an important tool for enterprise digital transformation. Cloud desktop users are mostly in office scenarios, where window dragging and dropping are frequent operations. These dragging operations consume significant bandwidth, necessitating optimization to reduce bandwidth consumption and improve the user experience in weak network environments. Summary of the Invention

[0003] In view of this, this application provides an image processing method, apparatus, device, medium, and product to solve the problem of high bandwidth consumption in cloud desktop drag-and-drop operations.

[0004] In a first aspect, this application provides an image processing method applied to a server, the method comprising: Obtain the first interface image to be transmitted to the client, and the second interface image that has already been transmitted to the client; The first interface image and the second interface image are divided into multiple image blocks; Identify the first image block in the first interface image that has changed; the first image block is not exactly the same as the second image block at the corresponding position in the second interface image. When a window dragging operation is detected, the first image block is encoded according to the second image block at the corresponding position in the second interface image to generate an encoded image block; The encoded image blocks corresponding to each of the first image blocks are sent to the client, instructing the client to decode the first interface image based on the encoded image blocks.

[0005] Secondly, this application provides an image processing method applied to a client, the method comprising: Obtain encoded information sent by the server, the encoded information including encoded image blocks; the encoded image blocks are generated based on the image processing method of the first aspect described above or any corresponding embodiment thereof; Based on the second image block in the second interface image that corresponds to the position of the encoded image block, the encoded image block is decoded to obtain the first image block; Replace the second image block in the second interface image with the first image block at the corresponding position to generate the current first interface image.

[0006] Thirdly, this application provides an image processing apparatus applied to a server, the apparatus comprising: The first acquisition module is used to acquire the first interface image to be transmitted to the client and the second interface image that has been transmitted to the client. A segmentation module is used to divide the first interface image and the second interface image into multiple image blocks; The processing module is used to determine a first image block that has changed in the first interface image; the first image block is not completely identical to a second image block at a corresponding position in the second interface image; when a window drag operation is detected to be triggered, the first image block is encoded according to the second image block at the corresponding position in the second interface image to generate an encoded image block; The sending module is used to send the encoded image blocks corresponding to each of the first image blocks to the client, instructing the client to decode the first interface image according to the encoded image blocks.

[0007] Fourthly, this application provides an image processing apparatus for use in a client, the apparatus comprising: The second acquisition module is used to acquire encoded information sent by the server, the encoded information including encoded image blocks; the encoded image blocks are generated based on the image processing method of the first aspect above or any corresponding embodiment; The decoding module is used to decode the encoded image block according to the second image block in the second interface image that corresponds to the position of the encoded image block, to obtain the first image block; The generation module is used to replace the second image block in the second interface image with the first image block at the corresponding position to generate the current first interface image.

[0008] Fifthly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the image processing method described in the first aspect, the second aspect, or any corresponding embodiment thereof.

[0009] In a sixth aspect, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the image processing method described in the first aspect, the second aspect, or any corresponding embodiment thereof.

[0010] In a seventh aspect, this application provides a computer program product, including computer instructions for causing a computer to execute the image processing method described in the first aspect, the second aspect, or any corresponding embodiment thereof.

[0011] The image processing method provided in this application, when the interface changes due to window dragging, utilizes the characteristic that most pixels of the first image block in the first interface image to be processed are the same as those of the second image block at the corresponding position in the previous second interface image. The first image block is encoded based on the second image block, so that the generated encoded image block has a smaller data volume than the first image block. The server can then transmit the encoded image block with a smaller data volume to the client. Under the premise that the client can restore the first interface image, the amount of data transmitted between the two can be effectively reduced, which can reduce the bandwidth consumption in window dragging scenarios and improve the user experience in weak network environments. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application; Figure 2 This is a schematic flowchart of a first embodiment of an image processing method according to this application. Figure 3 This is a schematic diagram illustrating a scenario where dragging does not display window content according to an embodiment of this application; Figure 4 This is a schematic diagram of a second process of an image processing method according to an embodiment of this application; Figure 5 This is a schematic diagram of a second interface image according to an embodiment of this application; Figure 6 This is a schematic diagram of a first interface image according to an embodiment of this application; Figure 7 This is a schematic diagram of a mask image according to an embodiment of this application; Figure 8 This is a schematic diagram of a third interface image according to an embodiment of this application; Figure 9 This is another schematic diagram of a mask image according to an embodiment of this application; Figure 10 This is a schematic diagram of dividing a first image block in a first interface image according to an embodiment of this application; Figure 11 This is a comparison diagram of two encoding results for the first image block according to an embodiment of this application; Figure 12 This is a schematic flowchart of a client-side image processing method according to an embodiment of this application; Figure 13 This is a detailed flowchart illustrating an image processing method according to an embodiment of this application; Figure 14 This is a structural block diagram of a server-side image processing apparatus according to an embodiment of this application; Figure 15 This is a structural block diagram of a client-side image processing apparatus according to an embodiment of this application; Figure 16 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0016] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0017] Cloud desktop protocols are technologies that use image encoding, data compression, and network optimization to efficiently transmit remote desktop images and operation commands to terminal devices. They support low-latency interaction, adapt to different bandwidth environments, and enable cross-platform access, centralized resource management, and data security. They are widely used in virtual office and cloud gaming scenarios.

[0018] To address the issue of high bandwidth consumption in driving scenarios, the main bandwidth optimization solutions include: Solution 1. Macroblock-based deduplication algorithm for consecutive frames: This algorithm divides the changing region image into multiple macroblocks based on a specific macroblock size. Then, it checks whether the content of macroblocks at the same position in the previous frame has changed on a macroblock-by-macroblock basis. Only macroblocks that have changed are encoded, while macroblocks that have not changed are not encoded, in order to reduce bandwidth.

[0019] Option 2. Full-screen streaming algorithm: For dragging operations such as dragging windows, if a large amount of bandwidth consumption is detected, full-screen streaming will be performed. This involves capturing images of the entire desktop, compressing them using video encoding algorithms, and then sending them to reduce bandwidth usage.

[0020] While both of these solutions can reduce bandwidth, they each have their own drawbacks. Solution 1, based on the large granularity of macroblocks, cannot remove unchanged regions within changing macroblocks. Solution 2, while reducing network bandwidth, will degrade image quality and increase client resource consumption during full-screen streaming.

[0021] As one optional application scenario in the embodiments of this application, such as Figure 1 As shown, application 101 is installed in terminal device 110, and user 130 can interact with application 101 through terminal device 110 and / or access device of terminal device 110.

[0022] For example, application 101 could be an application capable of cloud interaction, such as a cloud desktop client. Figure 1 In the application scenario shown, if application 101 is active, the terminal device 110 can display the interface 102 of application 101. The interface 102 may include various pages that application 101 can provide, such as interactive pages, settings pages, query pages, etc.

[0023] In some embodiments, terminal device 110 is communicatively connected to server 120 to provide services to application 101. Terminal device 110 may be a mobile terminal, fixed terminal, or portable terminal, including but not limited to mobile phones, desktop computers, laptop computers, multimedia tablets, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 may also support any type of interface, and server 120 may be various types of computing systems or servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, and computing devices in cloud environments.

[0024] It should be noted that, Figure 1 This is merely an example of an application scenario and does not limit the scope of protection of this application.

[0025] The image processing method provided in this embodiment allows the server to transmit smaller image blocks to the client. While ensuring that the client can restore the original interface image, it can effectively reduce the amount of data transmitted between the two, reduce bandwidth consumption in window dragging scenarios, and improve the user experience in weak network environments.

[0026] According to an embodiment of this application, an image processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] This embodiment provides an image processing method that can be used with the aforementioned server, such as a server providing cloud desktop functionality. Figure 2 This is a flowchart of an image processing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps.

[0028] Step S201: Obtain the first interface image to be transmitted to the client and the second interface image that has been transmitted to the client.

[0029] During the interaction between the server and the user's client, the server needs to transmit relevant information about the current interface to the client in real time so that the corresponding interface on the server side can be displayed on the client in real time. In this embodiment, the server processes the interface images that need to be transmitted to the client to reduce the amount of data transmission and reduce bandwidth consumption.

[0030] Specifically, at the current moment, the interface image to be transmitted to the client can be acquired, i.e., the first interface image; for example, the first interface image can be the interface image of the current frame. Furthermore, since the displayed interface images are multi-frame, i.e., there is an interface image located before the first interface image, and this interface image has already been transmitted to the client, i.e., the second interface image, in other words, the acquisition time of the second interface image is earlier than the acquisition time of the first interface image.

[0031] To reduce the amount of subsequent processing, the second interface image is time-adjacent to the first interface image. For example, if the first interface image is the t-th frame, then the second interface image is the (t-1)-th frame, which is the frame preceding the first interface image.

[0032] Among the servers that support cloud desktop functionality, dragging can hide window content. The corresponding settings can be completed by enabling or disabling the "drag to hide window content" or "drag to show window content" function.

[0033] When the window content is not displayed when dragging, the dragged window will not move when the user drags the window (at which point a window dragging operation occurs), but will only display the position corresponding to the window boundary after dragging.

[0034] Figure 3 This illustrates a scenario where dragging does not display the window content. For example... Figure 3 As shown, dragging is set to hide the window content. On the current desktop 300, window 301 is displayed. If the user drags window 301, specifically from the upper left to the lower right (e.g., ...), the window content is not displayed. Figure 3 As shown by the arrow corresponding to the mouse cursor, in this desktop 300, the position of window 301 remains unchanged; only the window boundary 302 after the window 301 has been moved is additionally displayed. It can be understood that the window boundary 302 has the same size as the boundary of window 301, only the position is different, and the specific position of the window boundary 302 is related to the position of the drag operation.

[0035] In a cloud desktop scenario, it is possible to set the window not to be displayed when dragging, so that when the window is dragged, only the window boundary changes in the interface image displayed on the server side. That is, the difference between two interface images (such as the first interface image and the second interface image) in adjacent frames is small. The method provided in this embodiment is more conducive to reducing the amount of data transmission between the server and the client.

[0036] It should be noted that the server executing this method can be a cloud server. If the user deploys a local terminal device (such as a personal computer) as a cloud desktop server, then that local terminal device can also serve as the server for executing this method. Furthermore, local terminal devices such as personal computers also support settings that prevent window content from being displayed when dragging.

[0037] Step S202: Divide the first interface image and the second interface image into multiple image blocks.

[0038] Both the first and second interface images will be divided into smaller image blocks. Furthermore, the first and second interface images can be divided into multiple image blocks using the same partitioning method.

[0039] For example, both the first and second interface images can be divided into multiple image blocks according to the BLOCK_WIDTH × BLOCK_HEIGHT division standard; BLOCK_WIDTH and BLOCK_HEIGHT are the width and height of the image block, respectively. For example, an image block can be 16×16. Specifically, this image block can be a macroblock.

[0040] Step S203: Determine the first image block that has changed in the first interface image; the first image block is not completely identical to the second image block at the corresponding position in the second interface image.

[0041] For each image block in the first interface image, it can be compared with the corresponding image block in the second interface image to determine if they are the same. If they are the same, it means that the image block has not changed and does not need to be sent to the client. If they are not completely the same (for example, one or more pixels are different), it means that the image block has changed and needs to be processed.

[0042] For ease of description, the image block that changes in the first interface image is called the first image block; for the first image block, it corresponds to the position of an image block in the second interface image, and the image block in the second interface image is called the second image block, that is, the first image block and the second image block are in one-to-one correspondence.

[0043] Step S204: When a window dragging operation is detected to be triggered, the first image block is encoded according to the second image block at the corresponding position in the second interface image to generate an encoded image block.

[0044] In this embodiment, if a changed first image block exists, it indicates that the server's interface has changed. At this time, it can be determined whether a window dragging operation is currently occurring, i.e., whether the user is changing the window's position. If no window dragging operation is detected, it means that the interface change is caused by other operations, such as the user editing information or watching a video. In this case, it can be processed in other ways, such as directly sending the changed first image block to the client.

[0045] If a window dragging operation is detected, the processing method provided in this embodiment can be executed to minimize the amount of data transmitted to the client.

[0046] Specifically, if a window dragging operation is currently in progress, since the window content is not displayed during dragging, the change area between the first and second interface images only involves the window boundaries. That is, most of the content in the first and second interface images is the same. Therefore, in this embodiment, the first image block at the corresponding position is encoded using the second image block in the second interface image. The encoded image block is then used as the image block to be sent to the client, i.e., the encoded image block. It can be understood that this encoded image block and the first image block can have a one-to-one correspondence.

[0047] In some optional implementations, image blocks can be encoded through XOR processing and compression processing. Specifically, step S204, "when a window dragging operation is detected to be triggered, the first image block is encoded according to the second image block at the corresponding position in the second interface image to generate an encoded image block," may include steps a1 to a2.

[0048] Step a1: Perform an XOR operation on the first image block and the corresponding second image block to generate an intermediate image block.

[0049] Step a2: Compress the intermediate image block to generate the coded image block corresponding to the first image block.

[0050] In this embodiment, for the first image block that has changed, an XOR operation can be performed between it and the corresponding second image block. Utilizing the characteristic that the XOR result for identical data is 0, all identical data (e.g., pixels that are the same in both the first and second image blocks) can be set to 0, while only data that are different can be set to 1. Using XOR operations is not only simple to calculate and ensures processing efficiency, but also avoids problems such as value inflation caused by subtraction, resulting in high reliability.

[0051] Furthermore, as shown above, the only difference between the first and second image blocks is their window boundaries, meaning that most pixels are the same across the two blocks. Therefore, most of the data in the XOR operation result is 0 and is continuous. When compressing the intermediate image block containing the XOR operation result, it has a better compression effect on multiple consecutive 0s, i.e., a higher compression ratio. This makes the data size of the compressed encoded image block much smaller than that of the original first image block, thereby effectively reducing the amount of data transmitted to the client.

[0052] Step S205: Send the encoded image blocks corresponding to each first image block to the client, instructing the client to decode the first interface image according to the encoded image blocks.

[0053] After generating the encoded image blocks corresponding to each first image block, these encoded image blocks can be sent to the client. The client then decodes the corresponding encoded image blocks based on the previous second interface image to reconstruct the current first interface image. In essence, this is equivalent to transmitting the first interface image to the client.

[0054] The server can combine various coded image blocks to generate a complete interface image and send it to the client. If the coded image blocks are generated using XOR and compression, after obtaining the XOR results of the first image blocks, an interface image containing the XOR results can be generated first. Data in all areas of this interface image except for the first image blocks can be set to zero. Then, compression processing is performed on the interface image to ensure a high compression ratio.

[0055] The image processing method provided in this embodiment, when the interface changes due to window dragging, utilizes the characteristic that most pixels of the first image block in the first interface image to be processed are the same as those of the second image block at the corresponding position in the previous second interface image. The first image block is encoded based on the second image block, so that the generated encoded image block has a smaller data volume than the first image block. The server can then transmit the encoded image block with a smaller data volume to the client. Under the premise that the client can restore the first interface image, the amount of data transmitted between the two can be effectively reduced, which can reduce the bandwidth consumption in window dragging scenarios and improve the user experience in weak network environments.

[0056] This embodiment provides an image processing method that can be used with the aforementioned server, such as a server providing cloud desktop functionality. Figure 4 This is a flowchart of an image processing method according to an embodiment of this application, such as... Figure 4 As shown, the process includes the following steps.

[0057] Step S401: With the setting that dragging does not display window content, obtain the first interface image to be transmitted to the client and the second interface image that has been transmitted to the client.

[0058] Please see details Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0059] Among them, with Figure 3 Taking the interface shown as an example, Figure 5 A schematic diagram of a second interface image is shown, for example, the interface image at frame t-1; Figure 6 This diagram illustrates a first interface image, specifically the interface image at frame t. Figure 5 and Figure 6 As shown, the user drags the window to the lower right: the second interface image 500 includes a window boundary 501, which represents the position of the window boundary in the (t-1)th frame; similarly, the first interface image 600 includes a window boundary 601, which represents the position of the window boundary in the tth frame.

[0060] To easily illustrate the difference between the two images, Figure 6 The window boundary 501 and the corresponding mouse cursor at frame t-1 are additionally shown and represented by dashed lines. It can be understood that the first interface image 600 at frame t does not actually include the window boundary 501 and the corresponding mouse cursor.

[0061] Step S402: Divide the first interface image and the second interface image into multiple image blocks.

[0062] Please see details Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0063] Step S403: Determine the first image block that has changed in the first interface image; the first image block is not completely identical to the second image block at the corresponding position in the second interface image.

[0064] Please see details Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0065] In some alternative implementations, the method further includes steps b1 to b3.

[0066] Step b1: Determine the number of the first image blocks.

[0067] Step b2: If the number of first image blocks is less than a first quantity threshold, determine the number of first pixels that have changed in the first image block; the first pixel is different from the second pixel at the corresponding position in the second interface image.

[0068] Step b3: If the number of first pixels is less than the second threshold, determine that the window drag operation is triggered.

[0069] In this embodiment, by comparing the first interface image with the previous second interface image, the first image block that has changed can be identified, and the number of the first image blocks can be counted.

[0070] For example, each image block in the first interface image can be compared with the corresponding image block in the second interface image in a certain order. For example, starting from the image block in the upper left corner of the first interface image, the comparisons can be made in order from left to right and from top to bottom.

[0071] When comparing two image blocks, a pixel-by-pixel comparison can be performed. For the image block currently being compared in the first interface image, if a pixel in that image block differs from the corresponding pixel in the second interface image, the comparison of that image block can be stopped, meaning it can be directly determined that this image block is the first image block to have changed. Furthermore, the number of currently changed image blocks is recorded, for example, using the `ChangedBLocksCount` variable. Each time a first image block is detected, this variable is incremented. After traversing all image blocks, the value recorded in the `ChangedBLocksCount` variable is the number of first image blocks.

[0072] A suitable quantity threshold can be set, namely the first quantity threshold. If the number of the first image blocks is less than the first quantity threshold, it means that the content involved in this interface change is relatively small, which is consistent with the situation where only the window boundary changes when dragging the window, that is, there may be a window dragging operation at this time.

[0073] The first quantity threshold can be a preset value or a value set based on actual conditions. For example, the first quantity threshold is related to the total number of image blocks, such as a direct proportional relationship between the two. In this case, the ratio between the number of first image blocks (ChangedBLocksCount) and the total number of image blocks (TotalBlocksCount) can be determined. If this ratio is less than a first preset ratio, it is equivalent to the number of first image blocks being less than the first quantity threshold (i.e., the first quantity threshold is the product of the first preset ratio and the total number of image blocks (TotalBlocksCount)).

[0074] In this embodiment, if the number of first image blocks is less than a first quantity threshold, further judgment is made based on the number of pixels that have changed. Specifically, for any first image block, it can be determined which pixels are different from the corresponding pixels in the second interface image. The pixels in the first image block are called the first pixels, and the pixels in the second image block corresponding to the position of the first pixels are called the second pixels. That is, the first pixels and the corresponding second pixels are different, specifically, their pixel values ​​are different.

[0075] For one or more first image blocks, the number of first pixels that have changed in these first image blocks can be counted to determine the number of first pixels. For example, the number of all first pixels that have changed in the first interface image can be determined. If the number of first pixels is less than a second threshold, it can be indicated that there are still a few pixels that have changed in the first image block. At this time, it can be determined more accurately that there is a window drag operation, that is, the window drag operation has been triggered.

[0076] Similar to the first quantity threshold mentioned above, the second quantity threshold can be a preset fixed value or a value set based on actual conditions. For example, the second quantity threshold is related to the total number of pixels in the image block; for instance, the two are directly proportional. In this case, the ratio between the number of first pixels (DiffPixelCount) and the total number of pixels in the first image block (TotalPixelsCount) can be determined. If this ratio is less than a second preset ratio, it is equivalent to the number of first pixels being less than the second quantity threshold (i.e., the second quantity threshold is the product of the second preset ratio and the total number of pixels (TotalPixelsCount)).

[0077] Step S404: When a window dragging operation is detected to be triggered, the first image block is encoded according to the second image block at the corresponding position in the second interface image to generate an encoded image block.

[0078] Specifically, step S404, "when a window dragging operation is detected to be triggered, the first image block is encoded according to the second image block at the corresponding position in the second interface image to generate an encoded image block", may include steps S4041 to S4043.

[0079] Step S4041: For the first pixel that has changed in the first interface image, compare each first pixel with the corresponding pixel in the third interface image; the first pixel is different from the second pixel at the corresponding position in the second interface image, and the third interface image is the interface image transmitted to the client before the second interface image.

[0080] Step S4042: Determine the third pixel in the first interface image; the third pixel is the same as the first pixel at the corresponding position in the third interface image.

[0081] Step S4043: Set the pixel value of the third pixel in the first image block to a preset pixel value, and encode the other pixels in the first image block except the third pixel according to the second image block at the corresponding position in the second interface image to generate an encoded image block.

[0082] In this embodiment, in addition to using the second interface image from the previous frame, an earlier third interface image will also be used. For example, the third interface image is the interface image of the (t-2)th frame, that is, the third interface image is the image of the first interface image two frames prior to the first interface image.

[0083] For a first pixel in the first interface image that differs from the second interface image, it can be compared again with the corresponding pixel in the third interface image. If a first pixel is the same as the corresponding pixel in the third interface image, then that first pixel is called the third pixel; that is, the third pixel is a special type of first pixel.

[0084] For each third pixel, its position information can be determined, specifically representing the coordinates of the third pixel. This position information can then be sent to the client, allowing the client to directly retrieve the third pixel from the third-view image based on this information.

[0085] Accordingly, when encoding the first image block, only the pixels in the first image block other than the third pixel need to be encoded; and for the third pixel in the first image block, its pixel value can be set to a uniform preset pixel value, for example, the preset pixel value can be 0, to ensure that the XOR result is consistent. By performing appropriate processing on the different pixels in the first image block, the encoded image block corresponding to the first image block is finally generated.

[0086] It is understood that if the XOR + compression encoding method is adopted, the above step a1 may specifically include steps S4041 to S4043, and step S4043 may specifically be: setting the pixel value of the third pixel in the first image block to a preset pixel value, and performing an XOR operation on the other pixels in the first image block except the third pixel according to the second image block at the corresponding position in the second interface image to generate an intermediate image block; then the intermediate image block can be compressed to obtain the corresponding encoded image block.

[0087] Optionally, a mask image can be introduced to indicate which pixel locations have changed. Specifically, the method further includes steps c1 to c3.

[0088] Step c1: Obtain the mask image; each pixel in the mask image has a first pixel value.

[0089] Step c2: For the first pixel that changes in the first image block, set a second pixel value for the pixel in the mask image corresponding to the position of the first pixel; the second pixel value is different from the first pixel value.

[0090] Step c3: After determining the third pixel in the first interface image, set the first pixel value for the pixel in the mask image that corresponds to the position of the third pixel.

[0091] This embodiment involves four images: a first interface image, a second interface image, a third interface image, and a mask image. The mask image has the same size as each interface image, i.e., it has the same resolution.

[0092] In this mask image, each pixel has a preset initial pixel value, also known as the first pixel value. For example, the first pixel value is 0.

[0093] After identifying the first image block that has changed, in the process of traversing each pixel in the first image block to find the first pixel that has changed, each first pixel can be identified in turn, and the second pixel value can be set for the pixel in the mask image that corresponds to the position of the first pixel.

[0094] For example, if the first interface image is like Figure 6 As shown, the second interface image is as follows Figure 5 As shown; if the first pixel value is 0, then the initial pixel value of each pixel in the mask image is 0, and the mask image is a black image. By comparing the first interface image 600 and the second interface image 500, it can be determined that the pixels that change are only at the window boundary 501 and the window boundary 601. Since the window boundary is generally represented by a uniform color (e.g., black), the pixel values ​​of the pixels corresponding to the intersection of the window boundary 501 and the window boundary 601 have not changed, that is, these pixels are not the first pixel.

[0095] If the second pixel value is set to 1, then the pixels at window boundaries 501 and 601 (excluding their intersection) in the mask image can be set to 1. In interface images, a pixel is typically represented by 32 bits of data; therefore, the mask image can be processed in the same way, resulting in a second pixel value of 0xFFFFFFFF. The mask image can then be specifically configured as follows: Figure 7 As shown, in the mask image 700, the second pixel value is represented by white, mainly including the positions corresponding to the boundaries of the two windows; furthermore, since the image at the mouse position may also change, Figure 7 The pixels corresponding to the mouse cursor in both the preceding and following frames are used as the first pixel for easy display.

[0096] Furthermore, as shown above, by comparing the first pixel in the first interface image and the third interface image, the identical third pixel can be identified. Then, the mask image is further processed based on this third pixel, that is, the pixel value of the pixel corresponding to the third pixel position in the mask image is set to the initial first pixel value, so that subsequent processing can be performed based on the latest mask image.

[0097] Figure 8 A schematic diagram of a third interface image is shown, for example, the interface image at frame t-2; wherein, the third interface image 800 includes the window boundary 801 at this time, and is related to... Figure 6 Similarly, to easily distinguish the differences between the three frames, in Figure 8 The window boundaries 501 and 601 are shown separately. It can be understood that the third interface image 800 of frame t-2 does not actually contain window boundaries 501 and 601.

[0098] Comparing the first interface image 600 and the third interface image 800, it can be seen that the only difference between them lies in the window boundaries 601 and 801 (for simplicity, the position corresponding to the mouse cursor is not considered here). In other words, the pixels at other positions are the same pixels, that is, the first pixel at window boundary 501 (excluding the intersection of window boundaries 501 and 801) is the same as the pixel in the third interface image 800, that is, the first pixel at window boundary 501 is the third pixel. Furthermore, since the mouse cursor is moving, in the aforementioned mask image 700, the position corresponding to the mouse cursor in the second interface image 500 is white (the pixel corresponding to the mouse cursor is the first pixel); however, in the third interface image 800 and the first interface image 600, there is no mouse cursor at the position of the mouse cursor (the mouse cursor in the second interface image 500), that is, it is also the same third pixel.

[0099] Therefore, after setting the pixel corresponding to the third pixel position in the mask image to the first pixel value (set to 0), the final mask image can be obtained as follows: Figure 9 As shown, in Figure 9 In the mask image 900, the main characteristics are: the pixels at window boundary 601 and the mouse cursor position are the second pixel value (white), and the intersection of window boundary 501 and window boundary 801 is also white. Subsequent processing can be performed based on the final determined mask image 900.

[0100] Furthermore, the aforementioned step S4043, "setting the pixel value of the third pixel in the first image block to a preset pixel value, and encoding the other pixels in the first image block except the third pixel according to the second image block at the corresponding position in the second interface image to generate an encoded image block", may include steps d1 to d3.

[0101] Step d1: For each pixel to be processed in the first image block, determine the pixel value in the mask image corresponding to the position of the pixel to be processed.

[0102] Step d2: If the pixel value corresponding to the position of the pixel to be processed in the mask image is the first pixel value, then the pixel value of the pixel to be processed in the first image block is set to the preset pixel value.

[0103] Step d3: If the pixel value corresponding to the position of the pixel to be processed in the mask image is the second pixel value, then the pixel to be processed is encoded according to the pixel value corresponding to the position of the pixel to be processed in the second interface image.

[0104] When encoding the first image block, different pixels can be distinguished based on the mask image. Specifically, each pixel in the first image block can be traversed, which can be done using the coordinate positions [x, y].

[0105] Specifically, the pixel at coordinates [x, y] in the first image block is taken as the pixel to be processed, based on the mask image (e.g., Figure 9 The mask image 900 shown determines the pixel value MASK_IMAGE[x,y] at the coordinate position [x,y]. If the pixel value MASK_IMAGE[x,y] is the first pixel value, for example, 0, it means that the pixel at the coordinate position [x,y] in the first image block is the third pixel and does not need to be specially processed. Therefore, the pixel value of the pixel at the coordinate position [x,y] in the first image block can be directly set to the preset pixel value, for example, 0.

[0106] If the pixel value MASK_IMAGE[x,y] is the second pixel value, for example, 1 or 0xFFFFFFFF, it means that the pixel at coordinate [x,y] in the first image block is a normal first pixel (i.e., not the third pixel), so it needs to be encoded. Specifically, an XOR operation can be performed on the pixel, followed by compression, which can make the final encoded image block have a smaller processing load.

[0107] by Figure 6 Taking the first interface image shown as an example, any first image block therein can be encoded in the manner described above. Figure 10 This diagram illustrates a method for dividing a first image block 1001 from a first interface image 600. Specifically, the first image block 1001 can be as follows: Figure 11 As shown.

[0108] Figure 11 A comparison diagram of the two encoding results for the first image patch is shown, such as... Figure 11 As shown, the first image block 1001 is an 8×8 image block, where each square corresponds to one pixel, meaning the first image block 1001 has a total of 64 pixels. Figure 11 Different pixel values ​​are represented by different gray levels. Specifically, the third row of the first image block 1001 corresponds to window boundary 501, and the sixth row corresponds to window boundary 601.

[0109] If the third pixel is not distinguished, that is, if the earlier third interface image is not used, then the first image block 1001 is XORed according to the second image block of the previous frame. The result of the XOR operation can be as shown in the intermediate image block 1101. The XOR operation results corresponding to the window boundary 501 and the window boundary 601 can be various cases, while the XOR operation results of the remaining pixels are all 0. Figure 11 In the image, for ease of display, white represents a pixel value of 0.

[0110] However, if a third-interface image is used to distinguish the third pixel, then based on Figure 9As shown in the mask image 900, in the first image block 1001, the pixel corresponding to the window boundary 501 is the third pixel. At this point, its pixel value is directly set to a preset pixel value, such as zero, to ensure consistency with the XOR operation result of the same pixel. This allows the generation of the intermediate image block 1102. Compared to the intermediate image block 1101, in the intermediate image block 1102, the pixel at the window boundary 501 has the same pixel value as the other pixels. This further improves the compression ratio of subsequent compression processing, resulting in a smaller data size for the compressed encoded image block.

[0111] Optionally, the method further includes: for a first image block that has changed in the first interface image, if all the first pixels that have changed in the first image block are the same as the pixels at the corresponding positions in the third interface image, then the first image block is removed, so as to encode the remaining first image block.

[0112] In this embodiment, before step S404 "when a window dragging operation is detected to be triggered, the first image block is encoded according to the second image block at the corresponding position in the second interface image to generate an encoded image block", a portion of the first image block can be removed to further reduce the amount of data transmitted.

[0113] Specifically, after identifying the pixels in the first pixel that are identical to the corresponding pixels in the third interface image—that is, after determining which first pixels are all third pixels—it can be further determined whether all the first pixels in the first image block are third pixels. If they are all third pixels, it means that the pixels in the first image block can be recovered from the third interface image, and no additional data needs to be transmitted to the client. Therefore, the first image block is discarded before encoding, meaning that no encoding is required for the first image block, and no information related to the first image block needs to be sent to the client.

[0114] Conversely, if at least one first pixel in a first image block is not a third pixel, the first image block will be retained and subsequently encoded in step S404.

[0115] Optionally, step S4042, "determining the position information of the third pixel in the first interface image," can specifically involve scanning by row and / or by column, and includes the following steps: The first pixel in the first interface image is scanned line by line to determine the fourth pixel in the first pixel that is the same as the pixel at the corresponding position in the third interface image; if the number of consecutive pixels corresponding to consecutive fourth pixels in a single line is greater than a preset number, then each consecutive fourth pixel is taken as the third pixel.

[0116] And / or, each of the first pixels in the first interface image is scanned column by column to determine the fifth pixel in the first pixel that is the same as the pixel at the corresponding position in the third interface image; if the number of consecutive pixels corresponding to consecutive fifth pixels in a single column is greater than a preset number, then each consecutive fifth pixel is taken as the third pixel.

[0117] In this embodiment, by scanning in the horizontal and / or vertical directions, consecutive fourth and / or fifth pixels can be obtained. These consecutive fourth and fifth pixels are then used as third pixels, and the starting pixel position of the corresponding row or column is determined. This allows for a unified representation of the overall position information without needing to record the position coordinates of each third pixel, resulting in a smaller data volume for the position information. When sending the position information to the client later, it also does not require excessive bandwidth.

[0118] The location information of the third pixel can be determined simultaneously with the search for the third pixel. Specifically, after obtaining the corresponding mask image (e.g., mask image 700) in step c2, the mask image can be scanned horizontally. If the pixel value of the scanned pixel is the second pixel value (e.g., 0xFFFFFFFF), the first pixel corresponding to that pixel can be compared with the pixel at the corresponding position in the third interface image. If they are the same, the first pixel can be marked as the third pixel. Similarly, for a line composed of consecutive third pixels on a horizontal scan line, if its length exceeds the number of consecutive pixels MIN_LINE_LENGTH, then its starting pixel position is...<x_start, y> The end pixel position is<x_end, y> Its starting pixel position can be represented as<x_start, x_end, y> This is used as the position information of the third pixel in this part.

[0119] If a vertical scan is performed, the starting pixel position can be represented as:<y_start, y_end, x> This can also serve as the position information for the corresponding third pixel. Further details will not be elaborated here.

[0120] In the process of scanning the mask image to determine the third pixel, if the third pixel is detected, the corresponding pixel value in the mask image can be set as the first pixel value as shown in step c3.

[0121] In some optional implementations, step S404, "when a window dragging operation is detected to be triggered, the first image block is encoded according to the second image block at the corresponding position in the second interface image to generate an encoded image block," may specifically include: for a first image block of a first type, encoding the first image block of the first type according to the second image block at the corresponding position in the second interface image to generate a first encoded image block, and setting a first type identifier for the first encoded image block; wherein, in the first image block of the first type, the number of first pixels that have changed is less than a third quantity threshold.

[0122] Furthermore, the method may also include: for a first image block of the second type, compressing the first image block of the second type to generate a second coded image block, and setting a second type identifier for the second coded image block that is different from the first type identifier; wherein, in the first image block of the second type, the number of first pixels that have changed is greater than or equal to a third quantity threshold.

[0123] In this embodiment, in step b3 above, the number of first pixels in each first image block can be determined. If the number of first pixels is less than the third quantity threshold, the data volume of the first image block can be effectively reduced through subsequent XOR and compression processing. Therefore, the first image block is regarded as the first type of image block and a first type identifier is set. XOR, compression and other encoding processing can be performed on it to obtain the first encoded image block. The third quantity threshold and the second quantity threshold can be different or the same (for example, both are thresholds related to the ratio). This embodiment does not limit this.

[0124] If the number of first pixels in the first image block is greater than or equal to the third quantity threshold, it means that even if XOR processing is performed on it, the change in data volume is not significant. Therefore, in this case, only compression processing is needed to obtain the second encoded image block. That is, no other second interface image is required. Subsequently, the client can directly restore the first image block by decompressing the second encoded image block. Among them, this type of first image block is a second type of image block and is set with a second type identifier.

[0125] In this embodiment, both the first coded image block and the second coded image block are sent to the client. The client can distinguish between the two types of coded image blocks according to the type identifier of the coded image block, and restore the original image block based on different processing methods.

[0126] Step S405: The position information of the third pixel and the encoded image blocks corresponding to each first image block are sent to the client, instructing the client to decode the first interface image according to the encoded image blocks.

[0127] In this embodiment, in addition to sending the encoded image block corresponding to the first image block to the client, it is also necessary to send the position information of the third pixel. After receiving this information, the client can reconstruct the first interface image using the second and third interface images, which will be explained later.

[0128] The image processing method provided in this embodiment, after determining the first pixel that is different between the first interface image and the second interface image, further determines the third pixel that is the same as the third pixel in the first pixel based on the earlier third interface image, and determines the position information of the third pixel; when encoding the first image block, it mainly encodes the first pixels other than the third pixel, which can further reduce the amount of data of the encoded image block; and for the third pixel, it only needs to transmit the corresponding position information to the client, which can further reduce the amount of data transmitted to the client and reduce bandwidth loss.

[0129] This embodiment provides an image processing method that can be used on a client, such as a cloud desktop client installed on a terminal device. Figure 12 This is a flowchart of an image processing method according to an embodiment of this application, such as... Figure 12 As shown, the process includes the following steps.

[0130] Step S1201: Obtain the encoding information sent by the server. The encoding information includes encoded image blocks; these encoded image blocks are based on the above embodiments (e.g., Figure 2 The image is generated by the image processing method provided in the embodiments shown, etc.

[0131] Step S1202: Based on the second image block in the second interface image that corresponds to the position of the encoded image block in the image that has been transmitted to the client, the encoded image block is decoded to obtain the first image block.

[0132] Step S1203: Replace the second image block in the second interface image with the first image block at the corresponding position to generate the current first interface image.

[0133] In this embodiment, after the server generates the encoded image block corresponding to the first interface image, it sends it to the client. The client can cache the second interface image of the previous frame to achieve decoding.

[0134] Specifically, after receiving the encoded image block sent by the server, the client can determine the corresponding second image block based on the previous second interface image. For example, the information sent by the server includes the identifier of the encoded image block, such as the ID of the encoded image block and its position in the entire interface image. Based on this, the client can determine the corresponding second image block.

[0135] After determining the second image block, the encoded image block can be decoded based on the second image block to restore the first image block. Then, each second image block in the second interface image is replaced with the first image block at the corresponding position to update the second interface image. The updated second interface image is the current first interface image.

[0136] If the encoded image block is obtained based on XOR and compression, the client needs to perform decompression and XOR processing sequentially when decoding the encoded image block.

[0137] Specifically, step S1202, "decoding the encoded image block according to the second image block corresponding to the position of the encoded image block in the previous second interface image to obtain the first image block," may include: decompressing the encoded image block to obtain the corresponding intermediate image block; and performing XOR processing on the intermediate image block according to the second image block corresponding to the position of the encoded image block in the previous second interface image to obtain the first image block.

[0138] Optionally, if the server also determines the third pixel based on the third interface image, the encoded information it sends to the client also includes: location information, i.e., the location information of the third pixel. In this case, the above step S1203, "replacing the second image block in the second interface image with the first image block at the corresponding position to generate the current first interface image," may include steps e1 to e3.

[0139] Step e1: Determine the sixth pixel in the third interface image that corresponds to the position information; the third interface image is the interface image preceding the second interface image.

[0140] Step e2: Update the first image block according to the sixth pixel; the pixel corresponding to the position information in the updated first image block is the sixth pixel.

[0141] Step e3: Replace the second image block in the second interface image with the first image block updated at the corresponding position to generate the current first interface image.

[0142] In this embodiment, after obtaining the first image block through decoding in step S1202, the pixel corresponding to the position information in the third interface image, i.e., the sixth pixel, can be determined based on the position information. It can be understood that since the position information represents the position of the same pixel in both the first and third interface images, the sixth pixel in the third interface image is the same as the third pixel at the corresponding position in the first interface image; that is, the sixth pixel can be used as the corresponding third pixel. Therefore, the first image block can be updated based on the sixth pixel, thereby updating the pixel in the first image block corresponding to the position information to the sixth pixel. Finally, by combining the second interface image and the updated first image block, the first interface image can be reconstructed.

[0143] The following example illustrates in detail the server's encoding process and the client's decoding process. Figure 13 As shown.

[0144] Get the first interface image CUR_FRAME of the current frame, the second interface image PRE1_FRMAE of the previous frame, and the third interface image PRE2_FRAME of the previous two frames. Set the mask image MASK_FRAME and initialize the mask image MASK_FRAME to all 0s, that is, the first pixel value is 0.

[0145] The first interface image CUR_FRAME and the second interface image PRE1_FRMAE are divided into multiple image blocks. Then, starting from the top left image block, each image block is processed sequentially from left to right and from top to bottom.

[0146] First, for the current image block to be analyzed, compare the first interface image CUR_FRAME and the second interface image PRE1_FRMAE pixel by pixel; the comparison can be done in 32-bit units, with each 32 bits representing one pixel.

[0147] Determine whether there are different pixels between the image blocks in the first interface image CUR_FRAME and the image blocks in the second interface image PRE1_FRMAE.

[0148] If there are no different pixels, it means that the image blocks of the two interface images are the same, and no processing is required. Instead, the next image block is traversed directly, and the next image block is processed.

[0149] If there are different pixels, the image block to be analyzed is the first image block, and the different pixel is the first pixel. At this time, the image block can be marked as Change (i.e., the second type of mark), and the number of the first image block can be incremented by one.

[0150] Furthermore, for the first pixel that changes within the image block (i.e., the first image block), the pixel value at the corresponding position in the mask image MASK_FRAME is set to 1, meaning the second pixel value is 1. The number of first pixels in the image block is also recorded.

[0151] Determine whether the number of the first image blocks is less than the first quantity threshold Th1, that is, determine whether the number of image blocks that have changed is less than the first quantity threshold Th1. If it is not less than the first quantity threshold Th1, it means that the interface change at this time may not be caused by the drag operation, and other methods can be used for processing, which is not limited here.

[0152] If the number is less than the first quantity threshold Th1, then it can be further determined whether the number of first pixels in each first image block is less than the second quantity threshold Th2.

[0153] If the number of first pixels in a first image block is less than the second threshold Th2, it indicates that the first image block is suitable for processing using the encoding method provided in this embodiment. In this case, the first interface image CUR_FRAME and the third interface image PRE2_FRAME can be compared and the same third pixels can be determined. The pixel values ​​at the corresponding positions in the mask image MASK_FRAME can then be set to 0. For example, this makes the mask image MASK_FRAME... Figure 7 Become Figure 9 For details, please refer to the relevant descriptions of steps c1 to c3 above, which will not be repeated here.

[0154] Furthermore, for the third pixel, its corresponding position information is recorded. For example, the starting position is recorded.<x_start, x_end, y> and / or<y_start, y_end, x> wait.

[0155] For a first image block with fewer than Th2 pixels, it is marked as a Mask (i.e., a first type of marker), and for each pixel to be processed, the pixel value in the mask image corresponding to the position of the pixel to be processed is determined.

[0156] If the value is 0, meaning it's the first pixel value, then the pixel value of the pixel to be processed is directly set to 0. If the value is 1, meaning it's the second pixel value, then an XOR operation is performed based on the corresponding pixel in the second interface image PRE1_FRAME. The pseudocode for this part can be as follows: if(MASK_FRAME[x,y]) CUR_FRAME[x,y]^=PRE1_FRAME[x,y] / / XOR operation else CUR_FRAME[x,y]=0 The intermediate image block can then be obtained.

[0157] If the number of first pixels in the first image block is greater than the second number threshold Th2, it indicates that the first image block is a second type of image block, and the first image block can be directly compressed to obtain the second coded image block.

[0158] For each first image block, including the two types marked Mask and Change, lossless compression is performed to obtain coded image blocks. For example, LZ4 or other compression algorithms can be used; this is not limited here. Each first image block can be compressed separately, or they can be compressed uniformly; this is also not limited here.

[0159] Specifically, for the first image block marked "Mask", the intermediate image block after XORing is compressed, while for the first image block marked "Change", it is compressed directly. After compression, the corresponding first coded image block and second coded image block are obtained.

[0160] Finally, the position information of the encoded image block (which may include the first encoded image block and the second encoded image block) and the third pixel is sent to the client to complete the encoding process for the first interface image CUR_FRAME.

[0161] After receiving the encoded image blocks and location information, the client first decompresses each encoded image block to obtain the decompressed image block DECODED_FRAME. Furthermore, the client records the interface image PRE1_FRAME_C from the previous frame, as well as the images PRE2_FRAME_C from the previous two frames. It can be understood that these interface images PRE1_FRAME_C and PRE2_FRAME_C are identical to the server's second interface image PRE1_FRAME and third interface image PRE2_FRAME.

[0162] If the final output image from the client is DST_FRAME, then the interface image PRE1_FRAME_C is assigned to the output image DST_FRAME, i.e., DST_FRAME=PRE1_FRAME_C.

[0163] For encoded image blocks marked as Change (i.e., image blocks of type 2), they can be drawn directly after decompression, i.e.: DST_FRAME[x,y]= DECODED_FRAME[x,y].

[0164] For encoded image blocks marked as Mask (i.e., second-type image blocks), an XOR operation needs to be performed after decompression, that is: DST_FRAME [x,y]=PRE1_FRAME_C[x,y]^ DECODED_FRAME[x,y].

[0165] Furthermore, based on the location information transmitted by the server, the pixels at the corresponding positions in PRE2_FRAME_C can be obtained and overwritten. That is, DST_FRAME[x,y] = PRE2_FAME_C[x,y].

[0166] After the above processing, the output image DST_FRAME[x,y] is the restored first interface image.

[0167] This embodiment also provides an image processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0168] This embodiment provides an image processing device applied to a server, such as... Figure 14 As shown, the device includes: The first acquisition module 1401 is used to acquire the first interface image to be transmitted to the client and the second interface image that has been transmitted to the client. The segmentation module 1402 is used to divide the first interface image and the second interface image into multiple image blocks; Processing module 1403 is used to determine a first image block that has changed in the first interface image; the first image block is not completely the same as a second image block at the corresponding position in the second interface image; when a window drag operation is detected to be triggered, the first image block is encoded according to the second image block at the corresponding position in the second interface image to generate an encoded image block; The sending module 1404 is used to send the encoded image blocks corresponding to each of the first image blocks to the client, instructing the client to decode the first interface image according to the encoded image blocks.

[0169] In some optional embodiments, the processing module 1403 is further configured to: If the number of the first image blocks is less than a first quantity threshold, determine the number of first pixels that have changed in the first image blocks; the first pixels are different from the second pixels at the corresponding positions in the second interface image; If the number of the first pixels is less than the second threshold, it is determined that the window dragging operation is triggered.

[0170] In some alternative implementations, The step of encoding the first image block according to the second image block at the corresponding position in the second interface image to generate an encoded image block includes: For the first pixel that changes in the first interface image, each first pixel is compared with the corresponding pixel in the third interface image; the first pixel is different from the second pixel at the corresponding position in the second interface image, and the third interface image is the interface image transmitted to the client before the second interface image; Determine the third pixel in the first interface image; the third pixel is the same as the first pixel at the corresponding position in the third interface image; Set the pixel value of the third pixel in the first image block to a preset pixel value, and encode the other pixels in the first image block except for the third pixel according to the second image block at the corresponding position in the second interface image to generate an encoded image block; The sending module 1404 is further configured to: The location information of the third pixel is sent to the client.

[0171] In some optional implementations, determining the third pixel in the first interface image includes: The first pixel in the first interface image is scanned line by line to determine the fourth pixel in the first pixel that is the same as the pixel at the corresponding position in the third interface image; if the number of consecutive pixels corresponding to consecutive fourth pixels in a single line is greater than a preset number, then each consecutive fourth pixel is taken as the third pixel. And / or, The first pixel in the first interface image is scanned column by column to determine the fifth pixel in the first pixel that is the same as the pixel at the corresponding position in the third interface image; if the number of consecutive pixels corresponding to consecutive fifth pixels in a single column is greater than a preset number, then each consecutive fifth pixel is taken as the third pixel.

[0172] In some optional embodiments, the processing module 1403 is further configured to: Obtain a mask image; each pixel in the mask image is assigned a first pixel value; For the first pixel that changes in the first image block, a second pixel value is set for the pixel in the mask image corresponding to the position of the first pixel; the second pixel value is different from the first pixel value. After determining the third pixel in the first interface image, the first pixel value is set for the pixel in the mask image that corresponds to the position of the third pixel; The step of setting the pixel value of the third pixel in the first image block to a preset pixel value, and encoding the other pixels in the first image block except for the third pixel according to the second image block at the corresponding position in the second interface image to generate an encoded image block includes: For each pixel to be processed in the first image block, determine the pixel value in the mask image corresponding to the position of the pixel to be processed; If the pixel value corresponding to the position of the pixel to be processed in the mask image is the first pixel value, then the pixel value of the pixel to be processed in the first image block is set to the preset pixel value; If the pixel value corresponding to the position of the pixel to be processed in the mask image is the second pixel value, then the pixel to be processed is encoded according to the pixel value corresponding to the position of the pixel to be processed in the second interface image.

[0173] In some optional embodiments, the processing module 1403 is further configured to: For each first image block that changes in the first interface image, if the first pixel that changes in the first image block is the same as the pixel at the corresponding position in the third interface image, then the first image block is removed before encoding the first image block, so that the retained first image block can be encoded.

[0174] In some optional implementations, the step of encoding the first image block based on the second image block at a corresponding position in the second interface image to generate an encoded image block includes: Perform an XOR operation on the first image block and the second image block at the corresponding position to generate an intermediate image block; The intermediate image block is compressed to generate the encoded image block corresponding to the first image block.

[0175] In some optional implementations, the step of encoding the first image block based on the second image block at a corresponding position in the second interface image to generate an encoded image block includes: For the first image block of the first type, the first image block of the first type is encoded according to the second image block at the corresponding position in the second interface image to generate a first encoded image block, and a first type identifier is set for the first encoded image block; wherein, in the first image block of the first type, the number of first pixels that have changed is less than a third quantity threshold; The processing module 1403 is also used for: For the first image block of the second type, the first image block of the second type is compressed to generate a second coded image block, and a second type identifier different from the first type identifier is set for the second coded image block; wherein, in the first image block of the second type, the number of first pixels that have changed is greater than or equal to the third quantity threshold.

[0176] This embodiment provides another image processing device, applied to a client, such as... Figure 15 As shown, the device includes: The second acquisition module 1501 is used to acquire encoded information sent by the server, the encoded information including encoded image blocks; the encoded image blocks are generated based on the image processing method of the first aspect described above or any corresponding embodiment thereof; Decoding module 1502 is used to decode the encoded image block according to the second image block in the second interface image that corresponds to the position of the encoded image block, to obtain the first image block; The generation module 1503 is used to replace the second image block in the second interface image with the first image block at the corresponding position to generate the current first interface image.

[0177] In some optional implementations, the encoded information further includes: location information; The step of replacing the second image block in the second interface image with the first image block at the corresponding position to generate the current first interface image includes: Determine the sixth pixel in the third interface image that corresponds to the location information; the third interface image is the interface image preceding the second interface image. The first image block is updated based on the sixth pixel; the pixel in the updated first image block that corresponds to the position information is the sixth pixel; The second image block in the second interface image is replaced with the first image block updated at the corresponding position to generate the current first interface image.

[0178] The image processing apparatus provided in this disclosure can execute the image processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0179] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0180] The following is a detailed reference. Figure 16 The diagram illustrates a structural schematic suitable for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 1601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1602 or a program loaded from memory 1608 into random access memory (RAM) 1603. The RAM 1603 also stores various programs and data required for the operation of the electronic device. The processor 1601, ROM 1602, and RAM 1603 are interconnected via a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0181] Typically, the following devices can be connected to the I / O interface 1605: input devices 1606 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; memory devices 1608 including, for example, magnetic tape, hard disk, etc.; and communication devices 1609. Communication device 1609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 16 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0182] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 1609, or installed from a memory 1608, or installed from a ROM 1602. When the computer program is executed by the processor 1601, it performs the functions defined in the image processing method of embodiments of this application.

[0183] Figure 16 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0184] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the image processing method shown in the above embodiments is implemented.

[0185] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0186] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. An image processing method, characterized in that, Applied to a server, the method includes: Obtain the first interface image to be transmitted to the client, and the second interface image that has already been transmitted to the client; The first interface image and the second interface image are divided into multiple image blocks; Identify the first image block in the first interface image that has changed; the first image block is not exactly the same as the second image block at the corresponding position in the second interface image. When a window dragging operation is detected, the first image block is encoded according to the second image block at the corresponding position in the second interface image to generate an encoded image block; The encoded image blocks corresponding to each of the first image blocks are sent to the client, instructing the client to decode the first interface image based on the encoded image blocks.

2. The method according to claim 1, characterized in that, The method further includes: If the number of the first image blocks is less than a first quantity threshold, determine the number of first pixels that have changed in the first image blocks; the first pixels are different from the second pixels at the corresponding positions in the second interface image; If the number of the first pixels is less than the second threshold, it is determined that the window dragging operation is triggered.

3. The method according to claim 1, characterized in that, The step of encoding the first image block according to the second image block at the corresponding position in the second interface image to generate an encoded image block includes: For the first pixel that changes in the first interface image, each first pixel is compared with the corresponding pixel in the third interface image; the first pixel is different from the second pixel at the corresponding position in the second interface image, and the third interface image is the interface image transmitted to the client before the second interface image; Determine the third pixel in the first interface image; the third pixel is the same as the first pixel at the corresponding position in the third interface image; Set the pixel value of the third pixel in the first image block to a preset pixel value, and encode the other pixels in the first image block except for the third pixel according to the second image block at the corresponding position in the second interface image to generate an encoded image block; The method further includes: The location information of the third pixel is sent to the client.

4. The method according to claim 3, characterized in that, Determining the third pixel in the first interface image includes: The first pixel in the first interface image is scanned line by line to determine the fourth pixel in the first pixel that is the same as the pixel at the corresponding position in the third interface image; if the number of consecutive pixels corresponding to consecutive fourth pixels in a single line is greater than a preset number, then each consecutive fourth pixel is taken as the third pixel. And / or, The first pixel in the first interface image is scanned column by column to determine the fifth pixel in the first pixel that is the same as the pixel at the corresponding position in the third interface image; if the number of consecutive pixels corresponding to consecutive fifth pixels in a single column is greater than a preset number, then each consecutive fifth pixel is taken as the third pixel.

5. The method according to claim 3, characterized in that, The method further includes: Obtain a mask image; each pixel in the mask image is assigned a first pixel value; For the first pixel that changes in the first image block, a second pixel value is set for the pixel in the mask image corresponding to the position of the first pixel; the second pixel value is different from the first pixel value. After determining the third pixel in the first interface image, the first pixel value is set for the pixel in the mask image that corresponds to the position of the third pixel; The step of setting the pixel value of the third pixel in the first image block to a preset pixel value, and encoding the other pixels in the first image block except for the third pixel according to the second image block at the corresponding position in the second interface image to generate an encoded image block includes: For each pixel to be processed in the first image block, determine the pixel value in the mask image corresponding to the position of the pixel to be processed; If the pixel value corresponding to the position of the pixel to be processed in the mask image is the first pixel value, then the pixel value of the pixel to be processed in the first image block is set to the preset pixel value; If the pixel value corresponding to the position of the pixel to be processed in the mask image is the second pixel value, then the pixel to be processed is encoded according to the pixel value corresponding to the position of the pixel to be processed in the second interface image.

6. The method according to claim 3, characterized in that, The method further includes: For each first image block that changes in the first interface image, if the first pixel that changes in the first image block is the same as the pixel at the corresponding position in the third interface image, then the first image block is removed before encoding the first image block, so that the retained first image block can be encoded.

7. The method according to any one of claims 1 to 6, characterized in that, The step of encoding the first image block according to the second image block at the corresponding position in the second interface image to generate an encoded image block includes: Perform an XOR operation on the first image block and the second image block at the corresponding position to generate an intermediate image block; The intermediate image block is compressed to generate the encoded image block corresponding to the first image block.

8. The method according to any one of claims 1 to 6, characterized in that, The step of encoding the first image block according to the second image block at the corresponding position in the second interface image to generate an encoded image block includes: For the first image block of the first type, the first image block of the first type is encoded according to the second image block at the corresponding position in the second interface image to generate a first encoded image block, and a first type identifier is set for the first encoded image block; wherein, in the first image block of the first type, the number of first pixels that have changed is less than a third quantity threshold; For the first image block of the second type, the first image block of the second type is compressed to generate a second coded image block, and a second type identifier different from the first type identifier is set for the second coded image block; wherein, in the first image block of the second type, the number of first pixels that have changed is greater than or equal to the third quantity threshold.

9. An image processing method, characterized in that, Applied to a client, the method includes: Obtain encoded information sent by the server, the encoded information including encoded image blocks; the encoded image blocks are generated based on the image processing method according to any one of claims 1 to 8; Based on the second image block in the second interface image that corresponds to the position of the encoded image block, the encoded image block is decoded to obtain the first image block; Replace the second image block in the second interface image with the first image block at the corresponding position to generate the current first interface image.

10. The method according to claim 9, characterized in that, The encoded information also includes: location information; The step of replacing the second image block in the second interface image with the first image block at the corresponding position to generate the current first interface image includes: Determine the sixth pixel in the third interface image that corresponds to the location information; the third interface image is the interface image preceding the second interface image. The first image block is updated based on the sixth pixel; the pixel in the updated first image block that corresponds to the position information is the sixth pixel; The second image block in the second interface image is replaced with the first image block updated at the corresponding position to generate the current first interface image.

11. An image processing apparatus, characterized in that, Applied to a server, the device includes: The first acquisition module is used to acquire the first interface image to be transmitted to the client and the second interface image that has been transmitted to the client. A segmentation module is used to divide the first interface image and the second interface image into multiple image blocks; The processing module is used to determine a first image block that has changed in the first interface image; the first image block is not completely identical to a second image block at a corresponding position in the second interface image; when a window drag operation is detected to be triggered, the first image block is encoded according to the second image block at the corresponding position in the second interface image to generate an encoded image block; The sending module is used to send the encoded image blocks corresponding to each of the first image blocks to the client, instructing the client to decode the first interface image according to the encoded image blocks.

12. An image processing apparatus, characterized in that, Applied to a client, the device includes: The second acquisition module is used to acquire encoded information sent by the server, the encoded information including encoded image blocks; the encoded image blocks are generated based on the image processing method according to any one of claims 1 to 8; The decoding module is used to decode the encoded image block according to the second image block in the second interface image that corresponds to the position of the encoded image block, to obtain the first image block; The generation module is used to replace the second image block in the second interface image with the first image block at the corresponding position to generate the current first interface image.

13. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the image processing method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the image processing method according to any one of claims 1 to 10.

15. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the image processing method according to any one of claims 1 to 10.

Images