Image processing method, device and equipment of virtual desktop, and storage medium

Abstract

The application provides a virtual desktop image processing method and device, equipment and a storage medium; the method comprises the following steps: sending a desktop update image to be updated by a virtual desktop in a first period to a terminal; obtaining a compensation image link table, the compensation image link table comprising a first compensation image corresponding to N second periods respectively, the N second periods being the first N continuous periods of the first period; if the desktop update image comprises a loss area, removing the area overlapping with the loss area from each first compensation image in the compensation image link table to obtain a second compensation image of each second period; performing encoding processing on the second compensation image of the Nth second period in the compensation image link table to obtain a target image, and sending the target image to the terminal, wherein the target image is used for image compensation on the second compensation image of the Nth second period in the virtual desktop. Through the application, the image compensation effect of the virtual desktop can be improved.

Description

Technical Field

[0001] This application relates to the field of virtual desktop technology, and in particular to an image processing method, apparatus, device and storage medium for virtual desktops. Background Technology

[0002] Virtual desktop (or cloud desktop) technology provides users with the ability to access cloud computing environments on their local devices. Screen content encoding in virtual desktops is a core aspect affecting user experience. To improve the experience, lossy encoding methods, or a combination of lossy and lossless encoding methods, are often used. Lossy encoding results in a noticeable loss of image quality; therefore, image compensation for the areas to be compensated in lossy encoded data is a problem that urgently needs to be solved. Summary of the Invention

[0003] This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium for virtual desktops, which can improve the image compensation effect of virtual desktops.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an image processing method for a virtual desktop, including: Send the updated desktop image to be updated in the first cycle to the terminal; Obtain a compensation image list, wherein the compensation image list includes the first image to be compensated corresponding to the virtual desktop in N second periods, and the N second periods are the first N consecutive periods of the first period, where N is an integer greater than 1; If the desktop update image includes a lossy region, remove the region that overlaps with the lossy region from each of the first images to be compensated in the compensation image chain to obtain a second image to be compensated for each second period; The second image to be compensated in the Nth second period of the compensation image chain is encoded to obtain a target image, and the target image is sent to the terminal. The target image is used to perform image compensation on the second image to be compensated in the Nth second period of the virtual desktop.

[0005] This application also provides an image processing apparatus for a virtual desktop, including: The sending module is used to send the updated desktop image of the virtual desktop to be updated in the first cycle to the terminal. The acquisition module is used to acquire a compensation image list, wherein the compensation image list includes the first image to be compensated corresponding to the virtual desktop in N second periods, and the N second periods are the first N consecutive periods of the first period, where N is an integer greater than 1. The removal module is used to remove regions that overlap with the damaged regions from each of the first images to be compensated in the compensation image chain if the desktop update image includes a damaged region, thereby obtaining a second image to be compensated for each second period; The encoding module is used to encode the second image to be compensated in the Nth second period of the compensation image chain to obtain a target image, and send the target image to the terminal, wherein the target image is used by the terminal to perform image compensation on the second image to be compensated in the Nth second period of the virtual desktop.

[0006] This application also provides an electronic device, including: Memory is used to store executable instructions for a computer; The processor, when executing computer-executable instructions stored in the memory, implements the image processing method for the virtual desktop provided in the embodiments of this application.

[0007] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the image processing method for the virtual desktop provided in this application.

[0008] The embodiments of this application have the following beneficial effects: A compensation image chain list for the first N consecutive second periods of the first period of the virtual desktop is obtained. When the desktop update image in the first period contains a lossy region, the region overlapping with the lossy region is removed from each first image to be compensated in the compensation image chain list. Then, the second image to be compensated in the Nth second period of the compensation image chain list is encoded and sent to the terminal to perform image compensation on the second image to be compensated in the Nth second period of the virtual desktop. Thus, by removing the static region (i.e., non-updating image) covered by the lossy region of the desktop update image in each first image to be compensated, the amount of data used for the target image for image compensation is reduced, and bandwidth control is optimized to ensure smooth data transmission. Simultaneously, by recording the first image to be compensated in each second period before the first period in the compensation image chain list, and only performing image compensation on the second image of the Nth second period, lossless compensation is achieved for long-term static regions to restore image quality, ensuring a balance between the smoothness of the virtual desktop update region and the clarity of the static region, thereby improving the image compensation effect of the virtual desktop. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the architecture of the image processing system for the virtual desktop provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 3This is a first flowchart illustrating the image processing method for a virtual desktop provided in this application embodiment; Figure 4 This is a schematic diagram of the segmentation of the smallest image region of the desktop update image provided in the embodiments of this application; Figure 5 This is a second flowchart illustrating the image processing method for virtual desktops provided in this application embodiment; Figure 6 This is a schematic diagram of the first update process of the compensation image linked list provided in the embodiments of this application; Figure 7 This is a schematic diagram of the second update process of the compensation image linked list provided in the embodiments of this application; Figure 8 This is a schematic diagram of the third update process of the compensation image linked list provided in the embodiments of this application; Figure 9 This is a schematic diagram of the fourth update process of the compensation image linked list provided in the embodiments of this application; Figure 10 This is a schematic diagram of the fifth update process of the compensation image linked list provided in the embodiments of this application.

[0010] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0013] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0014] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0015] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0016] 1) Client: An application running on an electronic device that provides various services, such as a client that supports virtual desktops.

[0017] 2) Lossy encoding is a method that actively discards some non-critical image information (such as high-frequency texture details) during data compression. Its core feature is to significantly reduce the amount of data by sacrificing some image quality. This method is based on the low sensitivity of the human eye to image quality loss in dynamic areas, and is suitable for dynamic update scenarios in virtual desktops (such as window dragging and video playback). It can effectively control transmission bandwidth and ensure the smooth transmission of dynamic content. Images obtained through lossy encoding are lossy images.

[0018] 3) Lossless encoding is an encoding method that completely preserves all the original pixel information of an image during data compression. Its core feature is that the decompressed image is completely identical to the original image, without any loss of detail. This method is suitable for static update scenarios in virtual desktops (such as document text and chart display). It ensures image clarity through lossless compression and is often used for image quality restoration of long-term static areas, solving the problem of detail loss caused by lossy encoding. The image obtained through lossless encoding is a lossless image.

[0019] 4) Image compensation refers to the technical operation of replacing low-quality image areas in the terminal display with high-quality images. Its core goal is to restore the original clarity of the image. In a virtual desktop scenario, this is specifically manifested as follows: the server performs lossless or high-quality encoding on static areas that have not been updated for a long time to obtain the target image. After sending it to the terminal, the terminal uses the target image to replace the original lossy encoded display area, repairing the image blurring caused by lossy encoding in long-term static areas and restoring the image quality of static areas.

[0020] This application provides an image processing method, apparatus, electronic device, and computer-readable storage medium for virtual desktops, which can improve the image compensation effect of virtual desktops. The embodiments of this application will now be described in detail based on the above description of the terms and concepts used in the embodiments.

[0021] The following describes the image processing system for a virtual desktop provided in an embodiment of this application. See also: Figure 1 , Figure 1 This is a schematic diagram of the architecture of the image processing system for a virtual desktop provided in an embodiment of this application. To support an exemplary application, the image processing system 100 for the virtual desktop includes: a server 200, a network 300, and a terminal 400. The terminal 400 is connected to the server 200 via the network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both, using wireless or wired links for data transmission.

[0022] Here, terminal 400 runs a client that supports virtual desktops and sends a virtual desktop data acquisition request to server 200; server 200 receives the virtual desktop data acquisition request sent by terminal 400, and in response to the data acquisition request, returns the virtual desktop data to terminal 400; terminal 400 receives the virtual desktop data returned by server 200, and renders and displays the virtual desktop based on the virtual desktop data.

[0023] Continuing, the user performs interactive operations (such as video playback, file dragging and dropping, document manipulation, etc.) on the virtual desktop displayed on the terminal 400. In response to the interactive operation, the terminal 400 sends a desktop update request to the server 200. Upon receiving the desktop update request from the terminal 400, the server 200 retrieves the desktop update image to be updated within the first period and sends the desktop update image to the terminal. The terminal 400 receives the desktop update image and updates the currently displayed virtual desktop based on it.

[0024] Continuing, server 200 also obtains a compensation image list, which includes the first images to be compensated corresponding to the virtual desktop in N consecutive second periods, where N second periods are the first N periods of the first period, and N is an integer greater than 1. If the desktop update image in the first period includes a lossy region, the region overlapping with the lossy region is removed from each first image to be compensated in the compensation image list to obtain the second image to be compensated in each second period. The second image to be compensated in the Nth second period of the compensation image list is encoded to obtain the target image, and the target image is sent to terminal 400. Terminal 400 receives the target image sent by server 200 and performs image compensation on the second image to be compensated in the Nth second period of the virtual desktop based on the target image.

[0025] The image processing method for virtual desktops provided in this application embodiment is implemented by an electronic device. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server working together. The electronic device implementing the image processing method for virtual desktops provided in this application embodiment can be various types of terminals or servers. The server (e.g., server 200) can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal (e.g., terminal 400) can be a laptop, tablet, desktop computer, smartphone, smart voice interaction device (e.g., smart speaker), smart home appliance (e.g., smart TV), smartwatch, in-vehicle terminal, wearable device, virtual reality (VR) device, aircraft, etc., but is not limited to these. The terminal and server can be connected directly or indirectly through wired or wireless communication, and this application embodiment does not impose any restrictions on this.

[0026] The following describes an electronic device implementing a virtual desktop image processing method according to embodiments of this application. See also... Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 500 provided in this embodiment can be a terminal or a server. Figure 2 As shown, electronic device 500 includes at least one processor 510, memory 550, at least one network interface 520, and user interface 530. The various components in electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 540.

[0027] The processor 510 may be an integrated circuit chip with signal processing capabilities. The user interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touchscreen display, camera, other input buttons, and controls. The memory 550 may be removable, non-removable, or a combination thereof. The memory 550 may include one or more storage devices physically located away from the processor 510. The memory 550 may include volatile memory or non-volatile memory, or both.

[0028] In some embodiments, memory 550 is capable of storing data to support various operations. Examples of such data include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below. Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks; network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including Bluetooth, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB); presentation module 553 is used to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 associated with user interface 530 (e.g., a display screen, a speaker, etc.); input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0029] In some embodiments, the image processing apparatus for the virtual desktop provided in this application can be implemented in software. Figure 2 An image processing device 555 for a virtual desktop stored in memory 550 is shown. It may be software in the form of programs and plug-ins, including the following software modules: sending module 5551, acquiring module 5552, removing module 5553, and encoding module 5554. These modules are logical and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0030] The image processing method for a virtual desktop provided in this application embodiment is described below. The image processing method for a virtual desktop provided in this application embodiment is implemented by an electronic device, for example, it can be implemented by a server alone, or by a server and a terminal working together. Therefore, the executing entity of each step will not be described again below. See Figure 3 , Figure 3 This is a first flowchart illustrating the image processing method for a virtual desktop provided in this application embodiment. The image processing method for a virtual desktop provided in this application embodiment includes: Step 101: Send the updated desktop image to be updated in the first cycle to the terminal.

[0031] Step 101 is the core operation where the virtual desktop server (i.e., the server) transmits the desktop update content for the first cycle to the terminal (i.e., the device on which the virtual desktop client runs). Its purpose is to achieve dynamic updates of the virtual desktop on the terminal. The first cycle is the current cycle. Users can run the virtual desktop client through the terminal and use the virtual desktop to perform interactive operations, such as video playback, document operations, file drag-and-drop operations, presentation operations, etc.

[0032] In practical applications, the duration of the first cycle is a fixed sending interval set by the server (which can be adjusted as needed). This sending interval ranges from a minimum interval T2 to a maximum interval T1. The server starts a timer and retrieves the desktop images (i.e., desktop update images) that need to be updated within the first cycle according to this sending interval for processing. For example, the server initially sets the sending interval to 100ms. If subsequent bandwidth is insufficient, the interval is increased to T1 (e.g., 200ms), and if bandwidth is sufficient, it is decreased to T2 (e.g., 50ms). The server maintains a local canvas for drawing the virtual desktop of the virtual machine. Each time, the graphics card driver transmits the image to be updated and its coordinate information (i.e., the desktop update image, a dirty matrix containing one or more rectangular regions) to the server. After receiving it, the server draws the desktop update image onto the local canvas. All desktop update images collected within the first cycle are processed. If multiple desktop update images overlap, they are re-divided into multiple sub-rectangular regions according to the principle of non-overlapping. For example, if three overlapping rectangular desktop update images are collected in the first cycle, the server divides them into five non-overlapping sub-rectangular regions to reduce the amount of data transmitted repeatedly. The server determines the encoding method and quality for the current cycle based on the transmission bandwidth of the previous cycle. Specifically, lossless encoding is prioritized; if the bandwidth consumption of the previous cycle exceeds the bandwidth threshold, lossy encoding is switched to; if the bandwidth of the encoded data still exceeds the bandwidth threshold after using lossy encoding, the encoding quality is reduced (adjusted from high to low); if the bandwidth still exceeds the bandwidth threshold after adjusting the encoding quality, the transmission interval of the first cycle is increased (not exceeding the maximum interval T1); if the amount of updated content (i.e., desktop update image) in the first cycle is lower than the data amount threshold, or the transmission bandwidth does not reach the bandwidth threshold, the transmission interval of the first cycle can be reduced (not lower than the minimum interval T2). The server encodes the updated desktop update data according to the determined encoding method and quality, generates a desktop update image, and sends the desktop update image to the terminal. If the desktop update image in the first cycle includes lossy regions obtained based on lossy encoding, then the lossy regions included in the desktop update image are recorded in the set of images to be compensated (i.e., the compensation image list).

[0033] In some embodiments, before performing step 101 "sending the desktop update image to be updated in the first period to the terminal", the following steps may also be performed: obtaining the desktop update image to be updated in the first period; if there are multiple desktop update images and there is overlap between the multiple desktop update images, determining the smallest image area covering the multiple desktop update images; dividing the smallest image area into multiple non-overlapping sub-image areas; based on this, step 101 "sending the desktop update image to be updated in the first period to the terminal" can be implemented by performing the following steps: sending each sub-image area to the terminal.

[0034] Here, we first collect all desktop update images to be updated within the first cycle of the virtual desktop. Each desktop update image corresponds to a rectangular coordinate region (i.e., the dirty matrix region), which can be represented by the coordinates of the top-left vertex (x1, y1) and the bottom-right vertex (x2, y2), used to identify the position of the desktop update image in the local canvas of the virtual desktop. For example, if three desktop update images to be updated are collected in the first cycle, the corresponding rectangular coordinate regions are: Desktop update image A (x1=10, y1=10, x2=30, y2=30), Desktop update image B (x1=20, y1=20, x2=40, y2=40), and Desktop update image C (x1=15, y1=25, x2=35, y2=35). Then, the overlap of the multiple desktop update images is checked. If the boundaries of two desktop update images intersect (e.g., x2 (30) of desktop update image A is greater than x1 (20) of desktop update image B and x1 (10) of desktop update image A is less than x2 (40) of desktop update image B, and y2 (30) of desktop update image A is greater than y1 (20) of desktop update image B and y1 (10) of desktop update image A is less than y2 (40) of desktop update image B), then the two desktop update images are determined to overlap. For the above example, desktop update image A overlaps with desktop update image B, desktop update image A overlaps with desktop update image C, and desktop update image B overlaps with desktop update image C. Then, the smallest image region covering multiple desktop update images is determined. This smallest image region is the region surrounded by the boundaries of multiple desktop update images. Finally, the smallest image region is divided into multiple non-overlapping sub-image regions. Specifically, the smallest image region is divided into several non-overlapping sub-image regions based on the boundary lines of multiple desktop update images.

[0035] As an example, see Figure 4 , Figure 4 This is a schematic diagram of the segmentation of the smallest image region of the desktop update image provided in the embodiments of this application. Here, the multiple desktop update images include desktop update image A, desktop update image B and desktop update image C, and the smallest image region is 410. The smallest image region 410 is segmented by the vertical boundaries of each of the multiple desktop update images (such as boundary 2-boundary 5) to obtain 5 non-overlapping sub-image regions, namely sub-image region 1, sub-image region 2, sub-image region 3, sub-image region 4 and sub-image region 5.

[0036] Based on this, step 101, "sending the desktop update image to be updated in the first cycle to the terminal", is specified as follows: each sub-image region after the above segmentation is encoded according to an adaptive encoding strategy (i.e., the encoding method and encoding quality are adjusted according to the data bandwidth of the previous cycle, lossless encoding is given priority, lossy encoding is switched to when the bandwidth is insufficient and the encoding quality is reduced, and the processing interval of the first cycle is adjusted if the bandwidth threshold is still exceeded), and each encoded sub-image region is sent to the terminal.

[0037] Applying the above embodiments, 1) by merging overlapping areas (i.e., determining a minimum image area covering multiple desktop update images) and dividing them into non-overlapping sub-regions (i.e., sub-image regions), duplicate transmission of overlapping parts is avoided, reducing the amount of data sent; 2) Virtual desktop scenarios are sensitive to bandwidth, especially dynamic intermediate processes such as file dragging (where users do not need to clearly view process details, but the original data volume is large). Through merging and segmentation processing, the server can significantly reduce bandwidth consumption in dynamic scenarios without affecting user experience, optimize bandwidth usage, and improve the smoothness of virtual desktop transmission; 3) The segmented sub-regions cover the content of all areas to be updated (i.e., desktop update images). After receiving the data, the terminal can completely restore the desktop update content within the first cycle, avoiding the loss of update content due to merging and segmentation, and ensuring update integrity; 4) For dynamic scenarios such as file dragging, users focus on the final result rather than the intermediate process. Transmission of merged and segmented sub-regions can reduce the amount of data while ensuring the completeness of the dynamic process seen by the user, balancing bandwidth optimization and experience consistency, and adapting to the needs of dynamic scenarios.

[0038] In some embodiments, see Figure 5 The step "determine the minimum image region covering multiple desktop update images" can be achieved by executing the following steps 201-204: Step 201, obtain the coordinates of the top-left and bottom-right corners of each desktop update image, where the top-left corner coordinates include the x-coordinate and y-coordinate of the top-left corner, and the bottom-right corner coordinates include the x-coordinate and y-coordinate of the bottom-right corner; Step 202, determine the minimum x-coordinate of the top-left corner from the x-coordinates of the top-left corners of the multiple desktop update images, and determine the minimum y-coordinate of the bottom-right corner from the y-coordinates of the top-left corners of the multiple desktop update images. The top-left vertex ordinate is determined from the bottom-right vertex ordinates of multiple desktop update images; the bottom-right vertex ordinate is determined from the bottom-right vertex ordinates of multiple desktop update images; step 203, the point indicated by the smallest top-left vertex ordinate and the smallest top-left vertex ordinate is used as the top-left vertex, and the point indicated by the largest bottom-right vertex ordinate and the largest bottom-right vertex ordinate is used as the bottom-right vertex, constructing a target rectangular region; step 204, the region that does not overlap with the desktop update image is removed from the target rectangular region to obtain the smallest image region.

[0039] For step 201, obtain the coordinate information (including the coordinates of the top-left and bottom-right corners) of each updated image of the virtual desktop during the first cycle. Each updated image corresponds to a rectangular area, which is uniquely identified by the coordinates of its top-left and bottom-right corners: the top-left corner coordinates include the x-coordinate of the top-left corner (denoted as...). ) and the ordinate of the top left vertex (denoted as ) The coordinates of the lower right corner are used to indicate the position of the upper left corner of the rectangular area in the virtual desktop canvas; the coordinates of the lower right corner vertex include the x-coordinate of the lower right corner vertex (denoted as ). ) and the ordinate of the lower right corner vertex (denoted as ) (), used to indicate the position of the rectangular area in the lower right corner of the virtual desktop canvas.

[0040] For step 202, extreme value extraction is performed on the coordinate information of multiple desktop update images to determine the boundary range covering all desktop update images: from the x-coordinate of the top left vertex of all desktop update images ( The minimum value is selected from all desktops and used as the x-coordinate of the minimum top-left vertex. The y-coordinate of the top-left vertex of the image is updated from all desktops. The minimum value is selected from all desktops and used as the ordinate of the smallest top-left vertex. The x-coordinate of the bottom-right vertex of the image is updated from all desktops. Select the maximum value from the list of images and use it as the x-coordinate of the bottom right vertex. Update the y-coordinate of the bottom right vertex of the image from all desktops. Select the maximum value from the list and use it as the ordinate of the bottom right corner vertex.

[0041] For step 203, a target rectangular region is constructed using the four extreme values ​​extracted in step 202 (i.e., the smallest x-coordinate of the top-left vertex, the smallest y-coordinate of the top-left vertex, the largest x-coordinate of the bottom-right vertex, and the largest y-coordinate of the bottom-right vertex). The coordinates of the top-left vertex of this target rectangular region are (the smallest x-coordinate of the top-left vertex and the smallest y-coordinate of the top-left vertex), and the coordinates of the bottom-right vertex are (the largest x-coordinate of the bottom-right vertex and the largest y-coordinate of the bottom-right vertex). That is, this target rectangular region is the smallest rectangle that can completely cover all the updated desktop images.

[0042] For step 204, remove regions from the target rectangular region that do not overlap with any desktop update images to obtain the smallest image region. Specifically, each sub-region within the target rectangular region can be checked one by one: if the coordinate range of a certain sub-region does not overlap with any desktop update image... arrive The range of the x-coordinate, and arrive If the ranges of the vertical coordinates intersect, the sub-region is determined to be a "non-overlapping region" and removed from the target rectangular region.

[0043] Applying the above embodiments: 1) The minimum image region only includes the coverage area of ​​the original desktop update image, avoiding interference from irrelevant blank areas in the initial rectangular region, ensuring that the subsequently processed area completely corresponds to the content to be updated, and accurately covers the content to be updated. 2) The range of the minimum image region is smaller than the initial rectangular region, reducing the computational burden on the server and reducing the amount of data processing. 3) Since the minimum image region does not contain irrelevant content, redundant data will not be transmitted during subsequent encoding and transmission, optimizing bandwidth utilization efficiency, especially suitable for bandwidth-sensitive dynamic scenarios (such as file dragging). 4) The minimum image region provides an accurate basic range for subsequent "segmentation into non-overlapping sub-regions," ensuring that all segmented sub-regions are related to the original image to be updated, avoiding the generation of invalid sub-regions, and further improving the bandwidth optimization effect.

[0044] In some embodiments, the step "segmenting the minimum image region into multiple non-overlapping sub-image regions" can be achieved by performing the following steps: obtaining the image boundary of each desktop update image, wherein the image boundary is a first boundary of the desktop update image in the horizontal direction or a second boundary of the desktop update image in the vertical direction; determining the target image boundary within the minimum image region from the image boundaries of the multiple desktop update images; and segmenting the minimum image region using each target image boundary as a dividing line to obtain multiple non-overlapping sub-image regions.

[0045] Here, the image boundaries of each desktop update image are first obtained. The image boundaries can be either a first boundary or a second boundary. The first boundary is represented by the horizontal coordinate (x1) of the top-left vertex and the horizontal coordinate (x2) of the bottom-right vertex, indicating the left and right boundaries of the desktop update image in the horizontal direction. The second boundary is represented by the vertical coordinate (y1) of the top-left vertex and the vertical coordinate (y2) of the bottom-right vertex, indicating the top and bottom boundaries of the desktop update image in the vertical direction. Then, the image boundaries of all desktop update images (such as the first boundaries of all desktop update images) are collected, and target image boundaries located within the minimum image region (the coordinate values ​​of the target image boundary do not exceed the range of the minimum image region) are selected. See also... Figure 4The vertical image boundaries of multiple desktop update images include boundaries 1-6, where boundaries 2-5 are located within the minimum image region. Finally, using each target image boundary as a dividing line, the minimum image region is segmented to obtain multiple non-overlapping sub-image regions. Specifically, when the target image boundary is a horizontal boundary, the minimum image region is divided horizontally into multiple continuous intervals (e.g., x=10-15, 15-20, 20-30, 30-35), with each continuous interval representing a sub-image region. When the target image boundary is a vertical boundary, the minimum image region is divided vertically into multiple continuous intervals (e.g., y=10-20, y=20-25, y=25-30, y=30-35), with each continuous interval representing a sub-image region. Figure 4 The sub-image regions 1-5 are shown.

[0046] By applying the above embodiments, the smallest image region is segmented by the boundary line of the desktop update image, ensuring that the sub-image regions do not overlap, avoiding repeated transmission of overlapping parts, and reducing the amount of data sent. At the same time, the sub-image region only contains the content to be updated of the virtual desktop (i.e., multiple desktop update images), which is suitable for dynamic scenarios where the virtual desktop is sensitive to bandwidth (such as file dragging). This reduces bandwidth consumption and improves the smoothness of virtual desktop transmission without affecting the user's perception of the update process.

[0047] In some embodiments, before executing step 101 "send the desktop update image to be updated in the first cycle to the terminal", the following steps may also be performed: obtain the update image data of the virtual desktop in the first cycle, and obtain the transmission data bandwidth of the previous cycle of the first cycle; if the encoding method used in the first cycle is determined to be lossless encoding based on the transmission data bandwidth, perform lossless encoding processing on the update image data to obtain the first desktop update image; if the data bandwidth corresponding to the first desktop update image is less than or equal to a preset bandwidth threshold, use the first desktop update image as the desktop update image; if the data bandwidth corresponding to the first desktop update image is greater than the bandwidth threshold, switch the lossless encoding method to a lossy encoding method, and perform lossy encoding processing on the update image data to obtain the desktop update image.

[0048] Here, the update image data of the virtual desktop in the first cycle is first obtained (i.e., the original pixel data and corresponding coordinate information of the desktop update image to be sent), and the transmission bandwidth of the previous cycle is obtained (i.e., the actual network bandwidth occupied by sending the desktop update image in the previous cycle, which is obtained by the server through the network detection module). For example, the data size of the update image data in the first cycle is 1.2MB, the transmission bandwidth of the previous cycle is 1Mbps, and the server's preset bandwidth threshold is 1.1Mbps. Based on the transmission bandwidth of the previous cycle, the encoding method of the first cycle is determined. For example, if the transmission bandwidth is less than or equal to the bandwidth threshold, the encoding method is lossless encoding; if the transmission bandwidth is greater than the bandwidth threshold, the encoding method is lossy encoding. If the encoding method used in the first cycle is determined to be lossless based on the transmission bandwidth, lossless encoding processing (such as using the LZ77 algorithm) is performed on the update image data of the first cycle to generate the first desktop update image, and the data bandwidth corresponding to the first desktop update image is calculated (i.e., the encoded data size divided by the transmission duration). If the data bandwidth of the first desktop update image is less than or equal to the bandwidth threshold, the first desktop update image is directly used as the final desktop update image to be sent. If the data bandwidth corresponding to the first desktop update image is greater than the bandwidth threshold, the encoding mode switching logic is triggered, and the lossless encoding mode is switched to the lossy encoding mode. At this time, lossy encoding processing is performed on the update image data of the first cycle (such as using the JPEG algorithm with the encoding quality parameter set to 80%) to generate the final desktop update image to be sent.

[0049] By applying the above embodiments, the encoding method is adaptively adjusted based on the transmission bandwidth of the previous cycle. Lossless encoding is prioritized to ensure the image quality of the updated desktop images (such as the clarity of document text and charts). When lossless encoding triggers the bandwidth threshold, it switches to lossy encoding. This controls bandwidth consumption without affecting the smoothness of dynamic scenes (such as file dragging), achieving a balance between image quality and performance in the virtual desktop, where "static areas are clear and dynamic areas are smooth," thus improving the user experience in different scenarios.

[0050] In some embodiments, the step "perform lossy encoding on the updated image data to obtain a desktop update image" can be achieved by performing the following steps: performing lossy encoding on the updated image data according to a first encoding quality parameter to obtain a second desktop update image; if the data bandwidth corresponding to the second desktop update image is greater than a bandwidth threshold, performing lossy encoding on the updated image data according to a second encoding quality parameter to obtain a third desktop update image, wherein the second encoding quality parameter is less than the first encoding quality parameter; if the data bandwidth corresponding to the third desktop update image is less than or equal to the bandwidth threshold, the third desktop update image is used as the desktop update image.

[0051] Here, a decreasing encoding quality parameter is preset. For example, the first encoding quality parameter (e.g., 80%) is higher than the second encoding quality parameter (e.g., 70%). The lower the encoding quality parameter, the higher the image compression rate and the smaller the data volume during lossy encoding, but the greater the image quality loss. Lossy encoding processing (e.g., using the JPEG algorithm) is performed on the updated image data according to the first encoding quality parameter to generate a second desktop update image, and the data bandwidth corresponding to the second desktop update image is calculated. The data bandwidth of the second desktop update image is compared with a bandwidth threshold. If the data bandwidth corresponding to the second desktop update image is less than or equal to the bandwidth threshold, the second desktop update image is directly used as the final desktop update image to be sent. If the data bandwidth corresponding to the second desktop update image is greater than the bandwidth threshold, the encoding quality parameter switching logic is triggered, and the second encoding quality parameter is used for re-encoding. That is, lossy encoding is re-performed on the updated image data according to the second encoding quality parameter (70%) to generate a third desktop update image, and the data bandwidth corresponding to the third desktop update image is calculated. If the data bandwidth corresponding to the third desktop update image is less than or equal to the bandwidth threshold, the third desktop update image is used as the desktop update image. In practical applications, if the data bandwidth corresponding to the third desktop update image is still greater than the bandwidth threshold, the encoding quality parameters can be further reduced until the bandwidth requirement is met (i.e., the data bandwidth is less than or equal to the bandwidth threshold).

[0052] It should be noted that if the data bandwidth is consistently (e.g., the duration reaches the duration threshold) less than the bandwidth threshold, the bandwidth threshold can be used as a target to increase the encoding quality of the third desktop update image, so that its data bandwidth reaches the bandwidth threshold, thereby improving the utilization of network bandwidth and the display effect of the virtual desktop screen.

[0053] By applying the above embodiments, the encoding quality parameters of lossy encoding are adjusted "from high to low" to control the transmission bandwidth while preserving the image quality of the updated desktop image as much as possible (e.g., the image clarity of 80% quality is better than 70%), avoiding excessive loss of image quality. At the same time, the adjustment of the encoding quality parameters is dynamically triggered based on the bandwidth threshold, ensuring the smooth transmission of the virtual desktop in bandwidth-sensitive scenarios (such as file dragging), achieving a balance between "minimizing image quality loss" and "bandwidth control", and improving the user experience in different scenarios.

[0054] In some embodiments, the following steps may also be performed: if the second coding quality parameter is the minimum coding quality parameter and the data bandwidth corresponding to the third desktop update image is greater than the bandwidth threshold, the duration of the first period is increased; if the data bandwidth corresponding to the first desktop update image is less than the bandwidth threshold, the duration of the first period is decreased.

[0055] Here, (1) a minimum coding quality parameter is preset (e.g., 60%, which is the lower limit of "adjusting quality from high to low"). When the second coding quality parameter is the minimum coding quality parameter, lossy coding of the updated image data of the first period is performed on the parameter to generate the third desktop update image. The data bandwidth corresponding to the third desktop update image is calculated (the amount of data after coding divided by the original time length of the first period). If the data bandwidth corresponding to the third desktop update image is greater than the bandwidth threshold, the adjustment logic of the period time length is triggered, that is: increase the time length of the first period (e.g., from 1 second to 1.1 seconds). At this time, the data bandwidth of the third desktop update image is 1.2MB÷1.1 seconds≈1.09Mbps, which meets the requirement of being less than or equal to the bandwidth threshold (1.1Mbps). The time length of the first period after the increase does not exceed the maximum interval T1 specified in advance. (2) Lossless coding is performed on the updated image data of the first period to generate the first desktop update image, and its corresponding data bandwidth is calculated. If the data bandwidth corresponding to the first desktop update image is less than the bandwidth threshold, the cycle length adjustment logic is triggered, that is, the length of the first cycle is reduced (e.g., from 1 second to 0.8 seconds). At this time, the data bandwidth of the first desktop update image is 0.9MB ÷ 0.8 seconds ≈ 1.125Mbps, which is close to the bandwidth threshold and can reduce the waste of data bandwidth. The length of the reduced first cycle is not less than the pre-defined minimum interval T2.

[0056] Applying the above embodiments, a three-layer bandwidth control logic of "encoding method → ​​encoding quality → cycle duration" is formed by adjusting the cycle length: when the bandwidth of the data encoded by the minimum encoding quality parameter is still greater than the bandwidth threshold, the cycle duration is increased to reduce the sending frequency of desktop update images, reduce total bandwidth consumption, and ensure smooth transmission; when the lossless encoding bandwidth is sufficient, the cycle duration is decreased to increase the sending frequency of desktop update images, enhancing dynamic response. This collaborative mechanism further balances the image quality, bandwidth, and smoothness of the virtual desktop, adapts to different needs in high / low bandwidth scenarios, and improves the flexibility of the solution and user experience.

[0057] Step 102: Obtain the compensation image linked list.

[0058] The compensation image list includes the first image to be compensated corresponding to the virtual desktop in each of the N second cycles. The N second cycles are the first N consecutive cycles of the first cycle, and N is an integer greater than 1.

[0059] Step 102 involves obtaining the core data structure for managing long-term static area compensation for virtual desktops—the compensation image linked list. Its purpose is to provide foundational data for subsequent dynamic cleaning of areas to be compensated and accurate compensation of long-term static areas. The compensation image linked list is an ordered data structure storing the images to be compensated for a virtual desktop within N second cycles (i.e., the N consecutive transmission cycles preceding the first cycle (the current cycle), where N is an integer greater than 1). Each second cycle corresponds to a node in the compensation image linked list, and the node stores the first image to be compensated for that second cycle (i.e., the lossy area included in the desktop update image transmitted via lossy encoding within that second cycle). In the compensation image linked list, the N second cycles are arranged in chronological order, meaning the first second cycle in the compensation image linked list is the latest, and the Nth second cycle is the earliest.

[0060] First, the number of compensation periods N is preset (e.g., N=3, where N is an integer greater than 1, meaning the compensation period includes N transmission intervals), and the compensation image linked list is initialized. Specifically, this compensation image linked list is an ordered linear structure containing N linked list nodes (i.e., nodes), where each node corresponds to a second period (i.e., the first N consecutive transmission periods of the first period (the current period). For example, if the first period (the current period) is period t, then the second period is period t). 1. t 2、…、t N cycles). For each transmission cycle (including the first cycle and each second cycle), at the end of the transmission cycle, the lossy regions (obtained through lossy encoding) included in the desktop update images transmitted in that transmission cycle are stored as the image regions to be compensated in the compensation image linked list corresponding to the transmission cycle. Following the linked list rotation rule, it is ensured that the compensation image linked list always stores the image regions to be compensated from the previous N consecutive second cycles of the current cycle. Specifically, for cycle t, the tail node of the compensation image linked list is obtained. If the tail node is not empty, the image regions to be compensated recorded in the tail node are obtained, and lossless or high-quality encoding is performed on the image regions to be compensated in the tail node, along with the transmission of the encoding results; then the tail node of the compensation image linked list is cleared, and the lossy regions included in the desktop update images transmitted in cycle t are copied into the cleared tail node (i.e., the original node 3); finally, all nodes of the compensation image linked list are rotated sequentially, i.e., the tail node of the compensation image linked list (corresponding to cycle t) is rotated sequentially. The Nth period) is moved to the head of the linked list, and the original 1 to Nth periods are moved to the head of the One node moves sequentially to the next (e.g., when N=3, the compensation image linked list before rotation is [node 1 (t-th node)]). 1st cycle), Node 2 (tth) 2 periods), node 3 (t-th) [3 cycles]; After rotation, it becomes [node 3 (corresponding to the t-th node before rotation)]. 3 periods), node 1 (t-th) 1st cycle), Node 2 (tth) (2 cycles) After the rotation is completed, the compensation image linked list for the t-th cycle is updated.

[0061] Step 103: If the desktop update image includes a lossy region, remove the region that overlaps with the lossy region from each first image to be compensated in the compensation image chain to obtain a second image to be compensated in each second period.

[0062] Step 103 is the core operation of dynamically cleaning up the images to be compensated in the compensation image list. Its purpose is to ensure that the images to be compensated in the compensation image list are long-term static images that have not been covered by recent desktop update images. Here, "lossy region" refers to the image region included in the desktop update images sent in the first cycle, obtained through lossy encoding; "first image to be compensated" refers to the images to be compensated for each of the second cycles (the first N cycles of the first cycle) stored in the compensation image list; and "second image to be compensated" refers to the remaining image after removing the overlapping portion of the first image to be compensated with the lossy region of the current cycle (i.e., the first cycle). Thus, by removing the overlapping region with the lossy region from each first image to be compensated in the compensation image list, a compensation image list including the second image to be compensated for each second cycle is obtained. Subsequent rotation of the compensation image list is implemented based on the compensation image list including the second image to be compensated for each second cycle.

[0063] Here, we first determine whether the desktop update image in the first cycle contains lossy regions (i.e., whether the desktop update image in the first cycle includes image regions obtained using lossy encoding). If it does, we obtain the region information corresponding to the lossy region (represented by rectangular coordinates, such as the coordinates of the top left vertex of region D (x1=25, y1=25) and the coordinates of the bottom right vertex (x2=35, y2=35)). We then traverse all nodes in the compensation image linked list (each node corresponds to a first image to be compensated in the second cycle) and perform the following operations for each first image to be compensated: calculate the overlapping area between the first image to be compensated and the lossy region in the current cycle; remove the overlapping area from the first image to be compensated (i.e., remove the lossy region) to obtain the second image to be compensated.

[0064] Step 104: Encode the second image to be compensated in the Nth second period of the compensation image chain to obtain the target image, and send the target image to the terminal.

[0065] The target image is used to perform image compensation on the second image to be compensated in the Nth second period of the virtual desktop.

[0066] Step 104 is the core operation for achieving long-term static image quality compensation in the virtual desktop. Its purpose is to repair the image blurring problem caused by lossy encoding of long-term static images on the terminal by sending a target image. Here, "the second image to be compensated in the Nth second cycle" refers to the image to be compensated stored in the last node of the compensation image chain (corresponding to the oldest uncovered image in the first N cycles of the first cycle); "target image" refers to the image generated after encoding the second image to be compensated in the Nth second cycle; "image compensation" means that the terminal replaces the original lossy encoded image with the target image to restore the clarity of the static image (such as document text and charts).

[0067] It should be noted that the "encoding process" in step 104 includes lossless encoding and high-quality encoding, so that the image quality of the obtained target image is higher than that of the second image to be compensated in the Nth second period, so as to achieve image compensation for the second image to be compensated in the Nth second period through the target image. Among them, the target image obtained by lossless encoding, after compression and decompression, is completely consistent with the original image in terms of bits, without any information loss, and can better achieve image compensation; high-quality encoding is a lossy encoding, which greatly reduces the amount of data by discarding information that is not sensitive to the human eye (such as slight color differences), but ensures that the visual effect is very close to the original image, even reaching "visual lossless", which can save bandwidth and ensure the smooth implementation of image compensation under limited network bandwidth.

[0068] Specifically, based on the structure of the compensation image linked list (containing N nodes, where N is the compensation period), the region to be compensated stored in the last node of the compensation image linked list is selected, namely, the "second image to be compensated in the Nth second period" (corresponding to the oldest region in the first N periods that has not been covered by recent updates). If the last node of the compensation image linked list is not empty, that is, if the last node of the compensation image linked list stores a region to be compensated, then encoding processing is performed on the region to be compensated stored in the last node, that is: encoding processing is performed on the second image to be compensated in the Nth second period (such as using a lossless image compression algorithm, such as PNG encoding), to generate the target image. This process retains the original data of all pixels in the second image to be compensated in the Nth second period, ensuring that the image clarity is consistent with the original desktop content (that is, obtaining the original images of these regions from the local canvas and performing lossless or high-quality encoding). The target image is sent to the virtual desktop client running on the terminal, and the coordinate information of the region corresponding to the target image is transmitted synchronously, so that the virtual desktop client of the terminal can accurately locate the screen position to be compensated. After receiving the target image and coordinate information, the virtual desktop client of the terminal performs the following operations: a) Based on the coordinate information, locate the target area to be compensated in the virtual desktop; b) Replace the original damaged area of ​​the target area with the received target image (the target area has problems of blurriness and loss of details due to lossy encoding); c) Refresh the display to restore the original clarity of the compensated target area (such as the text in the document becoming clear and legible from blurry).

[0069] In some embodiments, after performing the step "send target image to terminal", the following steps may also be performed: remove the second image to be compensated in the Nth second period from the compensation image chain to obtain the target compensation image chain; update the target compensation image chain based on the lossy region of the first period.

[0070] Here, the second image to be compensated for in the second period (i.e., the one stored in the node corresponding to the oldest period in the compensation image list) is removed from the compensation image list. For example, if the compensation period N=3, the compensation image list originally contains 3 nodes, corresponding to the second images to be compensated for in periods t-1, t-2, and t-3, respectively; the node in period t-3 (the oldest) is removed, resulting in the target compensation image list. The target compensation image list is then updated based on the lossy region of the first period (i.e., the region in the desktop update image sent in the first period that is encoded using lossy encoding).

[0071] By applying the above embodiments, the rotation mechanism of "removing the oldest node + adding a new node" ensures that the compensation image list always stores the images to be compensated for the most recent N periods, avoiding long-term storage of outdated information and optimizing storage space. At the same time, the lossy areas in the current period (i.e., the first period) are dynamically tracked, providing an accurate data basis for subsequent image quality compensation of long-term static areas that have not been updated for N periods. This achieves closed-loop management of "storage-update-compensation", further balancing the image quality and bandwidth consumption of the virtual desktop.

[0072] In some embodiments, the compensation image linked list includes N linked list nodes, where the nth linked list node is used to store the second image to be compensated in the nth second period, and n is an integer greater than 0 and less than or equal to N. Based on this, "removing the first image to be compensated in the Nth second period from the compensation image linked list to obtain the target compensation image linked list" can be achieved by performing the following steps: clearing the second image to be compensated in the Nth second period stored in the Nth linked list node from the compensation image linked list to obtain the target compensation image linked list. Based on this, "updating the target compensation image linked list based on the lossy region of the first period" can be achieved by performing the following steps: copying the lossy region of the first period to the cleared Nth linked list node; moving the Nth linked list node storing the lossy region of the first period to the head of the target compensation image linked list as the first node of the linked list, and moving all linked list nodes in the target compensation image linked list except for the Nth linked list node sequentially to the right.

[0073] Here, the compensation image linked list is an ordered linear structure containing N linked list nodes, and the nth linked list node is used to store the second image to be compensated in the nth second period (n is an integer of 1≤n≤N).

[0074] Based on this, (1) the Nth linked list node is cleared to obtain the target compensation image linked list. Specifically, the Nth linked list node of the compensation image linked list (i.e., the last node of the linked list, corresponding to the oldest (earliest) second image to be compensated in the second period) is located, and the image data stored in this node is cleared. For example, if the compensation period N=3, the compensation image linked list contains node 1 (stores the second image to be compensated A in the first second period), node 2 (stores the second image to be compensated B in the second second period), and node 3 (stores the second image to be compensated C in the third second period); the second image to be compensated C in node 3 is cleared, and at this time the linked list still retains 3 nodes (node ​​1, node 2, and empty node 3) to obtain the target compensation image linked list.

[0075] (2) Copy the lossy region of the first cycle (e.g., region D: x1=25, y1=25, x2=35, y2=35) to the last node after clearing (e.g., node 3 after clearing in the example above). At this time, the node structure of the linked list is: node 1 (stores the second image to be compensated in the first second cycle A), node 2 (stores the second image to be compensated in the second second cycle B), and node 3 (stores the lossy region of the first cycle), still maintaining a fixed size of N=3 nodes.

[0076] (3) Move the Nth linked list node (node ​​3) containing the first period of the lossy region from the end of the linked list to the head of the linked list to become the new head node of the linked list. Nodes 1 and 2 in the original linked list are moved to the right in sequence. At this time, the node structure of the linked list is: node 3 (containing the first period of the lossy region), node 1 (containing the first second period of the second image to be compensated A), and node 2 (containing the second second period of the second image to be compensated B), while maintaining the fixed size of N=3 nodes.

[0077] By applying the above embodiments, the mechanism of "clearing nodes - storing new data - rotating positions" maintains the number of nodes in the compensation image linked list at a constant N, avoiding the performance overhead of frequently creating / removing nodes. Simultaneously, the node storing the lossy region for the current period is moved to the head of the linked list, ensuring that the image to be compensated for the current period will be rotated to the end of the linked list (the oldest node) after N periods, triggering image compensation. This achieves closed-loop management of "lossy region storage - update - compensation." This mechanism optimizes the utilization efficiency of storage resources and ensures accurate compensation of long-term static areas (i.e., image areas that have not been updated for N periods), further balancing the image quality and bandwidth consumption of the virtual desktop.

[0078] In some embodiments, "copying the lossy region of the first period to the cleared Nth linked list node" can be achieved by performing the following steps: copying all image regions of the lossy region to the cleared Nth linked list node; or copying the target image region included in the lossy region to the cleared Nth linked list node, wherein the target image region is the image region in the lossy region whose image quality is lower than the image quality threshold.

[0079] Here, (1) the entire image region of the lossy region in the first period is directly copied to the cleared Nth linked list node. For example, the lossy region in the first period corresponds to region D (the coordinates of the upper left vertex (x1=25, y1=25) and the coordinates of the lower right vertex (x2=35, y2=35)), and the entire range (25-35, 25-35) of region D is copied to the cleared Nth linked list node. (2) The region in the lossy region whose image quality is lower than the preset threshold is taken as the target image region, and the target image region is taken as the image to be compensated in the first period. Specifically, the image quality threshold is preset (e.g., peak signal-to-noise ratio PSNR=30dB, the lower the PSNR, the worse the image quality), and the image quality is calculated for each sub-region of the lossy region; the sub-region whose image quality is lower than the threshold (e.g., D1) is selected as the target image region, and the target image region (D1) is copied to the cleared Nth linked list node.

[0080] Applying the above embodiments, two flexible options are provided: the full-area approach is simple to implement, covering all lossy areas and ensuring no omissions; the target area approach reduces the size of the area to be compensated through image quality filtering, optimizing storage space and subsequent bandwidth consumption. These two methods adapt to different scenarios (such as those prioritizing efficiency or accuracy), further balancing the flexibility and resource consumption of virtual desktop image quality compensation.

[0081] By applying the above embodiments of this application, a compensation image list for the first N consecutive second periods of the first period of the virtual desktop is obtained. When the desktop update image in the first period contains a lossy region, the region overlapping with the lossy region is removed from each first image to be compensated in the compensation image list. Then, the second image to be compensated in the Nth second period of the compensation image list is encoded and sent to the terminal to perform image compensation on the second image to be compensated in the Nth second period of the virtual desktop. In this way, by removing the static region (i.e., non-updated image) covered by the lossy region of the desktop update image in each first image to be compensated, the amount of data of the target image used for image compensation is reduced, and bandwidth control is optimized to ensure smooth data transmission. At the same time, by recording the first image to be compensated in each second period before the first period in the compensation image list, and performing image compensation only on the second image to be compensated in the Nth second period, lossless compensation is achieved for long-term static regions to restore image quality, ensuring a balance between the smoothness of the virtual desktop update region and the clarity of the static region, and improving the image compensation effect of the virtual desktop.

[0082] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0083] Cloud desktop (or virtual desktop) technology provides users with the ability to access cloud computing environments on local devices. Users can establish a secure connection with the cloud desktop server by installing cloud desktop client software on their local terminal or on a thin client. Both parties transmit data in real time through an efficient remote desktop protocol: the server is responsible for obtaining the screen image of the cloud desktop, encoding it, and then transmitting it to the client in a streaming manner; the client receives the data stream and decodes it, and finally presents the clear cloud desktop image to the user through the local display system. Screen content encoding is a core element affecting user experience. Currently, two main strategies are used: (1) Lossy encoding: Its advantage is a high compression ratio, which can effectively save bandwidth resources. However, its cost is the loss of image quality, which may lead to blurry text, unclear details, and affect visual clarity. (2) Lossless encoding: Its advantage is lossless fidelity, which ensures that the image quality is consistent with the original image. However, its compression efficiency is low, which will occupy significantly more network bandwidth and may affect the real-time performance and smoothness of operation. Therefore, in cloud desktop applications, lossy compression or a combination of lossy and lossless compression is often used to improve the experience. Lossy encoding results in a noticeable loss of image quality, especially in non-natural scenes. Therefore, lossless compensation is needed for the areas transmitted by lossy encoding.

[0084] In virtual desktop technology, lossless compression provides optimal image quality under unrestricted conditions. However, in practical applications, encoding / decoding capabilities and bandwidth are limited. To improve operational smoothness, lossy compression is often introduced to compress virtual desktop images before transmitting them to the client for decoding and display. Lossy encoding results in noticeable visual degradation, especially for text and charts in office scenarios, often failing to meet business clarity requirements. Therefore, this application proposes an image compensation scheme for lossy images, applicable to various complex scenarios such as video playback, file dragging, window dragging, document operations, and presentation (PPT) playback. In this application's embodiments, a fixed processing interval is designed as the sending interval (i.e., sending period), and the interval size can be adjusted as needed. The following terms are included in this application's embodiments and are subject to the following interpretations: (1) Compensation period: that is, it includes N transmission intervals, and the size of N can be adjusted as needed.

[0085] (2) Update area (i.e., the desktop update image mentioned above): that is, the dirty matrix area, which records the set of coordinates of the virtual desktop that needs to be updated on the local server, and is distributed in rectangular areas, containing one or more rectangular areas.

[0086] (3) Merging and updating regions (i.e., minimum image regions): For rectangular regions transmitted within a transmission cycle, the merged updated region is used to obtain multiple sub-image regions if the multiple rectangular regions included in the updated region overlap. Figure 4 As shown.

[0087] (4) Lossy region: The update region is transmitted after encoding. If it is transmitted with lossy encoding, it will be recorded in the lossy region. Since the encoding and transmission are not completed immediately after the transmission time arrives, the actual content in the lossy region is the update region of the previous transmission cycle or a subset thereof.

[0088] (5) Area to be compensated (i.e. image to be compensated): The set of rectangular areas of the target image that need to be compensated.

[0089] (6) Compensation region linked list: contains N elements, which are used to record the regions to be compensated in the 1st to Nth transmission cycles. Each time the transmission time is reached, in addition to sending the merged update region in the current transmission cycle, the target image for compensating the region to be compensated is also sent.

[0090] In this embodiment, the compensation period is N transmission cycles. A compensation region linked list is created, containing N nodes, each used to record the regions to be compensated in the first to N transmission cycles. Each time the transmission time arrives, in addition to sending the update regions to be updated within the current transmission cycle, the target image for compensating the regions to be compensated is also sent. When the update region includes a lossy region, the lossy region is removed from all regions to be compensated, so that the regions in the regions to be compensated represent regions that have not been updated within the N transmission cycles.

[0091] As an example, the compensation period N is 3, and the compensation area linked list contains 3 nodes, each of which stores the area to be compensated for one sending period.

[0092] like Figure 6 As shown, before and after the timer processing in the 0th cycle: the update region 61 includes a merged update region 611 obtained by merging multiple overlapping update regions and a single update region 612. At this time, sending the update region will not immediately generate a lossy region, so the lossy region and the regions to be compensated 1-3 are all empty.

[0093] like Figure 7 As shown in the middle part (1), before and after the timer processing in the first cycle: the update area includes the merged update area 62, the lossy area is the update area 61 sent in the 0th cycle, and the areas to be compensated 1-3 are empty; as Figure 7As shown in the middle part (2), the damaged area (i.e. 61) is stored in node 1 and recorded as the area to be compensated 1 (i.e., as the first node of the linked list).

[0094] like Figure 8 As shown in the middle part (1), before and after the timer processing in the second cycle: the area to be compensated 1 is 61 stored in node 1, the updated area is updated area 63, and the damaged area is the merged updated area 62 of the first cycle; 62 is removed from the area to be compensated 1 (i.e., 61 stored in node 1) to obtain area 64 (stored in node 1); as Figure 8 As shown in the middle part (2), node 1 (which stores 64) is moved to the area to be compensated 2; the damaged area (i.e., 62) is stored in node 3 (i.e., the blank or cleared tail node), and node 3 is moved to the area to be compensated 1 (i.e., as the first node of the linked list).

[0095] like Figure 9 As shown in the middle part (1), before and after the timer processing in the third cycle: the area to be compensated 1 stores 62, the area to be compensated 2 stores 64, the updated area is the updated area 65, and the damaged area is the updated area 63 of the second cycle; the damaged area (i.e., 63) is removed from the area to be compensated 1 (i.e., 62 stored in node 3) to obtain area 66 (stored in node 3); the damaged area (i.e., 63) is removed from the area to be compensated 2 (i.e., 64 stored in node 1) to obtain area 67 (stored in node 1); as Figure 9 As shown in the middle part (2), the damaged area (i.e., 63) is stored in node 2 (i.e., the blank or cleared tail node), and node 2 is moved to the compensation area 1 (i.e., as the first node of the linked list); node 1 (stored with 67) is moved to the compensation area 3; node 3 (stored with 66) is moved to the compensation area 2.

[0096] like Figure 10 As shown in the middle part (1), before and after the timer processing in the 4th cycle: the area to be compensated 1 stores 63, the area to be compensated 2 stores 66, the area to be compensated 3 stores 67, the update area is empty, and the lossy area is the update area 65 of the 3rd cycle; the lossy area (i.e., 65) is removed from the area to be compensated 1 (i.e., 63 stored in node 2) to obtain area 68 (stored in node 2); the lossy area (i.e., 65) is removed from the area to be compensated 2 (i.e., 66 stored in node 3) to obtain area 69 (stored in node 3); the lossy area (i.e., 65) is removed from the area to be compensated 3 (i.e., 67 stored in node 1) to obtain the first area (not in Figure 10 As shown in the diagram, it is still stored in node 1); as Figure 10 As shown in the middle part (2), the first region where node 1 is stored (not in) Figure 10(As shown in the diagram) Clear the node and store the damaged area (i.e., 65) into the cleared node 1; move node 1, which stores the damaged area (i.e., 65), to the compensation area 1 (i.e., as the first node of the linked list); move node 3 (which stores 69) to the compensation area 3; move node 2 (which stores 68) to the compensation area 2.

[0097] The key steps of this application include: Step 1, Sending and Resetting (Processing of the Current Period): Sending the merged update region; recording the lossy regions with lossy encoding in the merged update region (for use in the next period). Step 2, Cleaning the Compensation Region Linked List: Traversing all nodes (CR1, CR2, CR3); removing regions overlapping with lossy regions from each node, thereby removing recently updated regions; Step 3, Compensating the Oldest Region: Losslessly encoding the region to be compensated in the tail node CR3 of the linked list to obtain the target image; sending the target image to the client; clearing the region to be compensated in CR3 (the node becomes empty), and copying the lossy region of the current period to the cleared CR3. Step 4, Linked List Rotation: CR3, which stores the lossy region of the current period, is moved to the head of the linked list as the first node, and all node indices are moved sequentially, so that: Before rotation: [Original CR1] → [Original CR2] → [Original CR3]; After rotation: [CR3, which stores the lossy region of the current period] → [CR1, which removes the lossy region of the current period] → [CR2, which removes the lossy region of the current period]. This completes the rotation of the compensation nodes.

[0098] The compensation period N can be dynamically modified, involving the addition or removal of elements from the compensation region's linked list. Increasing the compensation period increases the number of linked list elements by simply adding an empty node at the end of the list; decreasing the compensation period decreases the number of linked list elements by merging the tail node forward. The compensation time for the target image can be determined by adjusting both the compensation node length and the processing interval.

[0099] By applying the above embodiments of this application, accurate compensation is achieved, compensating only areas that have not been updated for N consecutive cycles; dynamic updates are achieved, removing the most recently updated areas in real time; a linked list structure is used to implement a region lifecycle management mechanism; recently updated areas are not repeatedly compensated, further optimizing bandwidth; and lossless images are sent to long-term static areas to ensure image quality.

[0100] The following describes an exemplary structure of the image processing apparatus 555 for a virtual desktop provided in this application embodiment as a software module. In some embodiments, such as Figure 2As shown, the software modules in the image processing device 555 of the virtual desktop stored in the memory 550 may include: a sending module 5551, used to send the desktop update image to be updated in the first cycle of the virtual desktop to the terminal; an acquisition module 5552, used to acquire a compensation image list, wherein the compensation image list includes the first images to be compensated corresponding to the virtual desktop in N second cycles, the N second cycles being the first N consecutive cycles of the first cycle, and N being an integer greater than 1; a removal module 5553, used to remove the region overlapping with the lossy region from each of the first images to be compensated in the compensation image list if the desktop update image includes a lossy region, to obtain a second image to be compensated in each second cycle; and an encoding module 5554, used to encode the second image to be compensated in the Nth second cycle of the compensation image list to obtain a target image, and send the target image to the terminal, wherein the target image is used to perform image compensation on the second image to be compensated in the Nth second cycle of the virtual desktop.

[0101] In some embodiments, the sending module 5551 is further configured to: acquire the desktop update image to be updated in the first period of the virtual desktop before sending the desktop update image to be updated in the first period to the terminal; if there are multiple desktop update images and there is overlap between the multiple desktop update images, determine the smallest image region covering the multiple desktop update images; divide the smallest image region into multiple non-overlapping sub-image regions; the sending module 5551 is further configured to send each of the sub-image regions to the terminal.

[0102] In some embodiments, the sending module 5551 is further configured to obtain the coordinates of the top-left corner vertex and the coordinates of the bottom-right corner vertex of each desktop update image, wherein the coordinates of the top-left corner vertex include the x-coordinate and y-coordinate of the top-left corner vertex, and the coordinates of the bottom-right corner vertex include the x-coordinate and y-coordinate of the bottom-right corner vertex; determine the minimum x-coordinate of the top-left corner vertex from the x-coordinates of the top-left corner vertex of the plurality of desktop update images, determine the minimum y-coordinate of the top-left corner vertex from the y-coordinates of the top-left corner vertex of the plurality of desktop update images, and ... The maximum lower-right corner x-coordinate is determined from the x-coordinates of the lower-right corner vertices of the new image, and the maximum lower-right corner y-coordinate is determined from the y-coordinates of the lower-right corner vertices of the multiple desktop update images; the point indicated by the minimum upper-left corner x-coordinate and the minimum upper-left corner y-coordinate is taken as the upper-left corner vertex, and the point indicated by the maximum lower-right corner x-coordinate and the maximum lower-right corner y-coordinate is taken as the lower-right corner vertex, constructing a target rectangular region; the region that does not overlap with the desktop update image is removed from the target rectangular region to obtain the minimum image region.

[0103] In some embodiments, the sending module 5551 is further configured to acquire the image boundary of each desktop update image, wherein the image boundary is a first boundary of the desktop update image in the horizontal direction or a second boundary of the desktop update image in the vertical direction; determine the target image boundary within the minimum image region from the image boundaries of the plurality of desktop update images; and divide the minimum image region by using each target image boundary as a dividing line to obtain a plurality of non-overlapping sub-image regions.

[0104] In some embodiments, the encoding module 5554 is further configured to, after sending the target image to the terminal, remove the second image to be compensated in the Nth second period from the compensation image list to obtain the target compensation image list; and update the target compensation image list based on the lossy region in the first period.

[0105] In some embodiments, the compensation image linked list includes N linked list nodes, the nth linked list node is used to store the second image to be compensated in the nth second period, where n is an integer greater than 0 and less than or equal to N; the encoding module 5554 is further used to clear the second image to be compensated in the Nth second period stored in the Nth linked list node from the compensation image linked list to obtain the target compensation image linked list, wherein the second image to be compensated stored in the Nth period is a region that has not changed after N periods; the encoding module 5554 is further used to copy the lossy region of the first period to the cleared Nth linked list node; move the Nth linked list node storing the lossy region of the first period to the head of the target compensation image linked list as the first node of the linked list, and move the linked list nodes in the target compensation image linked list other than the Nth linked list node sequentially to the right.

[0106] In some embodiments, the sending module 5551 is further configured to: obtain update image data of the virtual desktop in the first period before sending the desktop update image to be updated in the first period to the terminal; obtain the sending data bandwidth of the previous period of the first period; if the encoding method used in the first period is determined to be lossless encoding based on the sending data bandwidth, perform lossless encoding processing on the update image data to obtain a first desktop update image; if the data bandwidth corresponding to the first desktop update image is less than or equal to a preset bandwidth threshold, use the first desktop update image as the desktop update image; and if the data bandwidth corresponding to the first desktop update image is less than the bandwidth threshold, reduce the time length of the first period; if the data bandwidth corresponding to the first desktop update image is greater than the bandwidth threshold, switch the lossless encoding method to a lossy encoding method, and perform lossy encoding processing on the update image data to obtain the desktop update image.

[0107] In some embodiments, the sending module 5551 is further configured to perform lossy encoding processing on the updated image data according to the first encoding quality parameter to obtain a second desktop update image; if the data bandwidth corresponding to the second desktop update image is greater than the bandwidth threshold, perform lossy encoding processing on the updated image data according to the second encoding quality parameter to obtain a third desktop update image, wherein the second encoding quality parameter is less than the first encoding quality parameter; if the data bandwidth corresponding to the third desktop update image is less than or equal to the bandwidth threshold, use the third desktop update image as the desktop update image.

[0108] In some embodiments, the sending module 5551 is further configured to increase the duration of the first period if the second coding quality parameter is the minimum coding quality parameter and the data bandwidth corresponding to the third desktop update image is greater than the bandwidth threshold.

[0109] It should be noted that the description of the device embodiments in this application is similar to the description of the method embodiments described above, and has similar beneficial effects as the method embodiments, so it will not be repeated here. Any technical details not covered in the image processing device for the virtual desktop provided in the embodiments of this application can be understood based on the description of the technical details in the above method embodiments.

[0110] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the image processing method for the virtual desktop provided in this application.

[0111] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0112] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0113] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0114] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0115] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image processing method for a virtual desktop, characterized in that, The method includes: Send the updated desktop image to be updated in the first cycle to the terminal; Obtain a compensation image list, wherein the compensation image list includes the first image to be compensated corresponding to the virtual desktop in N second periods, and the N second periods are the first N consecutive periods of the first period, where N is an integer greater than 1; If the desktop update image includes a lossy region, remove the region that overlaps with the lossy region from each of the first images to be compensated in the compensation image chain to obtain a second image to be compensated for each second period; The second image to be compensated in the Nth second period of the compensation image chain is encoded to obtain a target image, and the target image is sent to the terminal. The target image is used to perform image compensation on the second image to be compensated in the Nth second period of the virtual desktop.

2. The method as described in claim 1, characterized in that, Before sending the updated desktop image to be updated in the first cycle to the terminal, the method further includes: Obtain the desktop update image to be updated for the virtual desktop within the first period; If there are multiple desktop update images and there is overlap among the multiple desktop update images, determine the smallest image region that covers the multiple desktop update images; The minimum image region is divided into multiple non-overlapping sub-image regions; Sending the updated desktop image to be updated in the first cycle to the terminal includes: Each of the sub-image regions is sent to the terminal.

3. The method as described in claim 2, characterized in that, Determining the minimum image region covering the plurality of desktop update images includes: Obtain the coordinates of the top-left and bottom-right corners of each of the desktop update images, wherein the top-left corner coordinates include the x-coordinate and y-coordinate of the top-left corner, and the bottom-right corner coordinates include the x-coordinate and y-coordinate of the bottom-right corner. The minimum x-coordinate of the top-left vertex is determined from the x-coordinates of the top-left vertices of the multiple desktop update images; the minimum y-coordinate of the top-left vertex is determined from the y-coordinates of the top-left vertices of the multiple desktop update images; the maximum x-coordinate of the bottom-right vertex is determined from the x-coordinates of the bottom-right vertices of the multiple desktop update images; and the maximum y-coordinate of the bottom-right vertex is determined from the y-coordinates of the bottom-right vertices of the multiple desktop update images. The target rectangular region is constructed by using the point indicated by the x-coordinate and y-coordinate of the smallest top-left vertex as the top-left vertex, and the point indicated by the x-coordinate and y-coordinate of the largest bottom-right vertex as the bottom-right vertex. The minimum image region is obtained by removing the region that does not overlap with the desktop update image from the target rectangular region.

4. The method as described in claim 2, characterized in that, The step of dividing the minimum image region into multiple non-overlapping sub-image regions includes: Obtain the image boundary of each of the desktop update images, wherein the image boundary is a first boundary of the desktop update image in the horizontal direction or a second boundary of the desktop update image in the vertical direction; From the image boundaries of the multiple desktop update images, determine the target image boundary located within the smallest image region; Using the boundary of each target image as a dividing line, the smallest image region is segmented to obtain multiple non-overlapping sub-image regions.

5. The method as described in claim 1, characterized in that, After sending the target image to the terminal, the method further includes: Remove the second image to be compensated from the Nth second period from the compensated image list to obtain the target compensated image list; The target compensation image list is updated based on the lossy region of the first period.

6. The method as described in claim 5, characterized in that, The compensation image linked list includes N linked list nodes, and the nth linked list node is used to store the second image to be compensated in the nth second period, where n is an integer greater than 0 and less than or equal to N; The step of removing the second image to be compensated in the Nth second period from the compensated image linked list to obtain the target compensated image linked list includes: From the compensation image linked list, clear the second image to be compensated stored in the Nth linked list node in the Nth second cycle to obtain the target compensation image linked list. The second image to be compensated stored in the Nth second cycle is the region that has not changed after N cycles. The step of updating the target compensation image linked list based on the lossy region of the first period includes: Copy the lossy region of the first cycle to the cleared Nth linked list node; The Nth linked list node storing the damaged region of the first period is moved to the head of the target compensation image linked list to serve as the first node of the linked list, and all linked list nodes in the target compensation image linked list except for the Nth linked list node are moved sequentially to the right.

7. The method as described in claim 1, characterized in that, Before sending the updated desktop image to be updated in the first cycle to the terminal, the method further includes: Obtain the updated image data of the virtual desktop in the first period, and obtain the transmission data bandwidth of the previous period of the first period; If the encoding method used in the first period is determined to be lossless encoding based on the transmission data bandwidth, the updated image data is processed by lossless encoding to obtain the first desktop updated image; If the data bandwidth corresponding to the first desktop update image is less than or equal to a preset bandwidth threshold, the first desktop update image is used as the desktop update image; and if the data bandwidth corresponding to the first desktop update image is less than the bandwidth threshold, the duration of the first period is reduced. If the data bandwidth corresponding to the first desktop update image is greater than the bandwidth threshold, the lossless encoding method is switched to a lossy encoding method, and the update image data is subjected to lossy encoding processing to obtain the desktop update image.

8. The method as described in claim 7, characterized in that, The step of performing lossy encoding processing on the updated image data to obtain the updated desktop image includes: The updated image data is subjected to lossy encoding processing according to the first encoding quality parameter to obtain the second desktop updated image; If the data bandwidth corresponding to the second desktop update image is greater than the bandwidth threshold, the update image data is subjected to lossy encoding processing according to the second encoding quality parameter to obtain the third desktop update image, wherein the second encoding quality parameter is less than the first encoding quality parameter; If the data bandwidth corresponding to the third desktop update image is less than or equal to the bandwidth threshold, the third desktop update image is used as the desktop update image.

9. The method as described in claim 8, characterized in that, The method further includes: If the second encoding quality parameter is the minimum encoding quality parameter, and the data bandwidth corresponding to the third desktop update image is greater than the bandwidth threshold, the duration of the first period is increased.

10. An image processing device for a virtual desktop, characterized in that, The device includes: The sending module is used to send the updated desktop image of the virtual desktop to be updated in the first cycle to the terminal. The acquisition module is used to acquire a compensation image list, wherein the compensation image list includes the first image to be compensated corresponding to the virtual desktop in N second periods, and the N second periods are the first N consecutive periods of the first period, where N is an integer greater than 1. The removal module is used to remove regions that overlap with the damaged regions from each of the first images to be compensated in the compensation image chain if the desktop update image includes a damaged region, thereby obtaining a second image to be compensated for each second period; The encoding module is used to encode the second image to be compensated in the Nth second period of the compensation image chain to obtain a target image, and send the target image to the terminal, wherein the target image is used to perform image compensation on the second image to be compensated in the Nth second period of the virtual desktop.

11. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the image processing method for the virtual desktop as described in any one of claims 1 to 9.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by the processor, the image processing method of the virtual desktop as described in any one of claims 1 to 9 is implemented.

Images