An asynchronous rendering method and system based on dynamic region dependency

Abstract

The application relates to an asynchronous rendering method and system based on dynamic area dependence, and belongs to the technical field of image processing, which solves the problem of remote desktop image screen tearing and lagging caused by improper asynchronous processing timing or strict sequential processing mechanism. The method comprises the following steps: receiving screen update data packets, creating a rendering task, updating the rendering task after obtaining decoded image data by asynchronously decoding original image data which is not empty in the rendering task; establishing a dependence relationship between the new rendering task and the existing rendering task, updating the task state, and then putting the rendering task into the ready rendering queue if the rendering condition is met; taking out the rendering task from the ready rendering queue at a fixed time, obtaining effective tasks through target area coverage detection; merging and drawing the decoded image data of each effective task on an off-screen canvas, copying a frame of image on the screen canvas after generating the frame of image, and updating the task state of each effective task and the ready rendering queue. Smooth and stable remote desktop image rendering is realized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an asynchronous rendering method and system based on dynamic region dependence. Background Technology

[0002] Remote desktop technology allows users to display and control the desktop environment of a remote computer on their local device via a network connection. Its core functionality involves capturing, encoding, and transmitting the screen of the remote host, then decoding and rendering it on the client side, while simultaneously transmitting user input events back to the remote host. Among the many remote desktop protocols, SPICE (Simple Protocol for Independent Computing Environments) and VNC (Virtual Network Computing) are two representative open-source protocols. They each have their own focus in design philosophy and implementation methods, but also their own limitations.

[0003] The SPICE protocol employs a multi-channel architecture and asynchronous multiplexing mechanism. The server decomposes screen updates into multiple regions and sends them asynchronously, while the client is responsible for combining and rendering them. While this design prioritizes high performance, it shifts the burden of complex timing control to the client. If not handled properly, later-arriving regions may overlap with regions that have not yet been rendered, resulting in distorted screen content, i.e., a "screen tearing" phenomenon. Furthermore, it incurs significant CPU overhead when handling a large number of small-sized, high-frequency updates. The VNC protocol, based on frame buffer pixel transmission, uses a strict sequential processing model. The client updates the screen according to the order sent by the server, avoiding timing issues. However, network latency can cause subsequent updates to be blocked, resulting in a severe "stuttering" sensation, and the bandwidth utilization for transmitting raw pixel data is low.

[0004] Currently, most optimizations for remote desktop protocols focus on the server side, such as improving encoding algorithms, implementing smarter caching strategies, or optimizing data transmission protocols to reduce network bandwidth consumption and latency. However, these server-side optimizations cannot completely solve the rendering bottlenecks on the front-end (client). No matter how efficient the data sent by the server is, if there are performance issues in the client's processes of receiving, decoding, processing, and finally rendering this data, the user experience will still be severely affected. Especially in the web environment, the browser's JavaScript engine performance, rendering pipeline efficiency, and support for hardware acceleration all constitute potential bottlenecks for front-end rendering. Therefore, simply relying on server-side optimization cannot fundamentally solve the problems of screen tearing and stuttering caused by insufficient front-end processing capabilities or improper rendering timing. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide an asynchronous rendering method and system based on dynamic region dependency, in order to solve the problems of image distortion and screen stuttering caused by improper asynchronous processing timing or strict sequential processing mechanism when rendering web front-ends using existing remote desktop protocols.

[0006] On one hand, embodiments of the present invention provide an asynchronous rendering method based on dynamic region dependency, comprising the following steps:

[0007] Receive screen update data packets from the remote server, create rendering tasks, each rendering task includes: target area, task status and raw image data; asynchronously decode the non-empty raw image data to obtain decoded image data, and then update the task status of the rendering task.

[0008] Based on the target region of the new rendering task, establish its dependency relationship with the existing rendering tasks; identify the new rendering task according to its task status and dependency relationship, and if the preset rendering conditions are met, update its task status and put it into the ready rendering queue.

[0009] Periodically retrieve all rendering tasks from the ready rendering queue, and obtain valid tasks through target area coverage detection; merge the decoded image data of each valid task and draw it on the off-screen canvas, generate a frame image and copy it to the screen canvas, and then update the task status of each valid task and the ready rendering queue.

[0010] Based on the further improvement of the above method, the task state of the rendering task is a dual state describing its life cycle state, which includes: image processing state and region rendering state; the image processing state includes: processing and completed, and the region rendering state includes: processing, ready and rendered.

[0011] Based on further improvements to the above method, the dependencies of each rendering task include: a dependency set and a dependent set; based on the target region of a new rendering task, its dependencies with existing rendering tasks are established, including:

[0012] Among all existing rendering tasks, those whose target regions intersect spatially with the target regions of the new rendering task are identified as dependent tasks.

[0013] Add the task identifier of each task to be depended upon to the dependency set of the new rendering task, and add the task identifier of the new rendering task to the dependency set of each task to be depended upon.

[0014] Based on the further improvement of the above method, each rendering task also includes: a dependency counter; the value of the dependency counter is set to an initial value according to the size of the dependency set of the rendering task, and the initial value is decremented by 1 as the region rendering status of any rendering task in the dependency set is updated to "rendered".

[0015] Based on the further improvement of the above method, after obtaining the decoded image data, the task status of the rendering task is updated to "completed"; the preset rendering conditions are: the image processing status of the rendering task is "completed", and the value of the dependency counter of the rendering task is 0.

[0016] Based on the further improvement of the above method, if a new rendering task meets the preset rendering conditions, its region rendering status is updated to "ready" and then placed in the ready rendering queue.

[0017] Based on further improvements to the above method, effective tasks are obtained through target area coverage detection, including:

[0018] All rendering tasks to be tested taken from the ready rendering queue are compared pairwise. If the target area of ​​the first rendering task is completely contained by the target area of ​​the second rendering task, the first rendering task is determined to be an invalid task that is covered and is removed from the rendering tasks to be tested. Finally, the remaining rendering tasks to be tested are valid tasks.

[0019] Based on the further improvement of the above method, the decoded image data of each effective task are merged and drawn on an off-screen canvas to generate a single frame image, including:

[0020] Each valid task is sent to a background worker thread, and the minimum bounding box that surrounds all target areas is calculated based on the target area of ​​each valid task.

[0021] Create an off-screen canvas based on the size of the minimum bounding box;

[0022] The decoded image data of each effective task are sequentially drawn to the corresponding positions on the off-screen canvas according to their offset positions relative to the minimum bounding box, and then merged to generate a single frame image.

[0023] Based on further improvements to the above method, the task status and ready rendering queue of each valid task are updated, including:

[0024] Update the rendering status of each active task's region to "rendered";

[0025] Based on the dependencies of each valid task, the rendering tasks in the dependent set of each valid task are obtained in turn as tasks to be updated; after decrementing the dependency counter of each task to be updated, it is identified whether it meets the preset rendering conditions. If it does, it is added to the ready rendering queue.

[0026] On the other hand, embodiments of the present invention provide an asynchronous rendering system based on dynamic region dependency, comprising:

[0027] The data receiving module is used to receive screen update data packets from the remote server and create rendering tasks. Each rendering task includes: target area, task status and raw image data.

[0028] The asynchronous processing module is used to asynchronously decode the non-empty raw image data to obtain decoded image data and then update the task status of the rendering task.

[0029] The scheduling rendering module is used to establish the dependency relationship between the target area of ​​a new rendering task and existing rendering tasks; it identifies the new rendering task based on its task status and dependency relationship, and if the preset rendering conditions are met, it updates the task status and puts it into the ready rendering queue; it periodically retrieves all rendering tasks from the ready rendering queue, and obtains valid tasks through target area coverage detection; it merges the decoded image data of each valid task and draws it on the off-screen canvas, generates a frame image and copies it to the screen canvas, and then updates the task status of each valid task and the ready rendering queue.

[0030] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0031] 1. By separating computationally intensive image decoding and processing tasks from the main thread, UI blocking is avoided; by establishing dependencies between rendering tasks and updating task states, the correct rendering order is automatically derived from the spatial relationships between regions, eliminating screen glitches caused by timing errors from the root; at the same time, since there is no need to wait for all updates sequentially like in traditional solutions, multiple independent regions can enter the subsequent processing flow in parallel, significantly alleviating screen stuttering caused by network fluctuations or intensive updates.

[0032] 2. Coverage detection is performed on all tasks in the ready queue to remove invalid areas completely covered by other tasks, ensuring that only the ultimately visible screen content is included in the current processing scope, thus avoiding invalid pixel filling operations; by merging the image data of multiple valid tasks and drawing them onto the off-screen canvas, a complete image frame is generated and rendered to the screen canvas at once, significantly reducing the number of drawing calls and context switching overhead of the main thread; through a chain reaction mechanism of "rendering complete → notifying dependents → dependents status update → those meeting the conditions enter the ready rendering queue", the entire rendering pipeline is continuously and orderly driven to operate automatically, achieving a balance between high throughput and low latency while ensuring image accuracy, providing users with a smooth and stable remote desktop access experience.

[0033] 3. By combining the asynchronous computing capabilities of worker threads, the multi-threading and hardware acceleration capabilities provided by browsers, and the dual state management mechanism, a highly efficient and robust rendering method is constructed. This method optimizes the entire process of receiving, decoding, processing, and finally rendering remote desktop image data, thereby significantly improving the visual experience and smoothness of users accessing remote desktops in a web environment. It is not only suitable for traditional remote office and cloud desktop scenarios, but also addresses the growing demand for browser-based remote applications, promoting the further development and application of remote desktop technology in the web field.

[0034] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0035] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0036] Figure 1 This is a flowchart of an asynchronous rendering method based on dynamic region dependency in Embodiment 1 of the present invention;

[0037] Figure 2 This is a schematic diagram of an asynchronous rendering system based on dynamic region dependency in Embodiment 2 of the present invention. Detailed Implementation

[0038] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0039] Example 1

[0040] A specific embodiment of the present invention discloses an asynchronous rendering method based on dynamic region dependency, such as... Figure 1 As shown, it includes steps S1-S3.

[0041] S1. Receive screen update data packets from the remote server, create rendering tasks, each rendering task includes: target area, task status and original image data; asynchronously decode the non-empty original image data to obtain decoded image data, and then update the task status of the rendering task.

[0042] It should be noted that in typical web applications, remote desktop data is pushed from the server to the client via a WebSocket connection, and the client obtains screen update data packets by listening to the WebSocket's onmessage event.

[0043] Whenever a screen update data packet is received, the message type of the data packet is first parsed according to a predefined protocol format. These include: session initialization message, display channel message, input channel message, and cursor channel message. Among them, the session initialization message is the first batch of messages exchanged after the client and the remote server establish a connection. It is used to establish and control the main session and is not used for rendering. The input channel message is mainly sent from the client's user interaction events to the server. The cursor channel message is used to process the mouse cursor, such as updating the appearance of the cursor on the server or client. The display channel message is the most important channel in the remote desktop protocol. It is responsible for transmitting the image update data of the remote desktop. This embodiment only processes the data packets of the display channel message.

[0044] Specifically, the data packets of the display channel message are further parsed to obtain: original region data, and original image data or reused image identifiers. The original region data includes the absolute coordinates of the left, top, right, and bottom boundaries of the rectangular region; the original image data is either compressed image data or raw pixel data, where compressed image data includes, but is not limited to, JPEG-encoded image data and LZ_RGB compressed image data; the reused image identifier indicates that the current update needs to reuse some previous image data, and no new image data needs to be carried, i.e., the original image data is empty.

[0045] In this embodiment, a corresponding rendering task is created for each data packet of the display channel message. Each rendering task includes: task identifier, target area, task status, original image data, and reused image identifier.

[0046] Specifically, the task identifier is a globally unique identifier; the target region is the coordinates of the top-left corner and its dimensions of a rectangular region obtained by transforming the original region data, including: the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the width w of the rectangular region, and the height h of the rectangular region; the original image data is the unprocessed compressed image data or raw pixel data in the data packet, which may be empty; the reused image identifier is the image identifier reused in the image data reuse scenario; the task state is a dual state describing its lifecycle state, which includes: image processing state and region rendering state; among which, the image processing state includes: processing and completed, and the region rendering state includes: processing, ready, and rendered.

[0047] When creating a rendering task, the image processing status is initially set to "Processing" and the region rendering status is initially set to "Processing".

[0048] Unlike many traditional remote desktop protocols or rendering engines that use fixed-size tiles, in this embodiment, the target area for each rendering task is dynamically generated based on the data packets sent from the server. This means that a rendering task can correspond to a very small area (e.g., displaying a blinking cursor) or a large area covering the entire screen (e.g., switching desktop backgrounds). By using dynamic rectangular areas, the system responds to screen changes with the smallest processing unit, thereby improving overall rendering efficiency.

[0049] Furthermore, after the rendering task is created, considering that image data decoding is one of the most computationally intensive tasks in remote desktop applications, if the original image data in the rendering task is empty, step S2 is executed directly; if the rendering task contains non-empty original image data, in order to avoid blocking the main thread, the rendering task is sent to a background worker thread to asynchronously decode the original image data, thereby ensuring the smoothness and fast response of the user interface.

[0050] Specifically, the main thread sends the task identifier, target region, and raw image data of the rendering task to be decoded to the Web Worker thread via the `postMessage` method. The Worker thread runs independently in the background, working in parallel with the main thread. It employs different processing methods depending on the type of raw image data, but the final decoded image data is an `ImageBitmap` object that can be directly used for drawing, including:

[0051] For compressed image data, the Worker first converts it into a Blob object, and then calls the createImageBitmap(blob) method to perform asynchronous decoding and generate an ImageBitmap object;

[0052] For raw pixel data sent directly by the server, the Worker encapsulates it into an ImageData object, and then calls the createImageBitmap(imageData) method to convert it into an ImageBitmap object.

[0053] After decoding is complete, the Worker thread stores the generated ImageBitmap object in a local image storage collection (e.g., using a Map format) indexed by the task identifier for easy retrieval. Subsequently, the Worker sends a decoding completion notification to the main thread via postMessage, which includes at least the task identifier.

[0054] After receiving the notification, the main thread finds the corresponding rendering task and updates its image processing status to "completed", indicating that the image data for the rendering task is ready.

[0055] It should be noted that when the reused image identifier in the rendering task is not empty, the corresponding source rendering task is found based on the reused image identifier. When the image processing status of the source rendering task is updated, the image processing status of the reused rendering task is updated synchronously to ensure that the image processing status of the two is consistent.

[0056] This step removes computationally intensive image decoding and processing tasks from the main thread, avoiding UI blocking. When called in a browser that supports hardware acceleration, the decoding operation directly calls the underlying operating system's image decoding library and utilizes the GPU for acceleration, significantly reducing CPU load.

[0057] S2. Based on the target area of ​​the new rendering task, establish its dependency relationship with the existing rendering tasks; identify the new rendering task according to its task status and dependency relationship, and if the preset rendering conditions are met, update its task status and put it into the ready rendering queue.

[0058] It should be noted that updates to multiple areas on the screen may occur simultaneously or nearly simultaneously, and there may be spatial overlap between them. To ensure the correctness of the final rendered image and prevent later-drawn layers from being incorrectly covered by earlier-drawn layers, i.e., to solve the "screen tearing" problem, this embodiment manages the rendering order of these target areas by dynamically and correctly establishing dependencies between rendering tasks.

[0059] The dependencies of each rendering task include: a dependency set and a dependent set; the dependency set is used to track the rendering tasks that the current rendering task must render before rendering; the dependent set is used to notify subsequent rendering tasks that the current rendering task needs to render after rendering.

[0060] Specifically, existing rendering tasks are all tasks that have not yet been completed, including those with a region rendering status of "processing" or "ready". Based on the target region of the new rendering task, a bidirectional dependency relationship is established between it and existing rendering tasks, including:

[0061] Among all existing rendering tasks, those whose target regions intersect spatially with the target regions of the new rendering task are identified as dependent tasks.

[0062] Add the task identifier of each dependent task to the dependency set of the new rendering task, indicating that the rendering of the new rendering task depends on the completion of these dependent tasks; and add the task identifier of the new rendering task to the dependency set of each dependent task, indicating that when the dependent tasks complete their rendering, the new rendering task needs to be notified.

[0063] It should be noted that when a new rendering task does not intersect with the target region of an existing rendering task in any spatial manner, its dependency set is empty.

[0064] This embodiment maintains the parent-child relationship of target regions between rendering tasks by establishing bidirectional link dependencies, greatly optimizing the efficiency of dependency updates and notifications. When a parent region completes rendering, it does not need to traverse all rendering tasks; it only needs to directly traverse its dependent set to quickly notify all related child regions, thereby triggering subsequent rendering checks. This data structure is key to achieving efficient and intelligent scheduling.

[0065] Preferably, in order to achieve efficient dependency establishment, this embodiment introduces a spatial indexing mechanism. In the front-end JavaScript environment, an R-tree is constructed using the RBush library as the core spatial indexing data structure to store all target regions that are incomplete rendering tasks (i.e., the region rendering status is "processing" or "ready").

[0066] The `search()` method of the `RBush` instance is called, passing in the bounding box ({minX, minY, maxX, maxY}) corresponding to the target region of the new rendering task. This quickly queries all existing regions that intersect with the bounding box, obtaining the corresponding rendering tasks as dependencies. The hierarchical index structure of the R-tree reduces the query complexity from O(n) of linear traversal to O(logn), significantly improving system performance when processing large numbers of regions. After establishing dependencies, the new rendering task abstracts its target region as a JavaScript object and inserts it into the R-tree using the `insert()` method for subsequent querying and management.

[0067] It should be noted that the JavaScript object contains: information related to the rendering task, as well as four numerical fields: left boundary minX, top boundary minY, right boundary maxX, and bottom boundary maxY, which are used to define the bounding box of the object.

[0068] To efficiently track whether all dependencies are satisfied, each rendering task also includes a dependency counter. The initial value of the dependency counter is set based on the size of its dependency set, and it is decremented by 1 as the rendering status of any region of the rendering task in the dependency set is updated to "rendered". That is, when a parent rendering task (i.e., a rendering task in the dependency set) completes rendering, it iterates through its dependency set, decrementing the dependency counter of each child rendering task in the dependency set by 1. When the dependency counter value of a child rendering task becomes 0, it means that all its preceding dependent tasks have been rendered.

[0069] The ready-to-render queue is a priority queue that can be sorted according to the order in which rendering tasks are received or other strategies.

[0070] Once a new rendering task has established a bidirectional dependency, it is identified based on the task status and dependencies of the new rendering task according to preset rendering conditions to determine whether to add the new rendering task to the ready rendering queue. The preset rendering conditions are: the image processing status of the rendering task is "completed," and the dependency counter value of the rendering task is 0.

[0071] It should be noted that if a new rendering task meets the preset rendering conditions, its region rendering status will be updated to "ready" and then placed in the ready rendering queue to wait for the final rendering execution; otherwise, the region rendering status of the new rendering task will not be updated (it will still be "processing").

[0072] When a new rendering task's dependency set is empty, its region rendering state is updated to "ready" and placed in the ready rendering queue once its raw image data has been asynchronously decoded, awaiting rendering to the screen. This mechanism allows for the parallel processing of multiple independent screen updates. For example, if a user opens a menu in the upper left corner of the screen while simultaneously scrolling through a document in the lower right corner, and if the regions corresponding to these two operations do not overlap, they can be decoded and rendered simultaneously and independently, greatly improving the user experience and avoiding the situation in traditional serial rendering schemes where a region update blocks all other updates.

[0073] Compared to existing technologies, the image processing state in this embodiment ensures that the rendering thread never attempts to draw an image that is not yet ready, thus avoiding display errors caused by incomplete data. The dependency counter based on region rendering state and dependencies ensures the sequential nature of screen updates, guaranteeing that later-drawn layers do not incorrectly overwrite earlier-drawn layers. This double-check mechanism guarantees the correctness and efficiency of rendering from both data preparation and spatial temporal dimensions, which is key to simultaneously solving screen tearing and stuttering issues.

[0074] S3. Periodically retrieve all rendering tasks from the ready rendering queue, obtain valid tasks through target area coverage detection, merge the decoded image data of each valid task and draw it on the off-screen canvas, generate a frame image and copy it to the screen canvas, and then update the task status of each valid task and the ready rendering queue.

[0075] It should be noted that, in order to synchronize with the browser's rendering cycle, this embodiment binds timed rendering to the requestAnimationFrame mechanism, fundamentally solving the screen tearing problem. When the requestAnimationFrame callback is triggered, all rendering tasks for the current cycle are first retrieved from the ready rendering queue.

[0076] However, directly rendering all ready-to-use rendering tasks can lead to unnecessary performance overhead. For example, if a larger area completely covers a smaller area, rendering the covered smaller area is meaningless because it will eventually be covered by the larger area. To address this issue, this embodiment determines the effective tasks through target area coverage detection before performing rendering.

[0077] Specifically, all rendering tasks to be tested taken from the ready rendering queue are compared pairwise. If the target area of ​​the first rendering task is completely contained by the target area of ​​the second rendering task, the first rendering task is determined to be an invalid task that is covered and is removed from the rendering tasks to be tested. Finally, the remaining rendering tasks to be tested are valid tasks.

[0078] The condition for determining whether the target region A of the first rendering task is completely contained within the target region B of the second rendering task is: the left boundary of target region B is less than or equal to the left boundary of target region A, the upper boundary of target region B is less than or equal to the upper boundary of target region A, the right boundary of target region B is greater than or equal to the right boundary of target region A, and the lower boundary of target region B is greater than or equal to the lower boundary of target region A; that is: the left boundary of target region B is to the left of (or coincides with) the left boundary of target region A, the upper boundary of target region B is above (or coincides with) the upper boundary of target region A, the right boundary of target region B is to the right of (or coincides with) the right boundary of target region A, and the lower boundary of target region B is below (or coincides with) the lower boundary of target region A.

[0079] This pre-rendering overlay and cleanup strategy ensures that only the ultimately visible content is rendered per frame, avoiding invalid rendering of intermediate states and significantly improving rendering efficiency. It is particularly suitable for high-frequency update scenarios such as rapid window switching and video playback. For example, when a user quickly switches windows, multiple overlapping update areas may occur consecutively. This mechanism ensures that only the ultimately visible area is rendered, avoiding invalid rendering of intermediate states. In video playback scenarios, overlapping areas caused by consecutive frame updates can also be efficiently handled through this mechanism, maintaining the rational utilization of system resources.

[0080] The final rendering execution is responsible for presenting the processed and satisfied decoded image data to the user's screen. This step achieves an efficient, smooth, and tear-free asynchronous rendering process through a collaboration mechanism between the main thread and Workers, combined with an off-screen canvas merging strategy.

[0081] This embodiment merges the decoded image data of each effective task and draws it on an off-screen canvas. When generating an image frame, it supports two off-screen merging strategies: minimum bounding box off-screen canvas merging strategy and full-size off-screen canvas merging strategy. The selected strategy is pre-configured according to the actual scenario and performance requirements.

[0082] It should be noted that the minimum bounding box off-screen canvas merging strategy is suitable for scenarios where the update area is relatively concentrated and the overall range is significantly smaller than the full screen. This makes it easier to significantly reduce the size of the off-screen canvas, thereby reducing GPU memory usage and transmission bandwidth.

[0083] Specifically, it includes the following steps:

[0084] Send each valid task to the background worker thread; calculate the minimum bounding box that surrounds all target areas based on the target areas of each valid task.

[0085] An offscreen canvas is created based on the size of the minimum bounding box as a merge buffer; the background of the created offscreen canvas is transparent by default.

[0086] The decoded image data of each effective task are sequentially drawn to the corresponding positions on the off-screen canvas according to their offset positions relative to the minimum bounding box, and then merged to generate a frame image. This frame image is a local image relative to the entire screen canvas, in which the undrawn areas remain transparent.

[0087] It should be noted that the full-size off-screen canvas merging strategy is suitable for scenarios where the update area is relatively dispersed and almost covers the entire screen, or for simplifying the implementation logic.

[0088] Specifically, it includes the following steps:

[0089] Each valid task is sent to a background worker thread; the background worker thread maintains an offscreen canvas of the same size as the main canvas as a merging buffer;

[0090] Clear the off-screen canvas of the background worker thread and remove the drawing content of the previous frame (i.e., reset the off-screen canvas to transparent).

[0091] According to the task identifier of each valid task, the corresponding decoded image data is obtained from the local image storage set, and drawn from the target area of ​​the valid task to the corresponding position on the off-screen canvas.

[0092] After all valid tasks are drawn, they are merged to generate a single frame image. This frame image is a complete image relative to the entire screen canvas, in which only the target area corresponding to the valid task in this round contains actual image pixels, while other areas off-screen canvas have been cleared to transparent.

[0093] In this embodiment, both off-screen merging strategies perform their drawing operations on the off-screen canvas of the Worker thread, without blocking the main thread, ensuring consistently smooth user interface responsiveness. In browsers supporting hardware acceleration, the merging drawing operation is completed directly on the GPU, no longer limited by CPU performance. The GPU's parallel architecture can process the compositing of hundreds of regions simultaneously, completing the merging within milliseconds even if the canvas contains a large number of updating regions.

[0094] After obtaining a frame image according to any of the above offscreen merging strategies, the background worker thread calls the `transferToImageBitmap()` method of `OffscreenCanvas` to extract the merged frame image into an `ImageBitmap` object format, and then transmits the frame image back to the main thread with zero copy through the transferable object mechanism of `postMessage`. For the minimum bounding box offscreen canvas merging strategy, the position of the frame image (i.e., the coordinates of the top left corner of the minimum bounding box) also needs to be transmitted back to the main thread.

[0095] After receiving a frame of image transmitted from the background worker thread, the main thread immediately calls the drawImage method of the main screen canvas context to copy this frame of image onto the screen canvas at once. That is, only the area with image content is updated, keeping the screen unchanged, so as to achieve the most efficient refresh.

[0096] Specifically, for the minimum bounding box off-screen canvas merging strategy, the received data is a partial image frame and its position information. Based on the position information, the partial image frame is copied to the corresponding position on the screen canvas. Since the unupdated areas in the image frame are transparent, after copying, only the areas with actual image content will be refreshed, while the parts of the screen canvas outside the original area, as well as the originally transparent areas within the original area, will remain unchanged.

[0097] The full-size off-screen canvas merging strategy receives a complete image frame. When copied to the screen canvas, only the non-transparent areas of the image are updated to the corresponding positions on the screen canvas, while the content on the screen canvas that was originally behind the transparent areas remains unchanged because it is not covered.

[0098] Since the received image is a pre-merged frame, the main thread only needs to execute a drawImage call once to complete the rendering of the entire frame, resulting in minimal drawing overhead.

[0099] After the image is drawn onto the screen, the final stages of this rendering cycle begin, which is also the key step driving the continuous operation of the entire asynchronous pipeline:

[0100] First, update the region rendering status of all valid tasks in this round to "rendered".

[0101] Then, based on the dependencies of each valid task, the rendering tasks (i.e., the sub-rendering tasks that depend on the current valid task) in the dependent set of each valid task are obtained in turn as tasks to be updated, and the following operations are performed:

[0102] Decrement the dependency counter of the task to be updated by 1;

[0103] The system identifies whether the task to be updated meets the preset rendering conditions (i.e., the image processing status is "completed" and the dependency counter value is 0). If it does, the system updates the region rendering status of the task to be updated to "ready" and adds it to the ready rendering queue, waiting for the next browser refresh cycle to render.

[0104] It should be noted that if the target region of a rendering task is stored in an R tree, the target region corresponding to each valid task is removed from the R tree spatial index structure to ensure that only incomplete rendering tasks are maintained in the R tree, thus maintaining the efficiency of subsequent queries.

[0105] Through this chain reaction mechanism of "rendering complete → notifying dependents → dependents update their status → those that meet the conditions join the ready rendering team", the entire rendering pipeline can continuously and orderly move forward automatically and automatically unlock subsequent rendering tasks.

[0106] Compared with existing technologies, this embodiment provides an asynchronous rendering method based on dynamic region dependency. By decoupling computationally intensive image decoding and processing tasks from the main thread, it avoids UI blocking. Furthermore, by establishing dependencies between rendering tasks and updating task states, it ensures that the correct rendering order is automatically derived from the spatial relationships between regions, thus eliminating screen flickering caused by timing errors at the source. At the same time, since it does not need to wait for all updates sequentially like in traditional solutions, multiple independent regions can enter the subsequent processing flow in parallel, significantly alleviating screen stuttering caused by network fluctuations or intensive updates. Coverage detection is performed on all tasks in the ready queue to remove invalid areas completely covered by other tasks, ensuring that only the ultimately visible image content is included in the current processing scope, thus avoiding invalid pixel filling operations. By merging the image data of multiple valid tasks and drawing them onto the off-screen canvas, a complete image frame is generated and rendered to the screen canvas at once, significantly reducing the number of drawing calls and context switching overhead on the main thread. Through a chain reaction mechanism of "rendering complete → notifying dependents → dependents update their status → those meeting the conditions enter the ready rendering queue", the entire rendering pipeline is continuously and orderly driven to operate automatically. While ensuring image accuracy, it achieves a balance between high throughput and low latency, providing users with a smooth and stable remote desktop access experience. By combining the asynchronous computing capabilities of worker threads, the multi-threading and hardware acceleration capabilities provided by browsers, and a dual state management mechanism, a highly efficient and robust rendering method is constructed. This method optimizes the entire process of receiving, decoding, processing, and finally rendering remote desktop image data, thereby significantly improving the visual experience and smoothness of users accessing remote desktops in a web environment. It is not only suitable for traditional remote office and cloud desktop scenarios, but also addresses the growing demand for browser-based remote applications, promoting the further development and application of remote desktop technology in the web field.

[0107] Example 2

[0108] Another embodiment of the present invention discloses an asynchronous rendering system based on dynamic region dependency, thereby implementing the asynchronous rendering method based on dynamic region dependency in Embodiment 1. The specific implementation of each module is described in the corresponding description in Embodiment 1. Figure 2 As shown, the system includes:

[0109] The data receiving module is used to receive screen update data packets from the remote server and create rendering tasks. Each rendering task includes: target area, task status and raw image data.

[0110] The asynchronous processing module is used to asynchronously decode the non-empty raw image data to obtain decoded image data and then update the task status of the rendering task.

[0111] The scheduling rendering module is used to establish the dependency relationship between the target area of ​​a new rendering task and existing rendering tasks; it identifies the new rendering task based on its task status and dependency relationship, and if the preset rendering conditions are met, it updates the task status and puts it into the ready rendering queue; it periodically retrieves all rendering tasks from the ready rendering queue, and obtains valid tasks through target area coverage detection; it merges the decoded image data of each valid task and draws it on the off-screen canvas, generates a frame image and copies it to the screen canvas, and then updates the task status of each valid task and the ready rendering queue.

[0112] It's important to note that the data receiving module runs on the browser's main thread, parsing screen update data packets and constructing them into rendering tasks before sending them to the asynchronous processing module. The asynchronous processing module runs in a background Web Worker. It receives tasks from the main thread and uses technologies such as OffscreenCanvas, WebGL / WebGPU to decode and process the raw data. This processing is completely asynchronous and does not block the main thread. The scheduling and rendering module runs on the browser's main thread. It coordinates all rendering tasks by building a scheduler. For a new rendering task, it establishes the spatial dependency between the target region of that task and the target regions of all other tasks to be rendered. When the image processing state of a rendering task or the region rendering state of its dependent tasks changes, the scheduler updates the region rendering state of that task to "ready." Through iterative rendering operations, it continuously extracts from the ready rendering queue, merges them to generate a single frame, and copies it to the screen all at once.

[0113] Since the asynchronous rendering system based on dynamic region dependency in this embodiment and the aforementioned asynchronous rendering method based on dynamic region dependency can be mutually referenced, and this is a repetition, they will not be repeated here. Because this system embodiment shares the same principle as the aforementioned method embodiment, it also possesses the corresponding technical effects of the aforementioned method embodiment.

[0114] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0115] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An asynchronous rendering method based on dynamic region dependency, characterized in that, Includes the following steps: Receive screen update data packets from a remote server, create rendering tasks, each rendering task including: target area, task status and original image data; asynchronously decode the non-empty original image data to obtain decoded image data, and then update the task status of the rendering task. Based on the target region of the new rendering task, establish its dependency relationship with the existing rendering tasks; identify the new rendering task according to its task status and dependency relationship, and if the preset rendering conditions are met, update its task status and put it into the ready rendering queue. Periodically retrieve all rendering tasks from the ready rendering queue, and obtain valid tasks through target area coverage detection; merge the decoded image data of each valid task and draw it on the off-screen canvas, generate a frame image and copy it to the screen canvas, and then update the task status of each valid task and the ready rendering queue.

2. The asynchronous rendering method based on dynamic region dependency according to claim 1, characterized in that, The task state of the rendering task is a dual state describing its lifecycle state, which includes: image processing state and region rendering state; the image processing state includes: processing and completed, and the region rendering state includes: processing, ready and rendered.

3. The asynchronous rendering method based on dynamic region dependency according to claim 2, characterized in that, The dependencies of each rendering task include: a dependency set and a dependent set; establishing the dependencies between the target region of the new rendering task and existing rendering tasks includes: Among all existing rendering tasks, those whose target regions spatially intersect with the target regions of the new rendering task are identified as the tasks to be depended upon. Add the task identifier of each task to be depended upon to the dependency set of the new rendering task, and add the task identifier of the new rendering task to the dependency set of each task to be depended upon.

4. The asynchronous rendering method based on dynamic region dependency according to claim 3, characterized in that, Each rendering task also includes a dependency counter; the value of the dependency counter is set to an initial value based on the size of the dependency set of the rendering task, and the initial value is decremented by 1 as the region rendering status of any rendering task in the dependency set is updated to "rendered".

5. The asynchronous rendering method based on dynamic region dependency according to claim 4, characterized in that, The step of updating the rendering task status after obtaining the decoded image data is to update the image processing status to "completed"; the preset rendering conditions are: the image processing status of the rendering task is "completed", and the value of the dependency counter of the rendering task is 0.

6. The asynchronous rendering method based on dynamic region dependency according to claim 2 or 5, characterized in that, If a new rendering task meets the preset rendering conditions, its region rendering status is updated to "ready" and then placed in the ready rendering queue.

7. The asynchronous rendering method based on dynamic region dependency according to claim 2, characterized in that, The effective task obtained through target area coverage detection includes: All rendering tasks to be tested taken from the ready rendering queue are compared pairwise. If the target area of ​​the first rendering task is completely contained by the target area of ​​the second rendering task, the first rendering task is determined to be an invalid task that is covered and is removed from the rendering tasks to be tested. Finally, the remaining rendering tasks to be tested are valid tasks.

8. The asynchronous rendering method based on dynamic region dependency according to claim 2, characterized in that, The step of merging the decoded image data from each effective task and drawing it on an off-screen canvas to generate a single image frame includes: Each valid task is sent to a background worker thread, and the minimum bounding box that surrounds all target areas is calculated based on the target area of ​​each valid task. Create an off-screen canvas based on the dimensions of the minimum bounding box; The decoded image data of each effective task are sequentially drawn to the corresponding positions on the off-screen canvas according to their offset positions relative to the minimum bounding box, and then merged to generate a single frame image.

9. The asynchronous rendering method based on dynamic region dependency according to claim 4 or 5, characterized in that, The subsequent updating of the task status and ready rendering queue for each valid task includes: Update the rendering status of each active task's region to "rendered"; Based on the dependencies of each valid task, the rendering tasks in the dependent set of each valid task are obtained in turn as tasks to be updated; after decrementing the dependency counter of each task to be updated, it is identified whether it meets the preset rendering conditions. If it does, it is added to the ready rendering queue.

10. An asynchronous rendering system based on dynamic region dependency, characterized in that, include: The data receiving module is used to receive screen update data packets from a remote server and create rendering tasks. Each rendering task includes: target area, task status, and original image data. The asynchronous processing module is used to asynchronously decode the non-empty raw image data to obtain decoded image data and then update the task status of the rendering task. The scheduling rendering module is used to establish the dependency relationship between the target area of ​​a new rendering task and existing rendering tasks; it identifies the new rendering task based on its task status and dependency relationship, and if the preset rendering conditions are met, it updates the task status and puts it into the ready rendering queue; it periodically retrieves all rendering tasks from the ready rendering queue, and obtains valid tasks through target area coverage detection; it merges the decoded image data of each valid task and draws it on the off-screen canvas, generates a frame image and copies it to the screen canvas, and then updates the task status of each valid task and the ready rendering queue.

Images