Real-time rendering-oriented edge task partitioning and dynamic offloading method and system

Abstract

This invention discloses a method and system for edge-to-edge collaborative task partitioning and dynamic offloading for real-time rendering. The invention divides each frame of the rendered image into two mutually exclusive pixel blocks at the pixel block level. One block is rendered locally by the user terminal, and the other is rendered by an edge server. The edge server sends the rendered pixel block back to the user terminal, which then stitches the server-rendered pixel block with the locally rendered pixel block to form a complete frame. During the rendering process, the system periodically acquires runtime information such as terminal-side rendering time, edge-side rendering time, network bandwidth, and round-trip latency, and dynamically adjusts the rendering workload of the terminal and edge server based on the collected historical performance data. This invention achieves adaptive distribution of the rendering load between the edge and terminal sides and effectively mitigates the impact of network fluctuations on real-time rendering performance.

Description

Technical Field

[0001] This invention relates to the field of computer graphics rendering and edge computing technology, specifically to a method and system for edge-to-edge collaborative task partitioning and dynamic unloading for real-time rendering. Background Technology

[0002] With the development of computer graphics and hardware acceleration technologies, high-quality rendering has gradually expanded from the traditional offline rendering field to real-time rendering scenarios. By physically modeling the propagation path of light in a 3D scene, complex optical effects such as lighting, shadows, reflection, and refraction can be realistically simulated, becoming an important technological direction for improving the realism of graphics. In recent years, with the emergence of high-performance GPUs and dedicated rendering acceleration units, real-time rendering technology has received widespread attention in fields such as games, virtual reality, augmented reality, and 3D visualization.

[0003] In existing technologies, to meet the high computational resource demands of high-quality real-time rendering, two main technical solutions are typically employed: one relies on deploying high-performance graphics hardware locally on the user terminal, with the local GPU independently completing all rendering computations; the other offloads the entire rendering task to a remote cloud server or centralized rendering cluster, where the server GPU completes the rendering and then transmits the results to the user terminal for display via the network. Furthermore, with the development of edge computing technology, some research and engineering practices have begun to explore deploying GPU devices on edge servers closer to the user to provide rendering capabilities, thereby reducing network latency introduced by cloud-based rendering.

[0004] While existing local rendering, cloud rendering, and edge collaborative rendering solutions can meet application requirements in certain scenarios, they still have the following shortcomings in real-time rendering tasks for interactive applications: Relying entirely on user terminals for local rendering places high demands on terminal hardware performance. Typically, user terminals need to be equipped with or upgraded with high-end GPUs to maintain an acceptable real-time frame rate, resulting in high system deployment costs and limited applicability, which is not conducive to promotion on ordinary terminals or mobile devices.

[0005] While centralized rendering solutions based on cloud servers offer strong computing power, cloud servers are typically deployed in data centers far from users, requiring rendering results to be transmitted over long distances. This makes cloud rendering susceptible to network bandwidth fluctuations and round-trip latency, making it difficult to meet the low-latency requirements of interactive applications. This is especially true for real-time rendering tasks in interactive applications, where network latency has a significant impact on rendering results, failing to effectively support the need for low-latency real-time rendering.

[0006] In technologies combining edge computing and real-time rendering, many existing solutions place rendering tasks on edge servers, with the terminal only responsible for display. This approach ignores the terminal's computing power, leading to a waste of terminal computing resources. Some solutions attempt to distribute rendering tasks between the terminal and the server to fully utilize computing resources and improve system rendering performance. However, existing solutions typically require prior knowledge of the terminal and server's computing capabilities or offline performance modeling, relying on pre-defined task partitioning strategies or configuration based on static performance parameters. When network conditions, computing load, or hardware performance in the operating environment change, fixed or static task partitioning strategies are difficult to adjust in a timely manner, easily resulting in idle computing resources on one side while the other becomes a bottleneck, thus reducing overall rendering efficiency. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for edge-to-edge collaborative task partitioning and dynamic offloading for real-time rendering. It uses edge servers equipped with high-performance GPUs to accelerate users' real-time rendering tasks, enabling user terminals and edge servers to work together and optimizing rendering load distribution by dynamically adjusting the offloading ratio of tasks.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A method for edge-to-edge collaborative task partitioning and dynamic unloading for real-time rendering includes the following steps: By dividing each frame of the image to be rendered into pixel blocks, each frame of the image is split into two mutually exclusive pixel blocks. Two mutually exclusive pixel blocks are rendered by the user terminal and the edge server respectively, and the running status information of the rendering on the user terminal side and the rendering on the edge server side is periodically obtained during the rendering process. The edge server sends the rendered pixel blocks back to the user terminal; The user terminal stitches together the pixel blocks rendered by the edge server with the pixel blocks rendered locally on the user terminal to form a complete frame image. The rendering workload of user terminals and edge servers is dynamically adjusted based on the collected historical operational status information.

[0009] This invention aims to minimize the rendering time of a single frame. It abstracts the real-time rendering task of each frame into a collaborative processing problem consisting of computational latency and network transmission latency between the edge server and the user terminal. The rendering task of each image is split into two parts, handled by the user terminal and the edge server respectively. The edge server transmits the rendering result back to the user terminal via the network, and the user terminal stitches the two parts together to display the complete image.

[0010] According to the above technical solution, the image to be rendered is randomly divided into two mutually exclusive, non-empty rectangular pixel blocks along the image width direction. Each pixel block is identified by a task structure, which stores the global coordinates of the top-left pixel of the pixel block, as well as the width and height of the pixel block. Specifically: for a width of... Height is The image to be rendered is divided along its width into pixel blocks rendered by the user terminal and pixel blocks rendered by the edge server. One pixel block is rendered locally on the user's device, and the other... The pixel blocks are rendered on the edge server.

[0011] This invention effectively solves the problem that a single computing node cannot simultaneously meet the requirements of high computational complexity and real-time performance by dividing a frame of rendered image into two mutually exclusive pixel blocks and executing the rendering task in parallel between the user terminal and the edge server. Through edge-device collaborative rendering, the computing power of the edge server can be fully utilized, reducing the user terminal's dependence on high-performance graphics hardware, thereby improving the system's applicability and deployment flexibility.

[0012] According to the above technical solution, when the user terminal renders a pixel block, it sets an asynchronous reading offset parameter k and notifies the edge server of this parameter; the user terminal sends a rendering command to the edge server. After completing the rendering of the local pixel block, the user terminal initiates an asynchronous video memory read operation to asynchronously copy the pixel block result of the current rendering cycle from the GPU video memory to the host memory; at the same time, it extracts the local rendering pixel block result that is k frames backward relative to the current rendering cycle.

[0013] According to the above technical solution, after the edge server renders the pixel block, it asynchronously copies the rendering result of the pixel block of the current frame from the GPU memory to the host memory; at the same time, it reads the pixel block result of the kth frame before the offset k and performs real-time compression encoding, and sends the compressed pixel block rendering result to the user terminal.

[0014] According to the above technical solution, after the edge server renders the pixel block, it asynchronously copies the rendering result of the pixel block of the current frame from the GPU memory to the host memory; at the same time, it reads the pixel block result of the kth frame before the offset k and performs real-time compression encoding, and sends the compressed pixel block rendering result to the user terminal.

[0015] According to the above technical solution, the user terminal receives pixel block data from the edge server and decodes the received data in real time; based on the global coordinates of each pixel block, the decoded edge server rendered pixel block is spliced ​​with the local rendered pixel block to synthesize a complete frame image and display it.

[0016] This invention introduces asynchronous data reading and a pixel block stitching mechanism based on historical frames. While ensuring the correctness of the rendering results, it effectively masks the data copy latency from video memory to host memory, further reducing end-to-end rendering latency and improving the continuity and smoothness of real-time rendering. At the same time, it uses video compression technology to significantly reduce network data communication overhead and comprehensively considers the impact of network bandwidth and round-trip latency on the transmission process. By incorporating computation latency and transmission latency into the optimization objective, the overall rendering completion time of the system is optimized, and the stability of the rendering process is improved.

[0017] According to the above technical solution, after the interval T between the last pixel block division and the last division, the rendering task is re-divided using the pixel block division algorithm. If the rendering task division method changes, the rendering task load of the user terminal and the edge server is dynamically adjusted according to the size of the pixel block to be transmitted. The user terminal re-initializes its video decoder, and the edge server re-initializes its video encoder.

[0018] This invention introduces a dynamic task partitioning mechanism based on runtime data during the rendering process. Addressing the issue of fixed task partitioning ratios or reliance on static performance parameters in existing technologies, it uses historical rendering time and real-time network status to estimate current rendering performance online, thereby achieving adaptive allocation of rendering tasks between user terminals and edge servers. This approach eliminates the need to pre-build computing power models for terminals or servers and can adjust task partitioning strategies promptly when the runtime environment changes, avoiding increased rendering latency due to performance imbalances.

[0019] According to the above technical solution, the pixel block partitioning algorithm solves the pixel block partitioning optimization objective. and The rendering tasks are re-divided; the optimization objective of the pixel block division is to minimize the rendering time of each frame. The rendering time of each frame should be the maximum value of the local rendering time on the user terminal and the remote rendering time. The remote rendering time is the sum of the edge server rendering time, network transmission latency, and network propagation latency. Thus, the following optimization problem arises: (1); Pixel block partitioning constraints: (1.1) , ; (1.2) ; In the formula, Indicates the current time slot The width of the pixel block to be rendered by the mid-edge server. Indicates the current time slot The width of the pixel block to be rendered on the user terminal; w represents the height of the pixel block to be rendered on both the edge server and the user terminal. This indicates the height of the pixel blocks to be rendered on the edge server and user terminal; In time slot t, the edge server renders a frame with a width of The time taken for a pixel block with height h; In time slot t, the user terminal renders a frame with a width of The time taken for a pixel block with height h; This represents the amount of storage space required for one pixel; This indicates the compression ratio of the video encoder. For time slots medium network bandwidth, For time slots Network round-trip latency between mid-edge servers and user terminals.

[0020] Constraint (1.1) indicates that the pixel widths handled by the edge server and the user terminal are integers, and are within the range of 0 to 1. Within the range. Constraint (1.2) means that the sum of the pixel widths handled by the edge server and the user terminal is equal to the width of the image we want to render.

[0021] According to the above technical solution, if the rendering task partitioning method changes, the pixel block partitioning optimization objective is transformed into a rendering optimization objective solution to dynamically adjust the rendering task load undertaken by the user terminal and the edge server respectively; among which, historical time slot data is used to estimate the performance of the current time slot. For the most recent Non-zero time slots, For the most recent Non-zero time slots, initially and None of them are 0, so and It must exist.

[0022] The rendering time for user terminals and edge servers in this time slot is estimated using the most recent valid historical data. Assuming the rendering scene remains unchanged and the viewpoint variation is limited, the rendering time can be approximately assumed to be proportional to the size of the pixel block to be rendered. Therefore, the rendering width of the user terminal in time slot t can be estimated. Time taken for a pixel block with height h and edge server rendering width Time taken for a pixel block with height h : (2); (3); Substituting constraints (1.2), formulas (2) and (3) into (1), we obtain the rendering optimization objective: ; Rendering constraints: , ; Use the following notation to simplify the constants in the optimization objective: ; (6); (7); The optimization problem is as follows: ; Rendering constraints: , ; Examine the straight line and straight line ,because and Therefore, the two lines must intersect at a point, and the x-coordinate of the intersection point is... .

[0023] like , then take , The solution serves as the objective of pixel block partitioning optimization; like and , then take , The solution serves as the objective of pixel block partitioning optimization; Otherwise, take and ,Will and Incorporate into rendering optimization goals Let the minimum rendering optimization target be denoted as Thus obtain , The solution is used as the objective of pixel block partitioning optimization.

[0024] The system includes a technical solution for edge-to-edge collaborative task partitioning and dynamic offloading for real-time rendering, comprising: user terminal, edge server and communication network. The user terminal includes: The user interaction module is used to receive and process user interactions; The local rendering module is used to render the pixel block area allocated to the user terminal; The first asynchronous read module is used to control asynchronous read tasks from video memory to main memory; The network communication module is used to exchange rendering commands and pixel block data with the edge server; The decoding module is used to decode pixel block data from the edge server; The image synthesis module is used to stitch together pixel blocks from the edge server and local rendering results to generate a complete image; The edge server includes: The network communication and monitoring module is used to interact with the user terminal to exchange rendering commands and pixel block data, and to obtain network parameters; The edge rendering module is used to render pixel block regions allocated to the edge server; The second asynchronous read module is used to control asynchronous read tasks from video memory to main memory; The encoding module is used to compress and encode the pixel block results obtained from the rendering. The pixel block partitioning module is used to dynamically determine the size of the pixel blocks that the user terminal and the edge server are responsible for in the current rendering cycle based on the running status information collected during the process.

[0025] The execution steps of the edge-to-edge collaborative task partitioning and dynamic unloading system for real-time rendering include: The pixel block partitioning module randomly divides the image to be rendered into two mutually exclusive, non-empty rectangular pixel blocks along the image width direction. One pixel area is allocated to the user terminal for local rendering, and the other pixel area is allocated to the edge server for rendering.

[0026] The user terminal sets an asynchronous read offset parameter k and notifies the edge server of this parameter; where k represents the frame offset relative to the current rendering cycle. For example, k=1 indicates using the result of the previous rendering cycle. By adopting this asynchronous read method, the data copy latency from GPU memory to host memory can be effectively masked.

[0027] The user terminal updates the rendered view based on user interaction and sends rendering commands and some historical calculation information to the edge server via the network communication module. The edge server receives rendering commands and other information via the network communication module.

[0028] The edge server and the user terminal each start their renderers, corresponding to the local rendering module of the user terminal and the edge rendering module of the edge server, to render the rectangular pixel block area specified by the pixel block division module.

[0029] After the edge server completes the GPU rendering of the specified pixel block, the second asynchronous reading module initiates an asynchronous video memory reading task to asynchronously copy the rendering result of the pixel block of the current frame from the GPU video memory to the host memory; at the same time, it reads the pixel block result of the kth frame ahead according to the offset k.

[0030] The edge server inputs the pixel block data of the k-th frame forward into the encoding module for real-time compression encoding.

[0031] The edge server sends the compressed pixel block rendering results to the user terminal via the network communication and detection module.

[0032] After the user terminal completes the rendering of local pixel blocks, the first asynchronous reading module starts an asynchronous video memory reading operation to asynchronously copy the pixel block results of the current rendering cycle from the GPU video memory to the host memory; at the same time, it extracts the local rendering pixel block results that are k frames backward relative to the current rendering cycle from the asynchronous reading module.

[0033] The user terminal receives pixel block data from the edge server through the network communication module and uses the decoder module to decode the received data in real time.

[0034] The image synthesis module of the user terminal uses the global coordinates recorded in the task structure of each pixel block to stitch together the decoded edge server rendered pixel blocks with the acquired local rendered pixel blocks, thereby synthesizing a complete frame image for display.

[0035] After a specified time has elapsed since the last pixel block division, the pixel block division module re-divides the rendering task. If the division method changes, the user terminal reinitializes its video decoder, and the edge server reinitializes its video encoder, based on the size of the pixel blocks to be transmitted. This process is repeated until the user finishes the rendering task.

[0036] The invention includes a technical solution and a storage medium storing at least one instruction, which is loaded and executed by a processor to implement the edge-to-edge collaborative task partitioning and dynamic unloading method for real-time rendering as described in the above technical solution.

[0037] Compared with the prior art, the beneficial effects achieved by the present invention are: 1. This invention divides each frame of the rendered image into two mutually exclusive pixel blocks. One block is rendered locally by the user terminal, and the other is rendered by an edge server. The edge server sends the rendered pixel block back to the user terminal, which then stitches the server-rendered pixel block with the locally rendered pixel block to form a complete frame. During the rendering process, the system periodically acquires runtime information such as rendering time on the terminal side, rendering time on the edge side, network bandwidth, and round-trip latency. Based on the collected historical performance data, it dynamically adjusts the rendering workload of the terminal and the edge server. This allows for adaptive allocation of the rendering load between the edge and terminal sides without relying on prior computational capability models of the user terminal and the edge server, ensuring full utilization of computing resources on both sides and effectively mitigating the impact of network fluctuations on real-time rendering performance. It is suitable for real-time rendering applications with high real-time and interactivity requirements.

[0038] 2. This invention fully leverages the advantages of edge computing by distributing rendering tasks between the user terminal and the edge server, avoiding the problem that a single computing node cannot simultaneously meet the requirements of high computational complexity and real-time performance. While sharing the rendering tasks, the edge server effectively reduces the terminal's dependence on high-performance graphics hardware, greatly improving the system's deployment flexibility and applicability, especially suitable for low-power or low-configuration ordinary terminals and mobile devices. It improves edge computing efficiency and reduces the computational burden on the terminal side.

[0039] 3. This invention, by collecting and analyzing historical rendering data and network status in real time, enables the system to adaptively adjust task offloading strategies during runtime, thereby avoiding rendering delays caused by unreasonable task allocation in traditional solutions. This mechanism significantly improves overall rendering efficiency and ensures rendering performance across different devices and environments.

[0040] 4. This invention intelligently adjusts task allocation to cope with complex environmental factors such as network bandwidth fluctuations and latency variations. Even under unstable network or bandwidth fluctuation conditions, the system can still guarantee a stable real-time rendering frame rate and interactive response performance, thereby effectively improving the user experience. Regardless of whether the network environment is high-latency or low-bandwidth, the system can provide smooth and fluid real-time rendering effects. It can dynamically adapt algorithms to improve system stability and user experience.

[0041] 5. Because this invention reduces the terminal's reliance on high-performance GPUs, user terminals no longer need to upgrade to expensive high-end graphics hardware. Through edge-to-edge collaborative rendering and dynamic task offloading, real-time rendering technology can be widely used in ordinary terminals, lightweight devices, and multi-user shared scenarios, promoting the popularization and promotion of rendering technology. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an architecture diagram of the edge-to-edge collaborative task partitioning and dynamic unloading system for real-time rendering according to the present invention. Figure 2 This is a flowchart of the edge-to-edge collaborative task partitioning and dynamic unloading method for real-time rendering. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Ray tracing rendering works by emitting rays from the camera's viewpoint to each pixel on the screen. These rays interact with objects in the scene to generate secondary rays, recursively simulating the propagation of light to ultimately calculate the color value of each pixel. The rendering calculation for each pixel can be performed independently, giving ray tracing rendering a natural parallelism. This allows the rendering task to be broken down and processed in parallel across multiple computational units, thus achieving efficient task allocation and offloading.

[0045] Example 1: Based on the characteristic that rendering tasks can be processed in parallel, this invention provides an edge-to-edge collaborative real-time rendering system that supports runtime dynamic adaptation. Figure 1 The system includes a user terminal, an edge server, and a communication network connecting the two. The user terminal includes at least a user interaction module, a local rendering module, a first asynchronous reading module, a network communication module, a decoding module, and an image synthesis module. The edge server includes a network communication and monitoring module, an edge rendering module, a second asynchronous reading module, an encoding module, and a pixel block division module.

[0046] In this system, the edge server and user terminal work collaboratively via a network. After rendering, the edge server asynchronously reads the rendering result from the GPU memory into main memory, masking the time spent reading data from the GPU memory to main memory. Then, it uses video compression technology to compress the pixel blocks and sends them to the user terminal over the network. Upon receiving the data from the edge server, the user terminal decodes it and stitches the locally rendered pixel blocks with the received remote pixel blocks to generate a complete image. The specific steps include: Step S101: The pixel block division module randomly divides the image to be rendered into two mutually exclusive non-empty rectangular pixel blocks along the image width direction. Each pixel block is identified by a task structure containing the global coordinates of the upper left corner, width, and height. One pixel area is allocated to the user terminal for local rendering, and the other pixel area is allocated to the edge server for rendering.

[0047] Step S102: The user terminal sets the asynchronous read offset parameter k and notifies the edge server of this parameter; where k represents the frame offset relative to the current rendering cycle. For example, k=1 indicates using the result of the previous rendering cycle. By adopting this asynchronous read method, the data copy latency from GPU memory to host memory can be effectively masked.

[0048] Step S103: The user terminal updates the rendered view based on user interaction and sends rendering commands and some historical calculation information to the edge server through the network communication module. The edge server receives the rendering commands and other information through the network communication module.

[0049] Step S104: The edge server and the user terminal start their renderers respectively, corresponding to the local rendering module of the user terminal and the edge rendering module of the edge server, and render the rectangular pixel block area specified by the pixel block division module.

[0050] Step S105: After the edge server completes the GPU rendering of the specified pixel block, the second asynchronous reading module initiates an asynchronous video memory reading task to asynchronously copy the rendering result of the pixel block of the current frame from the GPU video memory to the host memory; at the same time, it reads the pixel block result of the kth frame ahead according to the offset k.

[0051] Step S106: The edge server inputs the pixel block data of the k-th frame forward obtained in step S105 into the encoding module for real-time compression encoding.

[0052] Step S107: The edge server sends the compressed pixel block rendering result obtained in step S106 to the user terminal through the network communication and detection module.

[0053] Step S108: After the user terminal completes the rendering of the local pixel block, the first asynchronous reading module starts the asynchronous video memory reading operation to asynchronously copy the pixel block result of the current rendering cycle from the GPU video memory to the host memory; at the same time, it extracts the local rendering pixel block result that is k frames backward relative to the current rendering cycle from the asynchronous reading module.

[0054] Step S109: The user terminal receives pixel block data from the edge server through the network communication module and uses the decoder module to decode the received data in real time.

[0055] Step S110: The image synthesis module of the user terminal splices the edge server rendered pixel block decoded in step S109 with the local rendered pixel block obtained in step S108 according to the global coordinates recorded in the task structure of each pixel block, so as to synthesize a complete frame image and display it.

[0056] Step S111: After a specified time has elapsed since the last pixel block division, the pixel block division module re-divides the rendering task using the algorithm shown in 6.3. If the division method changes, the user terminal re-initializes its video decoder and the edge server re-initializes its video encoder according to the size of the pixel blocks to be transmitted.

[0057] Step S112: Repeat steps S103 to S111 until the user ends the rendering task.

[0058] Example 2: The pixel block partitioning module aims to minimize the single-frame rendering completion time by dynamically partitioning the rendering tasks of the user terminal and the edge server.

[0059] Specifically, for a width of Height is The image to be rendered is divided along its width by the pixel block segmentation module into pixel blocks rendered by the user terminal and pixel blocks rendered by the edge server. One pixel block is [width value missing]. Gao Wei One pixel block is rendered locally on the user's device, and the other block is [width missing]. Gao Wei The pixel blocks are rendered on the edge server. Each time the pixel block partitioning algorithm is triggered, the algorithm will calculate the width of the pixel blocks that the user device and the edge server need to render.

[0060] During operation, the pixel block partitioning module periodically collects the actual rendering time of the user terminal and edge server in historical rendering cycles, and obtains the network bandwidth and round-trip latency parameters of the current time slot through the network communication and detection module. Based on the historical rendering time data, the pixel block partitioning module estimates the rendering time corresponding to different pixel block widths in the current rendering cycle.

[0061] The optimization problem of the pixel block segmentation module is as follows: Pixel block partitioning optimization goal: (1); Pixel block partitioning constraints: (1.1) , ; (1.2) ; In the formula, and These are the current time slots. The width of the pixel blocks to be rendered by the edge server and the user terminal is the value that the algorithm needs to solve for, and the height of the pixel blocks to be rendered by the edge server and the user terminal is... w represents the height of the pixel block to be rendered on the edge server and user terminal; This indicates the height of the pixel blocks to be rendered on the edge server and user terminal; In time slot t, the edge server renders a frame with a width of The time taken for a pixel block with height h; In time slot t, the user terminal renders a frame with a width of The time taken for a pixel block with height h; The amount of storage space required for one pixel. is the compression ratio of the video encoder, which is a fixed constant. For time slots medium network bandwidth, For time slots The network round-trip latency between the mid-edge server and the user terminal can be measured in real time using network tools.

[0062] Use historical time slot data to estimate the performance of the current time slot. For the most recent Non-zero time slots, For the most recent Non-zero time slots, initially and None of them are 0, so and It is bound to exist. The rendering time for user terminals and edge servers in this time slot is estimated using historical data: (2); (3); Substituting the constraints (1.2), formulas (2) and (3) into (1), we obtain the optimization problem: (4); , ; in: (5); (6); (7); Step S205: Examine the straight line and straight line ,because and Therefore, the two lines must intersect at a point, and the x-coordinate of the intersection point is... .

[0063] if , then take , As the solution to the pixel block partitioning optimization objective (Formula (1)).

[0064] if and , then take , As the solution to the pixel block partitioning optimization objective (Formula (1)).

[0065] Otherwise, take and (in To round down to the nearest integer, (for rounding up to the nearest integer), and Substitute into equation (4) Let the one that minimizes (4) be . Thus obtain , The solution is returned as the pixel block partitioning optimization objective (Equation (1)).

[0066] Through the above methods, the present invention can achieve adaptive allocation of rendering load between the edge side and the terminal side without relying on the prior computing capability model of the user terminal and the edge server, so that the computing resources on both sides can be fully utilized and the impact of network fluctuations on real-time rendering performance can be effectively mitigated. It is suitable for real-time rendering application scenarios with high requirements for real-time performance and interactivity.

[0067] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0068] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for edge-to-edge collaborative task partitioning and dynamic unloading for real-time rendering, characterized in that, The steps include: By dividing each frame of the image to be rendered into pixel blocks, each frame of the image is split into two mutually exclusive pixel blocks. Two mutually exclusive pixel blocks are rendered by the user terminal and the edge server respectively, and the running status information of the rendering on the user terminal side and the rendering on the edge server side is periodically obtained during the rendering process. The edge server sends the rendered pixel blocks back to the user terminal; The user terminal stitches together the pixel blocks rendered by the edge server with the pixel blocks rendered locally on the user terminal to form a complete frame image. The rendering workload of user terminals and edge servers is dynamically adjusted based on the collected historical operational status information.

2. The edge-to-edge collaborative task partitioning and dynamic unloading method for real-time rendering according to claim 1, characterized in that, The image to be rendered is randomly divided into two mutually exclusive, non-empty rectangular pixel blocks along the width direction. Each pixel block is identified by a task structure, which stores the global coordinates of the top-left pixel of the pixel block, as well as the width and height of the pixel block.

3. The edge-to-edge collaborative task partitioning and dynamic unloading method for real-time rendering according to claim 2, characterized in that, When the user terminal renders a pixel block, it sets an asynchronous read offset parameter k and notifies the edge server of this parameter; the user terminal sends a rendering command to the edge server. After completing the rendering of the local pixel block, the user terminal initiates an asynchronous video memory read operation to asynchronously copy the pixel block result of the current rendering cycle from the GPU video memory to the host memory; at the same time, it extracts the local rendering pixel block result that is k frames backward relative to the current rendering cycle.

4. The edge-to-edge collaborative task partitioning and dynamic unloading method for real-time rendering according to claim 3, characterized in that, After rendering the pixel block, the edge server asynchronously copies the rendering result of the pixel block of the current frame from the GPU memory to the host memory; at the same time, it reads the pixel block result of the kth frame ahead according to the offset k, performs real-time compression encoding, and sends the compressed pixel block rendering result to the user terminal.

5. The edge-to-edge collaborative task partitioning and dynamic unloading method for real-time rendering according to claim 4, characterized in that, The user terminal receives pixel block data from the edge server and decodes the received data in real time; based on the global coordinates of each pixel block, it splices the decoded edge server rendered pixel block with the local rendered pixel block to synthesize a complete frame image and display it.

6. The edge-to-edge collaborative task partitioning and dynamic unloading method for real-time rendering according to claim 5, characterized in that, After a time interval T since the last pixel block division, the rendering task is re-divided using the pixel block division algorithm. If the rendering task division method changes, the rendering task load of the user terminal and the edge server is dynamically adjusted according to the size of the pixel block to be transmitted. The user terminal reinitializes its video decoder, and the edge server reinitializes its video encoder.

7. The edge-to-edge collaborative task partitioning and dynamic unloading method for real-time rendering according to claim 6, characterized in that, The pixel block partitioning algorithm solves the pixel block partitioning optimization objective. and The rendering task is re-divided; wherein, the optimization objective of the pixel block division is: ； Pixel block partitioning constraints: 1.1） ， ； 1.2） ； In the formula, Indicates the current time slot The width of the pixel block to be rendered by the mid-edge server. Indicates the current time slot The width of the pixel block to be rendered on the user terminal; w represents the height of the pixel block to be rendered on both the edge server and the user terminal. This indicates the height of the pixel blocks to be rendered on the edge server and user terminal; In time slot t, the edge server renders a frame with a width of The time taken for a pixel block with height h; In time slot t, the user terminal renders a frame with a width of The time taken for a pixel block with height h; This represents the amount of storage space required for one pixel; This indicates the compression ratio of the video encoder. For time slots medium network bandwidth, For time slots Network round-trip latency between mid-edge servers and user terminals.

8. The edge-to-edge collaborative task partitioning and dynamic unloading method for real-time rendering according to claim 3, characterized in that, If the rendering task division method changes, the pixel block division optimization objective is transformed into a rendering optimization objective solution to dynamically adjust the amount of rendering tasks undertaken by the user terminal and the edge server respectively. The rendering optimization goal is: ； Rendering constraints: , ; Transform the rendering optimization target into a straight line. and straight line To solve for the rendering optimization objective, the two straight lines must intersect, and the x-coordinate of the intersection point is... ; like So take , The solution serves as the objective of pixel block partitioning optimization; like and So take , The solution serves as the objective of pixel block partitioning optimization; Otherwise, take and ,Will and Incorporate into rendering optimization goals Let the minimum rendering optimization target be denoted as Thus obtain , The solution is used as the objective of pixel block partitioning optimization.

9. A front-end collaborative task partitioning and dynamic unloading system for real-time rendering, characterized in that, include: User terminals, edge servers, and communication networks; The user terminal includes: The user interaction module is used to receive and process user interactions; The local rendering module is used to render the pixel block area allocated to the user terminal; The first asynchronous read module is used to control asynchronous read tasks from video memory to main memory; The network communication module is used to exchange rendering commands and pixel block data with the edge server; The decoding module is used to decode pixel block data from the edge server; The image synthesis module is used to stitch together pixel blocks from the edge server and local rendering results to generate a complete image; The edge server includes: The network communication and monitoring module is used to interact with the user terminal to exchange rendering commands and pixel block data, and to obtain network parameters; The edge rendering module is used to render pixel block regions allocated to the edge server; The second asynchronous read module is used to control asynchronous read tasks from video memory to main memory; The encoding module is used to compress and encode the pixel block results obtained from the rendering. The pixel block partitioning module is used to dynamically determine the size of the pixel blocks that the user terminal and the edge server are responsible for in the current rendering cycle based on the running status information collected during the process.

10. A storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the method as described in any one of claims 1-8.

Images