Cross-platform multi-target rendering method and device based on rendering buffer exchange mechanism

Abstract

The application relates to a cross-platform multi-target rendering method and device based on a rendering buffer exchange mechanism. The method comprises the following steps: encapsulating a 3D rendering kernel into a rendering kernel package and providing a unified interface; allocating a target identifier to each local rendering target and establishing a rendering context including endpoint handle information, a refresh cycle, a reference alignment time, a target state and a back pressure state; calculating a target rendering time aligned with the refresh cycle for each target in a rendering cycle, submitting a frame rendering output and obtaining synchronization barrier information; updating the back pressure state according to a back pressure index and the synchronization barrier, and executing a back pressure suppression strategy when the back pressure index exceeds a threshold value; and performing a state transition operation on the rendering context when a control message is received. Through systematic management of rendering timing and back pressure, consistent management of multiple target independent contexts and cross-platform unified abstraction of the rendering buffer exchange mechanism, the stability of multi-screen parallel rendering in a resource-limited scene can be improved.

Description

Technical Field

[0001] This application relates to the field of computer graphics, and in particular to a cross-platform multi-target rendering method and apparatus based on a rendering buffer exchange mechanism. Background Technology

[0002] With the widespread adoption of smart cockpits, multi-screen in-vehicle systems, industrial control screens, extended reality, and multi-window desktop systems, the parallel presentation of the same 3D content across multiple screens or windows has become a common requirement. The industry typically uses a single rendering kernel to drive multiple display targets, with the rendering side responsible for generating images and the display side responsible for handling windows and interactive input. However, the renderable buffer exchange mechanisms and graphics surface abstractions differ significantly across operating systems. Cross-platform solutions require decoupling the rendering kernel from platform-specific details and sharing the same rendering workflow across multiple targets, resulting in high engineering implementation complexity.

[0003] Existing multi-screen, multi-window rendering solutions often focus on cross-process handle transfer during implementation, lacking systematic management of rendering timing and backpressure. This leads to difficulties in matching rendering submission rhythm with display refresh rhythm in multi-target scenarios, resulting in frame queuing accumulation and screen jitter. Furthermore, due to the lack of maintaining an independent context and state loop for each rendering target, the differences in refresh rate, size, and system scheduling strategies among different targets cannot be managed consistently. Backpressure of one target spreads to other targets, causing resource leaks during binding, unbinding, or size changes. At the same time, the differences in rendering buffer exchange mechanisms across platforms require repeated adaptation for each platform, resulting in high cross-platform development costs. If a frame-by-frame pixel cross-process transmission method is used for remote display, it will continuously consume CPU and memory bandwidth and amplify latency, making it difficult to meet the stable operation requirements of resource-constrained scenarios.

[0004] Therefore, there has been a long-standing desire in this field to propose a cross-platform multi-target rendering method based on a rendering buffer exchange mechanism. This method would enable stable multi-screen and multi-window parallel rendering even in resource-constrained scenarios by systematically managing rendering timing and backpressure, independently managing the context and state of multiple rendering targets, unifying the abstraction of rendering buffer exchange mechanisms across platforms, and avoiding cross-process transmission of pixels per frame. Summary of the Invention

[0005] In order to systematically manage presentation timing and back pressure in multi-screen, multi-window parallel presentation scenarios, provide independent context and state management for multiple presentation targets, unify and abstract the presentation buffer exchange mechanism of different platforms, and avoid cross-process transmission of pixels frame by frame, this application provides a cross-platform multi-target presentation method and apparatus based on the presentation buffer exchange mechanism.

[0006] Firstly, this application provides a cross-platform multi-target rendering method based on a rendering buffer exchange mechanism, which adopts the following technical solution:

[0007] A cross-platform multi-target rendering method based on a rendering buffer exchange mechanism includes:

[0008] S1. Encapsulate the 3D rendering kernel into a rendering kernel package and provide a unified rendering control interface and presentation interface to the outside world. The 3D rendering kernel is at least one of a game engine kernel, a graphics rendering engine kernel, or a graphics interface implementation.

[0009] S2. Based on the rendering kernel package, a rendering service module is encapsulated on the target operating system. The rendering service module performs initialization and lifecycle management of the 3D rendering kernel, and starts the rendering thread, which then schedules the 3D rendering kernel.

[0010] S3. Assign a target identifier (targetId) to at least one local rendering target, and establish a corresponding rendering context for each targetId. Each rendering context is associated with a rendering buffer exchange mechanism. Each rendering context includes at least: endpoint handle information of the rendering buffer exchange mechanism, refresh cycle, etc. Reference alignment time, estimated lead time for rendering and submission, target status, and back pressure status.

[0011] S4. In the rendering loop, for each targetId, calculate the refresh cycle based on the current time, the estimated rendering and submission lead, and the reference alignment time. Aligned target rendering time ; Target presentation time The refresh cycle distance from the reference alignment time is no earlier than the sum of the estimated lead times for rendering and submission, and the current time. The most recent moment that is an integer multiple of the time.

[0012] S5. Based on target presentation time Submit the frame rendering output to the rendering buffer exchange mechanism and obtain the synchronization fence information corresponding to the frame rendering output. .

[0013] S6. Based on the back pressure index fed back by the buffer exchange mechanism. and synchronized fence information Update the back pressure status and in the back pressure index Greater than or equal to the preset back pressure threshold The back pressure suppression strategy is executed at the specified time, and the back pressure suppression strategy includes at least one of the following: waiting until the next target presentation time. Discard the current frame, reduce the internal rendering resolution, or pause frame rendering output submission for targetId.

[0014] S7. During the execution of S3 to S6, in response to receiving a control message related to the targetId, a state transition operation is performed on the presentation context according to the target state; wherein the control message is transmitted through inter-process communication or inter-thread communication.

[0015] By adopting the above technical solution, the rendering kernel is encapsulated into a unified interface rendering kernel package, and the initialization, lifecycle, and rendering thread scheduling are handled by the rendering service module, providing a platform-decoupled runtime foundation for multi-target scenes; the independent rendering context established at the targetId granularity ensures consistent management of the refresh cycle, alignment reference, status, and backpressure of each target, preventing anomalies in a single target from spreading to other targets; based on the target rendering time... Alignment of refresh grid commit cadence with backpressure metrics and synchronization fence The closed-loop feedback constitutes the coordinated governance of timing and backpressure; the endpoint handle is only passed when it is created or rebuilt and no longer moves pixels frame by frame, so that the method can run stably even in resource-constrained scenarios.

[0016] Optionally, S1 includes: packaging the 3D rendering kernel and its platform-related dependencies into a platform adaptation artifact, wherein the target operating system is any one of Android, iOS, Linux, HarmonyOS, or Windows; the platform adaptation artifact is an Android archive (AAR) on the Android platform, and a library or framework artifact that matches the target operating system on non-Android platforms.

[0017] By adopting the above technical solutions, the same rendering kernel can be uniformly delivered and integrated by multiple mainstream operating systems through platform adaptation artifacts, reducing the cost of cross-platform porting and release.

[0018] Optionally, S3 includes: creating a graphics carrier view in the display module and obtaining the rendering buffer exchange mechanism endpoint handle information corresponding to the local rendering target from the graphics carrier view; the graphics carrier view is a view object that can provide a local window handle, a graphics surface handle, or a renderable layer handle.

[0019] By adopting the above technical solution, the acquisition of endpoint handle information is bound to the graphics carrying view of the display module, enabling the rendering service module to access the rendering buffer of multiple platforms without relying on the specific window implementation details, thereby improving the uniformity of the handle acquisition path.

[0020] Optionally, the rendering buffer swapping mechanism endpoint handle information includes an endpoint handle identifier used to identify the same rendering buffer swapping mechanism across different processes or threads; the rendering buffer swapping mechanism endpoint handle information is sent only through inter-process communication or inter-thread communication when the local rendering target is created or rebuilt, and during subsequent frame rendering, only control messages are sent through inter-process communication or inter-thread communication, without sending frame-by-frame pixel data; wherein, the rendering buffer swapping mechanism is one of the following: BufferQueue on the Android platform, swap chain on the Windows platform, shared graphics buffer sequence or graphics interface-based swap chain on the Linux platform, and renderable texture or renderable drawing object sequence on the iOS platform.

[0021] By adopting the above technical solution, the transmission of endpoint handle information is limited to a one-time action during creation or reconstruction. Frame-by-frame pixels do not need to be moved across processes or threads, and a unified lower bound on transmission overhead is obtained across the rendering buffer exchange mechanisms of multiple mainstream platforms.

[0022] Optionally, S3 also includes: selecting or creating a graphics backend in the rendering service module based on the rendering buffer exchange mechanism endpoint handle information, and constructing a rendering context based on the graphics backend, wherein the graphics backend is one of OpenGLES, Vulkan, or Metal.

[0023] By adopting the above technical solution, the rendering service module adaptively selects the mainstream graphics interface as the graphics backend based on the endpoint handle information, so that the construction of the rendering context can obtain a consistent access point across different graphics stacks.

[0024] Optionally, in S7, control messages include at least one of the following: render target binding message, size change message, render target unbinding message, pause message, resume message, render backend switching message, scene switching message, or input event message; state transition operations performed on the rendering context include at least one of the following: create, rebuild, destroy, pause, or resume operations.

[0025] By adopting the above technical solution, the control message types and state transition operations are clearly listed, enabling the rendering service module to cover the full lifecycle management of the rendering target under typical events such as binding, size change, pause and resume, backend switching, and scene switching.

[0026] Optionally, S2 also includes: setting up a message queue through the rendering service module to decouple the communication thread from the rendering thread; and in each rendering loop of the rendering thread, the rendering thread first retrieves control messages from the message queue and updates the rendering state, and then executes the rendering and presentation operations in S4 to S6.

[0027] By adopting the above technical solution, the communication thread and the rendering thread are decoupled through a message queue. The rendering thread retrieves the control message and updates the rendering state once at the beginning of each loop, avoiding the preemption of the main rendering process by the arrival of external messages.

[0028] Optionally, it also includes: mapping the touch coordinates in the input event from the display coordinate system to the rendering coordinate system through the display module. The mapping includes: determining the viewport size of the rendered content relative to the local rendering target according to the principle of proportional conformity preservation, centering the viewport within the local rendering target, and normalizing the position of the touch coordinates relative to the viewport to normalized coordinates in the rendering coordinate system. The normalized coordinates are used to restore the corresponding rendering space coordinates or interaction rays in the rendering service module.

[0029] By adopting the above technical solution, the touch coordinates on the display side are mapped to the rendering coordinate system in a proportionally conformal and center-normalized manner, so that the interactive input maintains geometric consistency under different resolutions and viewport sizes.

[0030] Optional, back pressure index It is obtained by weighted fusion of at least two components of frame queue depth, synchronization barrier incomplete delay, and end-to-end delay of submission to presentation, or by taking the maximum value of multiple components.

[0031] By adopting the above technical solution, the back pressure index The congestion status of the buffer exchange mechanism is characterized by multi-component fusion, reflecting both queuing depth and time delay signals, thus reducing misjudgments caused by the failure of a single signal.

[0032] Optionally, the back pressure suppression strategy adopts a hysteresis band mode, that is, in the back pressure index When the back pressure index exceeds the upper threshold, it enters a suppressed state. When the threshold value is less than the lower threshold, the system exits the inhibition state, and a hysteresis interval with no action is formed between the upper threshold and the lower threshold.

[0033] By adopting the above technical solution, the hysteresis interval avoids the back pressure indicator. The frequent switching of the inhibition state caused by traversing back and forth around a single threshold keeps the inhibition behavior stable near the boundary.

[0034] Optionally, the backpressure suppression strategy can be cascaded in the following order: waiting, reducing the internal rendering resolution, discarding the current frame, and pausing the frame rendering output submission for targetId. The previous stage can be set based on the duration or backpressure metric. Once the growth rate exceeds the preset threshold, the system switches to the next level; and the back pressure indicator... When it falls back, it falls back in reverse order, step by step.

[0035] By adopting the above technical solution, the suppression behavior is gradually escalated from mild to severe and then gradually decreased in the opposite order, which avoids excessive suppression under mild congestion and prevents re-congestion caused by a one-time release.

[0036] Optionally, the selection of the back pressure suppression strategy is determined by a parameterized decision tree or a parameterized strategy table, the input of which includes the back pressure index. Each component and the synchronization fence The completion delay is given, and the output is the type and parameters of the suppression strategy.

[0037] By adopting the above technical solution, the selection of suppression strategy is externalized as an adjustable parameter. Customization can be achieved by adjusting the decision tree or strategy table under different hardware and scenarios without modifying the main method.

[0038] Optional, based on back pressure index The rate of change in the recent period predicts the back pressure index. Reaching the preset back pressure threshold When the predicted arrival time is earlier than the sum of the current time and the preset prediction window, the system enters the back pressure suppression state ahead of time.

[0039] By adopting the above technical solution, the entry of the suppressed state is changed from a delayed response to a predictive response, so that congestion is relieved in advance before reaching the threshold, reducing the probability of frame loss at peak time.

[0040] Optionally, when multiple targetIds share a presentation buffer exchange mechanism or related system resources, the backpressure suppression strategy can be coordinated across multiple targetIds, prioritizing budget allocation for critical targetIds and proactively reducing the submission pace for non-critical targetIds.

[0041] By adopting the above technical solutions, the presentation quality of key objectives is prioritized when shared resources are scarce, while the submission schedule of non-key objectives is proactively relinquished, thus avoiding the spread of chain-like back pressure.

[0042] Optional, refresh cycle The results are obtained by performing statistical estimation on the presentation completion time series of the presentation buffer swapping mechanism. The statistical estimation is independent of the operating system's open interface for variable refresh rates.

[0043] By adopting the above technical solution, the refresh cycle is reduced. The acquisition is freed from dependence on the operating system's variable refresh rate interface, enabling the method to track the refresh rate even on platforms where this interface is not exposed.

[0044] Optionally, responds to refresh cycle When a switch occurs, the reference alignment time is reset to the switch time, so that subsequent target rendering times are displayed. Realign to the new grid.

[0045] By adopting the above technical solution, the refresh cycle is reduced. During the switch, the raster reference is reset synchronously to avoid using calculations based on the old reference. It falls outside the phase of the new grid.

[0046] Optionally, a phase offset can be set for the reference alignment time of multiple targetIds to adjust the target rendering time of each targetId. It avoids concentrating on the same refresh grid, thus smoothing out peak resource usage.

[0047] By adopting the above technical solution, the submission times of multiple targets are artificially phased, reducing resource contention at the same time and facilitating a stable peak load when using a shared presentation buffer exchange mechanism.

[0048] Optionally, the estimated lead time for rendering and submission is adaptively adjusted based on recent rendering time statistics to ensure that rendering completion and display refresh are as close as possible.

[0049] By adopting the above technical solution, the estimated lead time dynamically converges with the rendering time distribution, which avoids unnecessary waiting caused by excessive lead time and avoids grid misses caused by insufficient lead time.

[0050] Optionally, the reference alignment time is initialized during the first frame rendering based on the first return time of the render buffer swapping mechanism and remains stable thereafter until the refresh cycle. Switch.

[0051] By adopting the above technical solution, the reference alignment time obtains an initialization method that does not rely on an external clock source, enabling the method to have a consistent grid reference establishment path on different platforms.

[0052] Optionally, at the time of target presentation When the actual deviation from the refresh grid exceeds the preset tolerance, a recalibration of the reference alignment time is triggered.

[0053] By adopting the above technical solution, grid drift is detected and corrected, avoiding long-term cumulative deviations. Move away from the nearest refresh grid.

[0054] Optionally, for synchronization fences The status query employs a throttling strategy, at each target presentation time. The nearest query is performed sparsely to avoid the extra overhead of high-frequency polling.

[0055] By adopting the above technical solution, the query frequency of the fence status is bound to the target presentation time. Nearby locations reduce the number of invalid queries.

[0056] Optionally, the synchronized fence object adopts a pooling reuse mechanism to avoid the memory and system call overhead caused by allocating and releasing fence objects every frame.

[0057] By adopting the above technical solution, the lifecycle of the fence object is reused, reducing the object management overhead per unit time in high frame rate scenarios.

[0058] Optional, responds to synchronization fence Completed and back pressure indicators Falling back to the preset back pressure threshold The following is an exit from the back pressure suppression state.

[0059] By adopting the above technical solution, the exit of the suppressed state is driven by the fence completion and back pressure fall, rather than a single time window, so that the exit time is aligned with the actual progress.

[0060] Optionally, when multiple targetIds share the same display device, the refresh cycle of one of the targetIds can be used. The refresh cycle for other targetIds is used as a benchmark. Perform mutual calibration.

[0061] By adopting the above technical solution, the refresh cycle of each target under the shared display device is achieved. The estimates are mutually constrained, reducing the relative drift caused by independent estimates.

[0062] Optional, synchronization fence The completion time is used as a feedback quantity for the estimated lead time of rendering and submission, and the estimated lead time is corrected based on the difference between the fence completion and submission times.

[0063] By adopting the above technical solution, the fence completion timing becomes a closed-loop feedback signal for the estimated lead time, enabling the lead time to converge adaptively with the time delay changes of the actual presentation path.

[0064] Secondly, this application provides a cross-platform multi-target rendering device based on a rendering buffer exchange mechanism, which adopts the following technical solution:

[0065] A cross-platform multi-target rendering device based on a rendering buffer exchange mechanism, comprising:

[0066] The encapsulation module is used to encapsulate the 3D rendering kernel into a rendering kernel package and provide a unified rendering control interface and presentation interface to the outside world;

[0067] The rendering service module runs on the target operating system and is built based on the rendering kernel package. The rendering service module is used for initializing the 3D rendering kernel, managing its lifecycle, and scheduling rendering threads.

[0068] The context module is used to assign a target identifier (targetId) to at least one local rendering target and to establish a corresponding rendering context for each targetId. Each rendering context is associated with a rendering buffer exchange mechanism. Each rendering context includes at least: endpoint handle information of the rendering buffer exchange mechanism, refresh cycle, etc. Reference alignment time, estimated lead time for rendering and submission, target status, and back pressure status;

[0069] The beat module is used to calculate the refresh cycle alignment for each targetId during the rendering loop. Target presentation time And based on the target presentation time Submit the frame rendering output;

[0070] The back pressure module is used to obtain the synchronization fence information corresponding to the frame rendering output. and back pressure indicators and in back pressure index Greater than or equal to the preset back pressure threshold The back pressure suppression strategy is executed at the specified time, and the back pressure suppression strategy includes at least one of the following: waiting until the next target presentation time. Discard the current frame, reduce the internal rendering resolution, or pause the frame rendering output submission for targetId;

[0071] The management module is used to perform a state transition operation on the presentation context based on the target state when it receives a control message related to the targetId; wherein the control message is transmitted through cross-process communication or cross-thread communication.

[0072] By adopting the above technical solution, the encapsulation module, rendering service module, context module, beat module, backpressure module and management module work together to implement rendering kernel encapsulation, platform adaptation, multi-target context management, timing alignment, backpressure closed loop and control message state machine in the form of device modularization, so that the step sequence on the method side has a one-to-one corresponding carrier component on the device side, and the stable governance of multi-screen and multi-window parallel presentation is completed through the cooperation between modules.

[0073] Optionally, the context module is used to abstract the local rendering target into a rendering target description that includes a platform identifier, platform handle, width, height, and at least one rendering hint parameter; the rendering service module is used to select a graphics backend and create a window surface or swap chain based on the rendering target description; and the beat module is used to write the frame rendering output to the renderable buffer corresponding to the rendering buffer swap mechanism in the rendering loop.

[0074] By adopting the above technical solution, the local rendering target is abstracted into a unified rendering target description, enabling the rendering service module to complete the graphics backend selection and surface creation based on this description. The output of the beat module is directly written to the corresponding renderable buffer, reducing the access complexity under multiple platforms.

[0075] In summary, this application includes at least one of the following beneficial technical effects:

[0076] 1. By encapsulating the 3D rendering kernel into a rendering kernel package and using the rendering service module to handle initialization, lifecycle, and rendering thread scheduling, the target rendering time is aligned with the refresh raster. Synchronous fence With back pressure index The closed-loop governance ensures that the rendering submission rhythm and display refresh rhythm are matched under the parallel presentation of multiple screens and multiple windows, and the accumulation of frame queues and screen jitter are suppressed.

[0077] 2. By establishing independent rendering contexts at the targetId level and managing refresh cycles, alignment benchmarks, lead time, target status, and back pressure status, the operating parameters and anomalies of each target are localized. The back pressure of a certain target no longer spreads to other targets, and a clear state transition path is obtained when binding, unbinding, or changing size, reducing resource leakage and screen anomalies in multi-target scenarios.

[0078] 3. By transmitting endpoint handles only once during creation or reconstruction, and subsequently transmitting only control and interaction messages without transmitting frame-by-frame pixels, as well as enhancing backpressure suppression strategies such as multi-component backpressure indices, hysteresis bands, cascading upgrades, cross-target coordination, predictive triggering, and adaptive lead estimation, the device achieves a stable lower bound on transmission overhead and suppression behavior in resource-constrained scenarios. Attached Figure Description

[0079] Figure 1 This is a schematic diagram of the overall architecture of a cross-platform multi-target presentation system in one embodiment of this application.

[0080] Figure 2 This is a flowchart illustrating a cross-platform multi-target presentation method in one embodiment of this application.

[0081] Figure 3 This is a schematic diagram of the structure of a cross-platform multi-target presentation device in one embodiment of this application. Detailed Implementation

[0082] The present application will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the scope of the application.

[0083] For ease of understanding, please refer to Figure 1First, we will explain some of the terms used in the embodiments of this application.

[0084] A 3D rendering kernel is a core component that carries fundamental rendering capabilities such as scene management, resource loading, rendering pipeline, material system, animation and physics, and a rendering adaptation layer that interacts with the graphics backend. As an example, a 3D rendering kernel can be a game engine kernel, such as one based on a scene graph and physical animation system; it can also be a graphics rendering engine kernel, such as one for data visualization or industrial simulation; or it can be a graphics interface implementation, such as a lightweight rendering implementation based on OpenGLES, Vulkan, or Metal.

[0085] The rendering service module is a runtime entity encapsulated on the target operating system based on the rendering kernel package. It is used to handle the initialization of the 3D rendering kernel, lifecycle management, rendering thread scheduling, rendering context management, synchronization fence processing, and backpressure closed-loop control. The rendering service module can run as an independent process or as an independent thread within the same process; the deployment form is not limited in this embodiment.

[0086] The display module is a runtime entity that hosts the graphical view, collects input events, and sends control or interaction messages to the rendering service module. The graphical view is a view object that can provide a local window handle, a graphical surface handle, or a renderable layer handle, such as SurfaceView or TextureView on Android, a window or graphical surface on Windows, or a renderable layer on iOS.

[0087] The rendering buffer swapping mechanism is a set of mechanisms or interfaces for exchanging renderable buffers between the rendering and display ends, serving as the cross-platform abstraction boundary of this application. Different operating system platforms correspond to different underlying forms. As examples, the rendering buffer swapping mechanism on the Android platform is a buffer queue mechanism BufferQueue and its endpoint handles; on the Windows platform, it is a swap chain and its associated object handles; on the Linux platform, it is a sequence of shareable graphics buffers or a swap chain based on a graphics interface; and on the iOS platform, it is a sequence of renderable textures or renderable drawing objects, such as renderable layers and their drawable object queues. The above are merely examples and do not constitute a limitation on the scope of protection.

[0088] The endpoint handle information of the rendering buffer exchange mechanism is an information carrier used to identify the same rendering buffer exchange mechanism between different processes or different threads; the endpoint handle information contains an endpoint handle identifier, which serves as a reference to the same rendering buffer exchange mechanism between the display module and the rendering service module.

[0089] The rendering context is a runtime data structure that corresponds one-to-one with each targetId, and includes at least the endpoint handle information of the corresponding rendering buffer swapping mechanism and the refresh cycle. Reference alignment time, estimated lead time for rendering and submission, target state, and backpressure state. Target states should include at least the following: unbound, bound, paused, destroyed, etc.; backpressure states should include at least the following backpressure metrics. Preset back pressure threshold And the current suppression strategy state. In optional implementations, the presentation context may also include runtime information such as graphics backend type, window surface or swap chain object, resource reuse pool, and the previous frame synchronization fence handle.

[0090] Target presentation time It is calculated by the rendering service module for each targetId during the rendering loop, and is aligned with the refresh cycle. Submission time of discrete grids. Synchronizing fence information. A synchronization object is a one-to-one correspondence with each frame of rendering output. It is used to express the execution progress of the GPU and the rendering pipeline, making it easy to determine whether the frame is complete and when the relevant resources can be safely reused. The synchronization barrier can be implemented by the synchronization object, event handle or timestamp returned by the graphics interface. This application does not limit its specific implementation.

[0091] Back pressure index Backpressure is a scalar or multi-component measure used to quantify the pressure caused by insufficient throughput at the display end. Backpressure suppression strategies are implemented when the backpressure index... Greater than or equal to the preset back pressure threshold At that time, the rendering service module performs a set of optional actions to alleviate or eliminate congestion.

[0092] Control messages are messages used to drive the rendering context state machine, and include at least one or more of the following: rendering target binding messages, size change messages, rendering target unbinding messages, pause messages, resume messages, rendering backend switching messages, and scene switching messages. Interaction messages are messages that carry input events. Control messages and interaction messages can be transmitted using key-value pairs, structured binary, or other serialization formats.

[0093] This application discloses a cross-platform multi-target presentation method based on a presentation buffer exchange mechanism. This method can be applied to scenarios where the same 3D content needs to be presented in parallel on multiple screens or windows, such as smart cockpits, in-vehicle multi-screen displays, industrial control large screens, extended reality head-mounted displays, and multi-window desktop systems.

[0094] This method uses a presentation buffer exchange mechanism as the cross-platform abstraction boundary, dividing the system into two roles: a rendering service module and a display module. It establishes optional cross-thread or cross-process communication channels between the two through control messages and interaction messages. The rendering service module can run as an independent process or as an independent thread within the same process. The rendering service module and the display module can run within the same process, across processes, or on different device nodes via a network carrying control messages and endpoint handle information. This application does not limit their deployment relationship.

[0095] For ease of explanation, the following implementation scenario uses a dual-screen smart cockpit environment as the benchmark throughout the text: the central control screen corresponds to targetId=1, with a rendering baseline size of 1920×1080; the passenger screen corresponds to targetId=2, with a rendering baseline size of 1280×720; the display device refresh rate is 60Hz, i.e., refresh cycle... Approximately 16.67 milliseconds; initial estimated lead time for rendering and submission is approximately 5 milliseconds; preset backpressure threshold. The value is 3, where the frame queue depth is used as the baseline component. The numerical examples in this baseline scenario will remain consistent in subsequent steps.

[0096] Reference Figure 2 This method includes steps S1 to S7, and each step is described in detail below.

[0097] S1. Encapsulate the 3D rendering kernel into a rendering kernel package and provide a unified rendering control interface and presentation interface to the outside world, wherein the 3D rendering kernel is at least one of a game engine kernel, a graphics rendering engine kernel, or a graphics interface implementation.

[0098] In S1, the 3D rendering kernel is encapsulated into a rendering kernel package, providing a unified rendering control interface and rendering interface. The rendering kernel package may include, but is not limited to, scene management, resource loading, rendering pipeline, material system, animation and physics, and a rendering adaptation layer for interacting with the graphics backend. The unified rendering control interface carries control capabilities such as initialization, destruction, pause, resumption, scene switching, and backend switching; the unified rendering interface carries rendering capabilities such as frame submission, synchronization fence acquisition, and backpressure feedback. By decoupling the 3D rendering kernel from its platform binding details and providing capabilities externally through a unified interface, upper-layer applications do not need to be aware of the specific source and implementation of the underlying rendering kernel, allowing the same rendering kernel package to handle rendering tasks for multiple local rendering targets.

[0099] In some embodiments, S1 packages the 3D rendering kernel and its platform-related dependencies into a platform adaptation artifact. For example, the target operating system is any one of Android, iOS, Linux, HarmonyOS, or Windows; the platform adaptation artifact is an Android archive (AAR) on the Android platform, containing a Java or Kotlin interface layer and native libraries; on the iOS platform, it is a static library, dynamic library, or framework; and on the Linux, Windows, and HarmonyOS platforms, it is a library or framework artifact that matches the target operating system. The unified interface exposed by the platform adaptation artifact remains consistent across platforms, enabling the same rendering kernel to be uniformly delivered and integrated by multiple mainstream operating systems through the platform adaptation artifact.

[0100] S2. Based on the rendering kernel package, a rendering service module is encapsulated on the target operating system. The rendering service module performs initialization and lifecycle management of the 3D rendering kernel, and starts the rendering thread through the rendering service module. The rendering thread performs scheduling of the 3D rendering kernel.

[0101] In S2, a rendering service module is encapsulated on the target operating system based on the aforementioned rendering kernel package. This module initializes and manages the lifecycle of the 3D rendering kernel, and starts rendering threads, which then schedule the rendering kernel. The rendering service module handles the runtime management of the 3D rendering kernel, while the rendering threads are the main execution body of the rendering loop. This separation of responsibilities ensures that initialization and lifecycle management actions are not blocked by the continuous execution of the rendering loop. In the baseline scenario, the rendering service module runs as an independent thread within the cockpit host process after system startup. Once started, the rendering thread enters the main loop, scheduling the 3D rendering kernel to drive the rendering of targetId=1 and targetId=2.

[0102] In some embodiments, in S2, a message queue is set up through the rendering service module to decouple the communication thread and the rendering thread. In each rendering loop of the rendering thread, the rendering thread first retrieves control messages from the message queue and updates the rendering state before executing the rendering and presentation operations in S4 to S6. The message queue adopts a first-in, first-out (FIFO) storage structure. After receiving control messages or interaction messages from the display module, the communication thread writes the messages to the message queue. At the beginning of the loop, the rendering thread retrieves all visible messages for that round and updates the presentation context corresponding to the targetId according to the message type. This mechanism ensures that the arrival rhythm of external messages does not preempt the main flow of the rendering loop, maintaining a stable timing within the rendering loop.

[0103] In some embodiments, the rendering service module merges or downsamples size change messages, pose change messages, or viewport change messages arriving consecutively within a preset time window for the same targetId, updating the rendering context only based on the last message within the time window to avoid resource waste caused by multiple reconstructions in a short period. In the baseline scenario, the rendering service module sets the merging time window to 50 milliseconds. When the passenger screen with targetId=2 receives three consecutive size change messages within 30 milliseconds, the rendering service module performs a reconstruction only based on the size carried by the last message. In some embodiments, the rendering service module processes control messages according to the priority or timestamp causal relationship of destruction, unbinding, reconstruction, size change, pause, resumption, and input events, ensuring that critical state changes are responded to before secondary messages. In the baseline scenario, if targetId=2 has both size change messages and input event messages to be processed, the rendering service module performs the reconstruction caused by the size change first, and then processes the input event, ensuring that subsequent input events fall into the correct viewport geometry.

[0104] S3. Assign a target identifier (targetId) to at least one local rendering target, and establish a corresponding rendering context for each targetId. Each rendering context is associated with a rendering buffer exchange mechanism. Each rendering context includes at least: endpoint handle information of the rendering buffer exchange mechanism, and refresh cycle. Reference alignment time, estimated lead time for rendering and submission, target status, and back pressure status.

[0105] In S3, the rendering service module assigns a target identifier (targetId) to at least one local rendering target and establishes a corresponding rendering context for each targetId. Each rendering context is associated with a rendering buffer exchange mechanism. Each rendering context includes at least the endpoint handle information and refresh cycle of the corresponding rendering buffer exchange mechanism. The system considers the alignment time, estimated lead time for rendering and submission, target status, and backpressure status. In the baseline scenario, the rendering service module assigns targetId=1 to the central control screen and establishes the first rendering context, and assigns targetId=2 to the passenger screen and establishes the second rendering context. Each rendering context maintains its own refresh cycle. Approximately 16.67 milliseconds, initial reference alignment time of 0 milliseconds, initial estimated lead time of 5 milliseconds for rendering and submission, and their respective target states and backpressure states. By establishing independent rendering contexts at the targetId granularity, the running parameters and anomalies of each target are handled locally, and the backpressure of one target will not spread to other targets.

[0106] In S3, the acquisition of endpoint handle information is bound to the display module. Specifically, the display module creates a graphics carrier view and obtains the endpoint handle information of the rendering buffer exchange mechanism corresponding to the local rendering target from the graphics carrier view. The graphics carrier view is a view object that can provide a local window handle, graphics surface handle, or renderable layer handle. In the baseline scenario, the display module creates a first graphics carrier view for targetId=1 and obtains the endpoint handle information associated with the central control screen from it; it creates a second graphics carrier view for targetId=2 and obtains the endpoint handle information associated with the passenger screen from it.

[0107] The rendering buffer exchange mechanism endpoint handle information includes endpoint handle identifiers used to identify the same rendering buffer exchange mechanism across different processes or threads. This endpoint handle information is sent only during the creation or reconstruction of the local rendering target via inter-process or inter-thread communication. During subsequent frame rendering, only control messages are sent via inter-process or inter-thread communication, without sending frame-by-frame pixel data. In the baseline scenario, the endpoint handle information for the central control screen and the passenger screen is sent once each during system startup. During subsequent frame rendering, the display module only transmits control and interaction messages to the rendering service module. Frame-by-frame pixel data is always directly written to the corresponding rendering buffer exchange mechanism within the process of the rendering service module. The bandwidth usage of inter-process communication is locked to the control plane rather than the data plane. On different mainstream platforms, the rendering buffer exchange mechanism corresponding to the endpoint handle information has different underlying implementations. On the Android platform, the rendering buffer swapping mechanism specifically manifests as one or more of the following: BufferQueue, IGraphicBufferProducer, Surface, or ANativeWindow. On the Windows platform, it specifically manifests as one or more of the following: IDXGISwapChain, shared texture handle, or renderable swapping chain. On the Linux platform, it specifically manifests as one or more of the following: dma-buf file descriptor, shared graphics buffer sequence, or graphics interface-based swapping chain. On the iOS platform, it specifically manifests as one or more of the following: CAMetalLayer, CAMetalDrawable, or renderable drawing object sequence. The specific platform implementations described above are merely examples and do not constitute a limitation on the scope of protection of this application.

[0108] In some embodiments, the endpoint handle information of the rendering buffer exchange mechanism can be further organized into a structured endpoint description message. This structured endpoint description message includes at least one of the following: targetId, endpoint handle identifier, handle version number, platform type identifier, native handle payload, number of renderable buffers, pixel format, color space, size information, lifecycle flag, permission token, and integrity verification field. The structured endpoint description message is sent via inter-process communication, inter-thread communication, or inter-device network communication when the local rendering target is created, rebuilt, the exchange chain is rebuilt due to size changes, the permission domain changes, or the handle version changes. The rendering service module identifies expired, forged, or unauthorized handles by comparing the handle version number, lifecycle flag, integrity verification field, and permission token. If an anomaly is detected, the corresponding message is discarded, a retransmission of the endpoint description message is requested, or a rebinding is triggered. In the baseline scenario, the display module assigns handle version number 1 to the central control screen when targetId=1 is first created. When the size of the central control screen changes from 1920×1080 to 1920×1200, causing the exchange chain to be rebuilt, the display module increments the handle version number to 2 and resends the structured endpoint description message. The rendering service module compares the version number and finds that the old version 1 endpoint description message has expired. It discards any lingering messages carrying version number 1 and only performs rebinding based on the latest structured endpoint description message with version number 2, thus avoiding the misuse of expired handles in rapid reconstruction scenarios.

[0109] In some embodiments, S3 further includes a rendering service module that selects or creates a graphics backend based on endpoint handle information, and constructs a rendering context based on the graphics backend. The graphics backend is one of OpenGLES, Vulkan, or Metal. In the baseline scenario, the rendering service module selects Vulkan as the graphics backend based on the endpoint handle information of the Android platform, creates window surfaces or swap chain objects for targetId=1 and targetId=2 respectively based on the Vulkan backend, and attaches them to the corresponding rendering contexts.

[0110] In some embodiments, on Windows platforms or other target operating systems that support the Direct3D interface, the rendering service module may also select Direct3D as the graphics backend to build the rendering context to adapt to the platform's local graphics stack. In some embodiments, the rendering service module shares at least one of the following among multiple targetIds: a graphics backend instance, a command allocator, or a synchronization fence object pool. Simultaneously, it independently maintains a window surface or swap chain object, refresh cycle T, reference alignment time t_0, estimated rendering and submission lead Δ, target state, and backpressure state for each targetId. In the baseline scenario, the rendering service module shares the same Vulkan backend instance and the same command allocator pool for the central control screen (targetId=1) and the passenger screen (targetId=2), but creates independent Vulkan window surface objects for each. This allows the two screens to share the GPU resource scheduling channel while their respective rendering rhythm, submission rhythm, and backpressure state are maintained locally, reducing resource overhead in multi-target scenarios.

[0111] In some embodiments, the reference alignment time is initialized at the first frame rendering based on the first return time of the render buffer swapping mechanism and remains stable thereafter until the refresh cycle. Switching. In the baseline scenario, after the rendering service module successfully submits the first frame with targetId=1 to the rendering buffer exchange mechanism and receives the first return time, it records this return time as the reference alignment time for targetId=1; targetId=2 also independently initializes its own reference alignment time. This method ensures that the establishment of the reference alignment time does not depend on an external clock source, providing a consistent raster baseline establishment path across different platforms.

[0112] S4. In the rendering loop, for each targetId, calculate the refresh cycle based on the current time, the estimated rendering and submission lead, and the reference alignment time. Aligned target rendering time The target presentation time The refresh cycle is no earlier than the sum of the estimated lead time for the current time and the estimated lead time for rendering and submission, and is located at the reference alignment time. The most recent moment that is an integer multiple of the time.

[0113] In S4, during the rendering loop, for each targetId, the refresh cycle is calculated based on the current time, the estimated rendering and submission lead, and the reference alignment time. Aligned target rendering time ; Target presentation time The refresh cycle distance from the reference alignment time is no earlier than the sum of the estimated lead times for rendering and submission, and the current time. The most recent moment that is an integer multiple of the time.

[0114] Specifically, the moment the target is presented The following computational relationship must be satisfied: Let for The floor value is then .in, For the current moment, Estimated lead time for rendering and submission, For reference alignment time.

[0115] In the baseline scenario, let's take the calculation of targetId=1 in a certain rendering loop as an example. Assume that at the current moment... 100 milliseconds, reference alignment time 0 milliseconds, estimated lead time for rendering and submission The refresh cycle is 5 milliseconds. It takes approximately 16.67 milliseconds. Substituting this into the calculation relationship, Round up to get ,therefore Milliseconds. Verification is available. Satisfying no earlier than That is, a constraint of 105 milliseconds, and aligned with the reference alignment time. The 7th grid refresh.

[0116] By presenting the target moment Align to refresh cycle The discrete raster ensures that the submission rhythm of the rendering service module matches the refresh rhythm of the display device in multi-target scenarios, thus suppressing frame queuing accumulation and screen jitter.

[0117] In some embodiments, the estimated lead time for rendering and submission This can be obtained from historical frame statistics, such as by using a sliding window to calculate the quantiles or weighted average of rendering time and submission time, and adaptively adjusting based on jitter. In the baseline scenario, the rendering service module uses the weighted average rendering time of the most recent 32 frames as... The estimate makes Dynamic convergence based on rendering time distribution avoids both excessive waiting caused by excessive lead time and missed raster frames caused by insufficient lead time.

[0118] In some embodiments, refresh cycle This is obtained by performing statistical estimation on the rendering completion time series of the rendering buffer swapping mechanism. This statistical estimation is independent of the operating system's open interface for variable refresh rates. For example, the rendering service module uses the median difference between the most recent multiple rendering completion times as the refresh period. The estimated value. This method makes the refresh cycle... The acquisition of refresh rate data is independent of the operating system's variable refresh rate interface, and the refresh rate can be tracked even on platforms where this interface is not open.

[0119] In some embodiments, in response to a refresh cycle When a switch occurs, the reference alignment time is reset to the switch time, so that subsequent target rendering times are displayed. Realign to the new grid. In the baseline scenario, if the display device switches from 60 Hz to 90 Hz, the refresh rate... The timeout changed from approximately 16.67 milliseconds to approximately 11.11 milliseconds. The rendering service module recorded the switch moment as the new reference alignment moment, and subsequently... The calculation is based on a new and Perform the calculations, avoiding the use of old benchmarks. It falls outside the phase of the new grid.

[0120] In some embodiments, a phase offset is set for the reference alignment times of multiple targetIds, so that the target rendering times of each targetId are... Not concentrated on the same refresh raster. In the baseline scene, the rendering service module introduces approximately [missing information] for the reference alignment time of targetId=2. The phase shift of approximately 8.33 milliseconds artificially disrupts the timing of submissions from the central control screen and the passenger screen, reducing resource contention for the presentation buffer exchange mechanism at the same time and achieving a stable peak load.

[0121] In some embodiments, at the time of target presentation When the actual deviation from the refresh raster exceeds the preset tolerance, a recalibration of the reference alignment time is triggered. In the baseline scenario, the rendering service module sets the tolerance to approximately 1 millisecond. When several consecutive rounds of observation... When the deviation from the actual refreshed grid exceeds 1 millisecond, the original reference alignment time is replaced with the most recent refreshed grid time, thereby correcting the long-term accumulated grid drift.

[0122] In some embodiments, when multiple targetIds share the same display device, the refresh cycle of one of the targetIds is used. The refresh cycle for other targetIds is used as a benchmark. Perform mutual calibration. In the baseline scenario, when the central control screen and the passenger screen are connected to the same graphics subsystem, the targetId=1 is used. Based on the estimate, for targetId=2 Estimation should include convergence constraints to reduce relative drift caused by independent estimations.

[0123] S5. Based on the target presentation time Submit the frame rendering output to the rendering buffer exchange mechanism and obtain the synchronization fence information corresponding to the frame rendering output. .

[0124] In S5, the rendering service module is based on the target rendering time. Submit the frame rendering output to the rendering buffer exchange mechanism and obtain the synchronization fence information corresponding to the frame rendering output. Synchronize fence information Synchronization barriers are used to express the execution progress of the GPU and rendering pipeline, facilitating the determination of whether a frame is complete and when related resources can be safely reused. For different graphics backends, synchronization barriers can be implemented by synchronization objects, event handles, or timestamps returned by the graphics interface. This application does not limit the specific implementation of synchronization barriers.

[0125] In the baseline scenario, the rendering service module is At millisecond intervals, the rendering output of the frame with targetId=1 is submitted to the corresponding presentation buffer exchange mechanism of the central control screen through the Vulkan graphics backend, and the corresponding synchronization fence object for this frame is obtained; the submission of targetId=2 and the process of obtaining the synchronization fence are carried out in parallel, each mounted in the corresponding presentation context, without blocking each other.

[0126] In some embodiments, for synchronization fences The status query employs a throttling strategy, at each target presentation time. Nearby objects are sparsely queried to avoid the overhead of high-frequency polling. In the baseline scenario, the rendering service module only performs rendering at the next target rendering time. Query within the 2-millisecond window before arrival The fence status is completed, and no queries are performed at other times, so that the query frequency of the fence status is tied to the vicinity of the critical moment.

[0127] In some embodiments, the synchronized fence objects employ a pooling reuse mechanism to avoid the memory and system call overhead associated with allocating and releasing fence objects every frame. In the baseline scenario, the rendering service module maintains a fence object pool with a capacity of 4 for each targetId. When a frame is submitted, a free object is retrieved from the pool as the frame's... Once the completed state is confirmed, it is returned to the pool, allowing the lifecycle of the fence object to be reused.

[0128] S6. Based on the back pressure index fed back by the aforementioned buffer exchange mechanism. and the synchronization fence information Update the back pressure state, and in the back pressure index Greater than or equal to the preset back pressure threshold A backpressure suppression strategy is executed, which includes at least one of the following: waiting until the next target presentation time. Discard the current frame, reduce the internal rendering resolution, or pause frame rendering output submission for the targetId.

[0129] In S6, the rendering service module uses the backpressure metrics fed back by the rendering buffer swapping mechanism. and synchronized fence information Update the back pressure status and in the back pressure index Greater than or equal to the preset back pressure threshold The back pressure suppression strategy is executed at the specified time, and the back pressure suppression strategy includes at least one of the following: waiting until the next target presentation time. Discard the current frame, reduce the internal rendering resolution, or pause frame rendering output submission for targetId.

[0130] Back pressure index This is used to quantify the pressure caused by insufficient throughput at the display end. The specific definition can vary depending on the platform implementation, and its underlying implementation can be a mapping of any one or a combination of the following: queue depth or number of unconsumed frames; number of blocking attempts or blocking duration to acquire a writable buffer; rendering latency estimation; synchronization barrier. Statistics on completion delay.

[0131] In the baseline scenario, taking a rendering loop with targetId=2 as an example. Assume the depth of the frame queue in the rendering buffer exchange mechanism connected to the passenger screen is 4, meaning the number of unconsumed frames is 4. Set to 4; preset back pressure threshold It is 3. Because The rendering service module enters a suppressed state for targetId=2, executing "wait until the next target rendering time". The strategy is to not submit new frames in this round until the next target rendering time arrives and a re-evaluation is performed. After one or two refresh cycles, the consumption rate on the display side... The rendering service module returns to 2, exiting the suppressed state. Simultaneously, targetId=1... The value is always 2, unaffected by the suppression state of targetId=2. The rendering rhythm of the central control screen remains stable, demonstrating that the back pressure suppression based on targetId ensures that the back pressure of one rendering target will not spread infinitely and affect other rendering targets.

[0132] In some embodiments, back pressure index The latency is calculated by fusing at least two weighted components from the following: frame queue depth, synchronization fence incomplete delay, number of blocking attempts or duration of blocking when acquiring the writable buffer, and end-to-end latency submitted to rendering; or by taking the maximum value among multiple components. In the baseline scenario, the rendering service module fusing the frame queue depth component, fence incomplete delay component, writable buffer blocking duration component, and end-to-end latency component with weights of 0.4, 0.3, 0.2, and 0.1 respectively. ,make It reflects four types of signals: queuing depth, fence completion timing, buffer request blocking, and end-to-end delay, reducing misjudgments caused by the failure of a single signal.

[0133] In some embodiments, the back pressure suppression strategy employs a hysteresis band mode, i.e., in the back pressure index... Greater than or equal to the preset upper threshold It enters a suppressed state at the time, under the back pressure index Less than or equal to the preset lower threshold And synchronous fence Exit the inhibited state when the completion condition is met, and set an upper threshold. With lower threshold A hysteresis interval of inactivity is formed between these points. In the baseline scenario, the rendering service module will... Set to 3. Set to 2, when When the value rises to a value greater than or equal to 3, it enters the inhibition state. Fall back to less than or equal to 2 and the most recent frame The inhibition state is exited only when the inhibition is completed, so that the inhibition behavior remains stable near the boundary and avoids premature exit when the fence is not completed, which would cause re-congestion.

[0134] In some embodiments, the backpressure suppression strategy is cascaded in the following order: waiting, reducing the internal rendering resolution, discarding the current frame, and pausing the frame rendering output submission for targetId. The previous stage is determined by the duration or backpressure index. Once the growth rate exceeds the preset threshold, the system switches to the next level; and the back pressure indicator... The fallback occurs in reverse order, level by level. In the baseline scenario, the rendering service module sets a maximum of 4 refresh cycles per level for targetId=2. The growth rate threshold is 0.5 per refresh cycle. After the waiting period lasts for 4 refresh cycles... The resolution has not yet dropped; the upgrade will reduce the internal rendering resolution. Continue to escalate, continuing to upgrade until the current frame level is discarded, until the targetId submission level is paused; When the congestion subsides, it should be done in reverse order, step by step, to avoid excessive suppression under mild congestion and to prevent re-congestion caused by releasing the congestion all at once.

[0135] In some embodiments, the selection of the back pressure suppression strategy is determined by a parameterized decision tree or a parameterized strategy table, the input of which includes a back pressure index. Each component and the synchronization fence The completion delay is calculated, and the output is the type and parameters of the suppression strategy. In the baseline scenario, the rendering service module loads a set of strategy tables optimized for the smart cockpit scenario. When the queue depth component is greater than 3 and the fence completion delay component is greater than 8 milliseconds, the policy table outputs the "reduce internal rendering resolution" action; when the queue depth component is greater than 3 but the fence completion delay component is less than 8 milliseconds, the policy table outputs the "wait" action. The selection of the suppression policy is externalized as an adjustable parameter, which can be customized by adjusting the decision tree or policy table under different hardware and scenarios.

[0136] In some embodiments, based on back pressure index The rate of change in the recent period predicts the back pressure index. Reaching the preset back pressure threshold When the predicted arrival time is earlier than the sum of the current time and the preset prediction window, the system enters the backpressure suppression state ahead of schedule. In the baseline scenario, the rendering service module uses the data from the last 8 refresh cycles... The slope of the linear regression is used as the rate of change when estimating... It will reach [the desired level] within 2 refresh cycles. When switching to the suppressed state, the system switches to the suppressed state one refresh cycle in advance. This method changes the entry into the suppressed state from a delayed response to a predictive response, reducing the probability of frame drops at peak times.

[0137] In some embodiments, when multiple targetIds share a rendering buffer exchange mechanism or related system resources, the backpressure suppression strategy is coordinated across multiple targetIds. Budgets are prioritized for critical targetIds, and the submission frequency is proactively reduced for non-critical targetIds. In the baseline scenario, the rendering service module marks the central control screen with targetId=1 as a critical target. When overall system resources are strained, the submission frequency of targetId=1 is prioritized, while the submission frequency or resolution of the secondary screen with targetId=2 is proactively reduced. This ensures that the rendering quality of critical targets is prioritized when shared resources are scarce, preventing chain-like backpressure propagation.

[0138] In some embodiments, in response to a synchronization fence Completed and back pressure indicators Falling back to the preset back pressure threshold The module then exits the backpressure suppression state. In the baseline scenario, the rendering service module continuously polls during the suppression state. The completion status, when targetId=2 in the most recent frame. Completed and current When the temperature drops back to 2, the state switches from the suppressed state back to the normal state, so that the exit time is driven by the fence and the back pressure drop together rather than a single time window.

[0139] In some embodiments, a synchronization fence The completion time is used as feedback for the estimated lead time between rendering and submission, and the estimated lead time is adjusted based on the difference between the fence completion time and the submission time. In the baseline scenario, the rendering service module uses the median difference between the fence completion time and the submission time of the most recent 16 frames as the baseline. The feedback correction item makes It converges adaptively as the actual path delay changes.

[0140] In S7, the rendering service module runs through S3 to S6. When it receives a control message related to targetId, it performs a state transition operation on the rendering context according to the target state. The control message is transmitted through cross-process communication or cross-thread communication.

[0141] Specifically, the control messages in S7 include at least one of the following: render target binding message, size change message, render target unbinding message, pause message, resume message, render backend switching message, scene switching message, or input event message; the state transition operations performed on the rendering context include at least one of the following: create, rebuild, destroy, pause, or resume operations.

[0142] Regarding the binding process, after the display module creates the graphics-carrying view and obtains the endpoint handle information, it sends a binding message to the rendering service module. Upon receiving the message, the rendering service module executes the following sequentially: creating or updating the rendering context corresponding to targetId; creating a window surface, swap chain, or equivalent rendering object based on the endpoint handle information; initializing the refresh cycle and reference alignment time; and entering the bound state and starting to participate in the rendering loop. In the baseline scenario, the binding process for the central control screen corresponding to targetId=1 is completed when the system starts, and the binding process for the passenger screen corresponding to targetId=2 is completed when the passenger screen is enabled. The rendering contexts of the two screens are initialized independently.

[0143] Regarding the size change and reconstruction process, when the display module detects a change in the size of the local rendering target, it sends a size change message. The rendering service module performs one of the following operations based on the target state: only updates the size of the viewport or renderable objects; or destroys and rebuilds the swap chain, window surfaces, and resets the reference alignment time and related resources to ensure that the rendering output is consistent with the display target size.

[0144] Regarding the unbinding and destruction process, when the display module detects that the local rendering target has been destroyed or unbound, it sends an unbinding message. The rendering service module enters the destruction state and executes the following steps in sequence: stop the submission of the targetId; wait for the necessary synchronization barriers to complete, or adopt a safe resource reclamation strategy; destroy the swap chain, window surface, or equivalent rendering object; reclaim the rendering context resources and enter the unbound state.

[0145] Regarding the pause and resume process, when the display module enters the background or needs to pause rendering, it sends a pause message. The rendering service module sets the corresponding targetId to a paused state and stops submitting. Upon resumption, the display module sends a resume message, and the rendering service module re-enables the beat alignment calculation and submission for that targetId.

[0146] In some embodiments, the rendering service module performs resource budget allocation, message sorting, and exception isolation at the targetId granularity. Resource budget allocation refers to the rendering service module allocating independent rendering time and video memory budgets for each targetId, limiting the resource consumption of a single targetId to its budget limit and preventing a single targetId from excessively occupying shared resources. Message sorting refers to the rendering service module sorting multiple control messages received by the same targetId according to the priority or timestamp causality of destruction, unbinding, reconstruction, size change, pause, resumption, and input events before processing them. Exception isolation refers to the rendering service module performing partial recycling and restart when a single targetId fails to rebuild, experiences a fence timeout, or a rendering exception, without affecting the rendering process of other targetIds. In the baseline scenario, the rendering service module allocates a rendering time budget of 12 milliseconds per unit refresh cycle and a video memory budget of 256 megabytes to the central control screen with targetId=1, and allocates a rendering time budget of 8 milliseconds per unit refresh cycle and a video memory budget of 128 megabytes to the passenger screen with targetId=2. When targetId=2 triggers an exception due to the failure of the swap chain reconstruction, the rendering service module only performs partial recycling and re-establishment of the rendering context of targetId=2, and the rendering process of the central control screen with targetId=1 is not affected, demonstrating the exception isolation capability at the targetId granularity.

[0147] S7 also includes coordinate mapping operations. Specifically, the touch point coordinates in the input event are mapped from the display coordinate system to the rendering coordinate system through the display module. The mapping includes: determining the viewport size of the rendered content relative to the local rendering target according to the principle of proportional conformal preservation, centering the viewport within the local rendering target, and normalizing the position of the touch point coordinates relative to the viewport into normalized coordinates in the rendering coordinate system; the normalized coordinates are used to restore the corresponding rendering space coordinates or interaction rays in the rendering service module.

[0148] Specifically, let the coordinates of the touch point in the display coordinate system be... The size of the target displayed locally is The base size of the rendered content is The normalized coordinates in the rendering coordinate system are Scaling factor Viewport width Viewport height Viewport top left corner offset satisfy , ;and , .in, To display the coordinates of the touch point in the coordinate system, , These represent the width and height of the target to be displayed locally. , These are the width and height, respectively, representing the baseline width and height of the rendered content. The rendering service module is based on... Restore rendering space coordinates or interaction rays to achieve consistency in multi-window and multi-screen interactions.

[0149] In the baseline scenario, taking the passenger screen with targetId=2 as an example. Assuming the actual display size of the locally rendered target is 1280×720, and the baseline size of the rendered content is also 1280×720, then the scaling factor... Viewport size , Viewport top left corner offset , When the user touches the screen on the passenger side... That is, when the screen is centered, the normalized coordinates This corresponds to the center position of the rendering coordinate system. When the actual display size of the locally rendered target differs from the base size of the rendered content—for example, a sub-region is displayed as 1280×480, while the base size of the rendered content remains 1280×720—then... (Retain three decimal places) , Viewport offset , This ensures that the rendered content is placed proportionally and centered within the display sub-area, and the normalized mapping of the touch point coordinates maintains geometric consistency.

[0150] The multi-objective collaborative behavior of this method will be further explained through several branch scenarios.

[0151] The first branch scenario is a simplified runtime scenario with a single targetId. In this scenario, the display module creates only one graphics-carrying view, the rendering service module maintains only one rendering context, and optional mechanisms related to cross-targetId coordination are not activated. At this time, the target rendering time in S4... The calculations do not require consideration of phase misalignment. The backpressure suppression strategy in S6 is executed with a single target as the limited scope, and the cross-targetId coordination mechanism does not participate in the decision-making. This simplified scenario is suitable for single-screen devices or working modes where only one locally rendered target is enabled.

[0152] The second branch scenario is the dual-targetId collaborative operation scenario, which is also the baseline scenario throughout the entire text. In this scenario, the display module creates two graphical views, corresponding to the central control screen with targetId=1 and the passenger screen with targetId=2, respectively. The display module obtains the endpoint handle information of both views and sends the binding message and endpoint handle information only during creation. The rendering service module establishes rendering contexts for targetId=1 and targetId=2 respectively, and initializes their refresh cycles and reference alignment times. In the rendering loop, the rendering service module calculates the target rendering times for targetId=1 and targetId=2 respectively. And submit frames to the corresponding presentation buffer exchange mechanism to obtain their respective synchronization fence information. And based on their respective back pressure indicators With preset back pressure threshold Backpressure suppression is implemented. When the size of targetId=2 changes, the rendering service module only performs reconstruction or viewport update for targetId=2, without affecting the rendering process of targetId=1. The display module collects input events, maps the touch point coordinates, and sends them. The rendering service module then completes the interaction logic processing accordingly. When any graphic-carrying view is destroyed, the display module sends an unbinding message, and the rendering service module safely releases the corresponding rendering context resources according to the unbinding and destruction process.

[0153] The third scenario is a variable refresh rate switching scenario. In this scenario, the display device switches from 60Hz to 90Hz, with a refresh rate of... The refresh rate changed from approximately 16.67 milliseconds to approximately 11.11 milliseconds. The rendering service module identified the change in refresh rate by performing statistical estimation on the rendering completion time series of the rendering buffer swapping mechanism, and recorded the switching moment as the new reference alignment moment; subsequent target rendering moments... The calculation is based on a new The alignment with the reference is performed at the moment of switching. If there is a residual deviation between the actual grid phase and the new reference alignment moment at the instant of switching, the tolerance detection mechanism will continuously perform multiple rounds. The deviation between the actual refreshed grid and the preset tolerance is compared, triggering further fine-tuning at the reference alignment time, so that the grid drift converges within several refresh cycles.

[0154] The fourth scenario is the backpressure diffusion and suppression scenario. In this scenario, the backpressure index of the passenger screen with targetId=2 increases due to a surge in rendering load. Continued to rise. Predictive triggering mechanisms are based on recent time periods. Rate of change estimation It will reach [the desired level] within 2 refresh cycles. The system switches to the suppressed state one refresh cycle in advance; the cascading upgrade mechanism unfolds step by step in the order of waiting, reducing the internal rendering resolution, discarding the current frame, and pausing targetId submission. After the initial drop, the drop-off occurs in reverse order. During this process, the rendering service module marks the control panel with targetId=1 as a critical target. The cross-targetId coordination mechanism ensures that the submission rhythm of targetId=1 is prioritized when shared resources are scarce, and the suppression behavior of targetId=2 will not be transmitted to targetId=1, thus achieving localized backpressure processing.

[0155] In some embodiments, the rendering service module can run as an independent process, exchanging control messages and endpoint handle information with the display module through cross-process communication. In other embodiments, the rendering service module can run as an independent thread within the same process, exchanging messages with the display module through cross-thread communication. In still other embodiments, the rendering service module and the display module are deployed on different device nodes, transmitting control messages and endpoint handle information over the network; since endpoint handle information is only transmitted once during creation or reconstruction, and subsequently only control messages and interaction messages are transmitted without transmitting frame-by-frame pixel data, network bandwidth usage in cross-device deployment is locked to the control plane rather than the data plane. This application does not limit the deployment relationship between the rendering service module and the display module.

[0156] In the aforementioned cross-device deployment configuration, control messages, interaction messages, and structured endpoint description messages are all transmitted via cross-device network communication. The carrier medium for cross-device network communication can include at least one of Ethernet, automotive Ethernet, Wi-Fi, 5G cellular networks, or industrial fieldbus; this application embodiment does not limit the network carrier medium. Since the structured endpoint description message already contains handle version number, lifecycle flag, and integrity verification fields, the rendering service module can identify expired, forged, or unauthorized messages in network transmission based on these fields when receiving messages across devices. In case of anomalies, the corresponding message can be discarded or a retransmission requested, enabling endpoint handle management in the cross-device deployment configuration to have the same robustness as in the same-device deployment configuration.

[0157] In some embodiments, the endpoint handle information of the rendering buffer exchange mechanism may carry a version number field. The version number increments with each rendering target reconstruction. The rendering service module confirms whether the received endpoint handle information comes from the latest reconstruction by comparing the version number, avoiding the misuse of expired handles in rapid reconstruction scenarios. This method serves as an auxiliary verification means for the one-time transmission mechanism of endpoint handle information, improving the robustness of resource management in scenarios with high reconstruction frequency.

[0158] In some embodiments, the input events collected by the display module include, in addition to touch events, one or more of keyboard events, gesture events, gaze events, and custom interaction events; different event types follow their own processing paths in the display module or rendering service module and are transmitted to the rendering service module through a unified interaction message format.

[0159] Based on the above-described cross-platform multi-target rendering method, this application also provides a cross-platform multi-target rendering device. (Refer to...) Figure 3 The device comprises six components: an encapsulation module, a rendering service module, a context module, a timing module, a backpressure module, and a management module. The encapsulation module encapsulates the 3D rendering kernel into a rendering kernel package and provides a unified rendering control interface and presentation interface, corresponding to step S1 in the method embodiment. The rendering service module runs on the target operating system and is built based on the rendering kernel package. It is used for initializing the 3D rendering kernel, managing its lifecycle, and scheduling rendering threads, corresponding to step S2 in the method embodiment. The context module assigns a target identifier (targetId) to at least one local rendering target and establishes a corresponding rendering context for each targetId. Each rendering context is associated with a rendering buffer exchange mechanism, corresponding to step S3 in the method embodiment. The timing module calculates the target rendering time aligned with the refresh cycle for each targetId during the rendering loop and submits the frame rendering output based on the target rendering time, corresponding to steps S4 and S5 in the method embodiment. The backpressure module is used to acquire the synchronization fence information and backpressure index corresponding to the frame rendering output, and executes a backpressure suppression strategy when the backpressure index is greater than or equal to a preset backpressure threshold. The backpressure suppression strategy includes at least one of the following: waiting until the next target rendering time, discarding the current frame, reducing the internal rendering resolution, or pausing the frame rendering output submission for targetId, corresponding to steps S5 and S6 in the method embodiment. The management module is used to perform creation, reconstruction, destruction, pause, or resumption operations on the rendering context according to the target state when it receives binding, size change, unbinding, pause, or resumption control messages related to targetId, corresponding to step S7 in the method embodiment. Each module establishes a collaborative relationship within the device through control messages and synchronization fences, so that the sequence of steps S1 to S7 in the method embodiment obtains a one-to-one corresponding carrier component on the device side.

[0160] In some embodiments, the context module is used to abstract the local rendering target into a rendering target description that includes a platform identifier, platform handle, handle version number, number of renderable buffers, pixel format, color space, width, height, and at least one rendering prompt parameter. In the baseline scenario, the rendering target description of the central control screen includes the platform identifier Android, the platform handle being the corresponding graphics-carrying view handle, handle version number 1, number of renderable buffers 3, pixel format RGBA8888, color space sRGB, width 1920, height 1080, and a variable refresh rate support flag in the rendering prompt parameters.

[0161] In some embodiments, the management module is further configured to allocate rendering time budgets and video memory budgets among multiple targetIds, and to perform isolation recovery for reconstruction failures, fence timeouts, or rendering anomalies of a single targetId. In the baseline scenario, the management module allocates a rendering time budget of 12 milliseconds per refresh cycle and a video memory budget of 256 megabytes to the central control screen with targetId=1, and allocates a rendering time budget of 8 milliseconds per refresh cycle and a video memory budget of 128 megabytes to the passenger screen with targetId=2. When a fence timeout occurs for targetId=2, the management module performs partial reclamation and reconstruction of the rendering context for targetId=2, while the rendering process of the central control screen with targetId=1 remains unaffected. This demonstrates the device-level localized handling capability for targetId-level anomalies, forming a one-to-one correspondence with the targetId-level resource management in the method embodiment.

[0162] In the baseline scenario, the central control screen's presentation target description includes the platform identifier "Android," the platform handle (corresponding to the graphics view handle), a width of 1920, a height of 1080, and presentation prompt parameters such as color space, number of renderable buffers, and variable refresh rate support flags. The rendering service module selects the graphics backend and creates the window surface or swap chain based on the presentation target description; the beat module writes the frame rendering output to the renderable buffer corresponding to the rendering buffer swap mechanism during the rendering loop. This approach encapsulates the access information of the local presentation target with a unified description structure, allowing the rendering service module to select the graphics backend and create the window surface based on this description, reducing the complexity of access across multiple platforms.

[0163] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0164] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A cross-platform multi-target rendering method based on a rendering buffer exchange mechanism, characterized in that, include: S1. Encapsulate the 3D rendering kernel into a rendering kernel package and provide a unified rendering control interface and presentation interface to the outside world; S2. Based on the rendering kernel package, a rendering service module is encapsulated on the target operating system. The rendering service module performs initialization and lifecycle management of the 3D rendering kernel, and starts the rendering thread through the rendering service module. The rendering thread performs scheduling of the 3D rendering kernel. S3. Assign a target identifier targetId to at least one local rendering target, and establish a corresponding rendering context for each targetId, wherein each rendering context is associated with a rendering buffer exchange mechanism; Each rendering context includes at least: endpoint handle information of the rendering buffer exchange mechanism, and refresh cycle. Reference alignment time, estimated lead time for rendering and submission, target status, and back pressure status; S4. In the rendering loop, for each targetId, calculate the refresh cycle based on the current time, the estimated rendering and submission lead, and the reference alignment time. Aligned target rendering time The target presentation time The refresh cycle is no earlier than the sum of the estimated lead time for the current time and the estimated lead time for rendering and submission, and is located at the reference alignment time. The most recent moment that is an integer multiple of the total time. S5. Based on the target presentation time Submit the frame rendering output to the rendering buffer exchange mechanism and obtain the synchronization fence information corresponding to the frame rendering output. ; S6. Based on the back pressure index fed back by the aforementioned buffer exchange mechanism. and the synchronization fence information Update the back pressure state, and in the back pressure index Greater than or equal to the preset back pressure threshold A backpressure suppression strategy is executed, which includes at least one of the following: waiting until the next target presentation time. Discard the current frame, reduce the internal rendering resolution, or pause the frame rendering output submission for the targetId; S7. During the execution of S3 to S6, in response to receiving a control message related to the targetId, a state transition operation is performed on the presentation context according to the target state; wherein the control message is transmitted through inter-process communication or inter-thread communication.

2. The cross-platform multi-target presentation method according to claim 1, characterized in that, S3 includes: In the display module, a graphics carrier view is created, and the rendering buffer exchange mechanism endpoint handle information corresponding to the local rendering target is obtained from the graphics carrier view; the graphics carrier view is a view object that can provide a local window handle, a graphics surface handle, or a renderable layer handle.

3. The cross-platform multi-target presentation method according to claim 2, characterized in that: The presentation buffer swapping mechanism endpoint handle information includes an endpoint handle identifier used to identify the same presentation buffer swapping mechanism across different processes or threads; The endpoint handle information of the rendering buffer exchange mechanism is sent only through inter-process communication or inter-thread communication when the local rendering target is created or rebuilt, and during subsequent frame rendering, only the control messages are sent through inter-process communication or inter-thread communication, without sending frame-by-frame pixel data; The rendering buffer swapping mechanism is one of the following: BufferQueue on the Android platform, swap chain on the Windows platform, shared graphics buffer sequence or graphics interface-based swap chain on the Linux platform, and renderable texture or renderable drawing object sequence on the iOS platform.

4. The cross-platform multi-target presentation method according to claim 2, characterized in that, S3 further includes: In the rendering service module, a graphics backend is selected or created based on the endpoint handle information of the rendering buffer exchange mechanism, and the rendering context is constructed based on the graphics backend, wherein the graphics backend is one of OpenGLES, Vulkan or Metal.

5. The cross-platform multi-target presentation method according to claim 1, characterized in that, In S7: The control messages include at least one of the following: rendering target binding message, size change message, rendering target unbinding message, pause message, resume message, rendering backend switching message, scene switching message, or input event message; The state transition operation performed on the presentation context includes at least one of the following: creation, reconstruction, destruction, pause, or resumption operations.

6. The cross-platform multi-target presentation method according to claim 1, characterized in that, S2 further includes: setting a message queue through the rendering service module to decouple the communication thread from the rendering thread; and in each rendering loop of the rendering thread, the rendering thread first retrieves the control message from the message queue and updates the rendering state, and then executes the rendering and presentation operations in S4 to S6.

7. The cross-platform multi-target presentation method according to claim 2, characterized in that, Also includes: The display module maps the touch coordinates in the input event from the display coordinate system to the rendering coordinate system. The mapping includes: determining the viewport size of the rendered content relative to the local rendering target according to the principle of proportional conformity preservation; centering the viewport within the local rendering target; and normalizing the position of the touch coordinates relative to the viewport to normalized coordinates in the rendering coordinate system. The normalized coordinates are used to restore the corresponding rendering space coordinates or interaction rays in the rendering service module.

8. The cross-platform multi-target presentation method according to claim 1, characterized in that, S1 includes: The 3D rendering kernel and its platform-related dependencies are packaged into a platform adaptation artifact, wherein the target operating system is any one of Android, iOS, Linux, HarmonyOS or Windows; the platform adaptation artifact is an Android archive (AAR) on the Android platform, and a library or framework artifact that matches the target operating system on non-Android platforms.

9. A cross-platform multi-target rendering device based on a rendering buffer exchange mechanism, characterized in that, include: The encapsulation module is used to encapsulate the 3D rendering kernel into a rendering kernel package and provide a unified rendering control interface and presentation interface to the outside world; The rendering service module runs on the target operating system and is built based on the rendering kernel package. The rendering service module is used to initialize the 3D rendering kernel, manage its lifecycle, and schedule rendering threads. The context module is used to assign a target identifier targetId to at least one local rendering target and to establish a corresponding rendering context for each targetId, wherein each rendering context is associated with a rendering buffer exchange mechanism; Each rendering context includes at least: endpoint handle information of the rendering buffer exchange mechanism, and refresh cycle. Reference alignment time, estimated lead time for rendering and submission, target status, and back pressure status; The beat module is used to calculate the refresh cycle alignment for each targetId during the rendering loop. Target presentation time And based on the target presentation time Submit the frame rendering output; The back pressure module is used to acquire the synchronization fence information corresponding to the frame rendering output. and back pressure indicators and in the back pressure index Greater than or equal to the preset back pressure threshold A backpressure suppression strategy is executed, which includes at least one of the following: waiting until the next target presentation time. Discard the current frame, reduce the internal rendering resolution, or pause the frame rendering output submission for the targetId; The management module is used to perform a state transition operation on the presentation context based on the target state when it receives a control message related to the targetId; wherein the control message is transmitted through cross-process communication or cross-thread communication.

10. The apparatus according to claim 9, characterized in that, The context module is used to abstract the local rendering target into a rendering target description that includes a platform identifier, platform handle, width, height, and at least one rendering prompt parameter; The rendering service module is used to select a graphics backend and create a window surface or swap chain based on the rendering target description; The beat module is used to write the frame rendering output into the renderable buffer corresponding to the rendering buffer exchange mechanism during the rendering loop.

Images