High-performance video and UI display method based on layered rendering architecture and storage medium

Abstract

The invention discloses a high-performance video and UI (User Interface) display method based on a layered rendering architecture, which comprises the following steps of: defining image display layers and a priority sequence thereof, sequentially rendering each layer according to a sequence from low priority to high priority, and ensuring that the high priority layer can correctly cover the low priority layer; and controlling the display sequence of each layer by adopting a double-track system that a local display sequence is determined according to a z-index value in a lamination priority and the same priority, and synthesizing a final screen display image. The technical scheme of the invention has the main advantages that real-time rendering of a high-quality video stream is supported while UI responsiveness is kept, and the requirement of a high-performance scene is met; it is ensured that the UI layer is always kept above the video layer, or the stacking sequence is dynamically adjusted according to application requirements; and a hierarchical design principle is adopted, and loose coupling is realized among the modules, so that independent upgrading and maintenance are facilitated, and the expandability of the system is enhanced.

Description

Technical Field

[0001] This invention belongs to the field of intelligent building technology, and in particular relates to a high-performance video and UI display method based on a layered rendering architecture for integrating intelligent building surveillance video. Background Technology

[0002] In smart building solutions, common approaches typically include UI virtual scene control and video surveillance, which are usually separate. However, existing technologies also exist that combine and overlay these two elements for mixed output. Common display drivers fall into two categories: fbdev and drm (Direct Rendering Manager), with application access methods corresponding to Framebuffer and libdrm, respectively. Common solutions are as follows: FrameBuffer approach: This approach uses the CPU (Central Processing Unit) or GPU (Graphics Processing Unit) to mix the video and UI, and then submits the composite buffer to the driver for display. This approach is relatively easy to implement, but for multi-screen, high-resolution displays, it consumes significant CPU or GPU resources, easily leading to insufficient refresh rates and causing stuttering, frame drops, and other issues.

[0003] GPU+DRM solution: This solution uses the GPU to mix video and UI, and then submits the composite buffer to the driver for display. The disadvantage of this solution is the same as the FrameBuffer solution: it consumes a lot of GPU performance and is prone to stuttering and frame drops.

[0004] DRM direct display solution: Video and UI are in the same process, with their respective buffers submitted and displayed directly by plane. The advantage of this solution is good overall performance. The disadvantage is a complex overall structure, making it difficult to separate UI and video, and it's also more likely to restrict integration with third-party apps. Open-source Weston + DRM solution: This solution uses the open-source Weston graphical desktop application to composite video and UI images, which are then submitted and displayed by DRM. The advantage of this solution is its ease of implementation; in single-video and single-UI scenarios, video can achieve normal performance requirements. The disadvantages are complex interface window management and DRM plane allocation not adhering to the video-first principle. In multi-layered video scenarios, it can easily encounter problems encountered with GPU compositing. Surface management for video or UI clients is also more complicated. Most importantly, the window hierarchy between video and UI cannot be guaranteed, and video can easily be displayed on top of the UI.

[0005] To address this, Chinese patent CN113050899B discloses a method and system for direct display of video and UI based on the Wayland protocol. However, the layered display scheme for video and UI based on the Wayland protocol has significant limitations in high-performance scenarios: 1. Issue of Video Layer Overlaying UI Layer: In the existing Wayland ecosystem, the video display layer easily overlays the UI layer. This is due to the compositor's inflexible mechanism for handling layer stacking order. Video playback typically uses a separate buffer, and when processing these buffers, the Wayland compositor by default places the video layer above the UI layer. In some application scenarios, this can cause UI elements to be obscured by the video, hindering the implementation of IoT-based control for smart buildings. 2. Performance Bottleneck: The Wayland compositor experiences significant performance bottlenecks when processing large volumes of video streams or complex UI interactions. This is primarily due to several key issues in its architecture design: frequent communication between the compositor and the client leads to high latency; a lack of targeted optimization for layered video and UI processing; and resource management strategies that fail to meet the real-time demands of high-performance scenarios. Summary of the Invention

[0006] The purpose of this invention is to solve the above-mentioned problems in the prior art and provide a high-performance video and UI display method based on a layered rendering architecture for intelligent building control.

[0007] The objective of this invention is achieved through the following technical solution: A high-performance video and UI display method based on a layered rendering architecture includes the following steps: S1. Define the image display layers and their priority order, including a background layer, a UI layer for displaying user interface elements, a video layer for displaying video content, and a system layer for displaying system-level elements; the UI layer has a higher priority than the video layer; each of the UI layer, video layer, and system layer is assigned a unique identifier and priority value, and the priority value can be adjusted. S2. Render each layer in order from low priority to high priority, ensuring that high priority layers can correctly cover low priority layers. S3. A dual-track system is used to control the display order of each layer, which is determined by the stacking priority order and the local display order based on the z-index value within the same priority, and the final screen display image is synthesized; where the higher priority layer will cover the lower priority layer.

[0008] Preferably, in step S2, only the content within the visible area is rendered to reduce unnecessary rendering operations; only the changed areas are re-rendered to avoid full-screen redrawing; and for static content, it is pre-composited and cached to reduce the overhead of real-time compositing.

[0009] Preferably, in step S2, the rendering operation can be submitted asynchronously, allowing the application to continue processing other tasks before rendering is complete; at the same time, the Fence object is used to achieve cross-thread and cross-process synchronization to ensure the correct order of operations.

[0010] Preferably, in step S3, a transparent area is set on the video layer by defining an alpha channel mask on the video layer. The area with an alpha value of 0 will be regarded as a transparent area, allowing the content of the UI layer to be displayed through the transparent area.

[0011] Preferably, in step S3, an improved edge collapse algorithm is adopted, which introduces a multi-factor cost function: Cost(e) = w1*geometric error + w2*curvature weight + w3*texture deviation + w4*boundary influence. The geometric error is calculated by using a quadric error metric to measure the volume change before and after edge collapse, and the curvature weight is calculated by the angle between the normal vectors of adjacent triangles to ensure that more details are preserved in high curvature areas.

[0012] Preferably, in step S3, texture streaming technology is used to dynamically load and unload textures based on view frustum analysis, while a machine learning-based texture prefetching model is used to predict the user's gaze point and texture requirements.

[0013] Preferably, in step S3, a block-based video memory allocation algorithm is used to optimize the physical storage of texture resources according to the MIPMAP level.

[0014] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the high-performance video and UI display method based on a layered rendering architecture as described above.

[0015] The advantages of the technical solution of this invention are mainly reflected in: 1. High-performance display: While maintaining UI responsiveness, it supports real-time rendering of high-quality video streams to meet the needs of high-performance scenarios. 2. Flexible layering control: Provides a fine-grained layering control mechanism to ensure that the UI layer always remains on top of the video layer, or to dynamically adjust the layering order according to application requirements. 3. Simplified resource management: Design efficient memory management and resource allocation strategies to reduce system overhead and improve resource utilization. 4. Modular architecture: Adopting a layered design principle, the modules are loosely coupled, which facilitates independent upgrades and maintenance and enhances the scalability of the system. Attached Figure Description

[0016] Figure 1 : A flowchart illustrating a preferred embodiment of the present invention; Figure 2 : A schematic diagram of an application scenario of a preferred embodiment of the present invention. Detailed Implementation

[0017] The objectives, advantages, and features of this invention will be illustrated and explained through the following non-limiting description of preferred embodiments. These embodiments are merely typical examples of applying the technical solutions of this invention, and all technical solutions formed by equivalent substitutions or equivalent transformations fall within the scope of protection claimed by this invention.

[0018] This invention discloses a high-performance video and UI display method based on a layered rendering architecture for intelligent building control. First, the system hardware architecture of this invention is introduced. It adopts a four-layer vertical structure to decouple the rendering logic from the hardware. From bottom to top, the layers are: hardware layer, driver layer, rendering layer, and application layer.

[0019] The top layer is the application layer, providing a declarative UI description language and video playback interface. It defines the interface structure and video playback area through a component tree, and the framework automatically generates a rendering instruction queue. Its core components include: UI framework: Provides a standard UI component library and layout system, and supports declarative UI description; Dirty area detection: Only marks UI areas that have changed, avoiding full-screen redraws and improving efficiency; Asynchronous task scheduling: Distribute layout calculation and rendering tasks to independent threads to reduce main thread blocking.

[0020] The second layer is the rendering layer, the core of the system, responsible for converting the application layer's UI descriptions and video data into hardware-executable rendering instructions. Its core components include: Scene graph management: Rendering nodes are organized in a tree structure, supporting layer merging and pruning optimization; Rendering instruction compiler: Converts scene graphs into rendering commands executable by the GPU; Resource Manager: Unifies the management of GPU resources such as textures, shaders, and vertex buffers, enabling automatic recycling and reuse; Synthesis Controller: Manages the synthesis order and method of each layer, enabling flexible layer stacking control; Scene graph management: A hybrid spatial index structure combining quadtrees and BVH is used to enable rapid clipping and merging of dynamic scenes. Each rendering node contains information such as position, size, opacity, and transformation matrix, facilitating the compositing controller's calculation of the stacking order. Resource management strategy: A block-based memory allocation algorithm is adopted, and texture resources are physically stored in an optimized manner according to the MIPMAP hierarchy. An intelligent cache eviction mechanism based on LRU-K is developed. At the same time, a unified management pool for resources such as textures, shaders, and vertex buffers is established to realize automatic resource reclamation and reuse. The third layer is the driver layer, which defines the hardware abstraction interface, supports seamless switching between multiple rendering backends, and achieves decoupling from specific hardware platforms. Its core components include: Unified Command Buffer: Design a unified command buffer converter for Vulkan / Metal / OpenGLES and establish a graphics API abstraction layer; Multi-rendering backend support: Supports multiple graphics APIs, including OpenGL ES 3.2, Vulkan 1.2, and a self-developed rendering engine; Hardware characteristics detection: Automatically detects hardware characteristics and selects the optimal rendering path; Resource allocation strategy: Optimize resource allocation based on hardware characteristics to improve resource utilization; Multiple rendering backend support: To cater to different hardware platforms and performance requirements, the system supports multiple rendering backends, such as OpenGL ES 3.2 and Vulkan 1.2; and provides extreme performance through deep optimization for specific GPU architectures.

[0021] The lowest layer is the hardware layer, which interfaces with the display controller through standardized interfaces to achieve advanced features such as vertical synchronization and dynamic refresh rate. Its core components include: Video memory management: Provides an efficient video memory allocation and reclamation mechanism; Rendering pipeline: Implements a complete graphics rendering pipeline, supporting stages such as vertex processing, rasterization, and fragment processing; Display output: Controls the output of the display device, supporting multiple monitors and different resolutions; Hardware acceleration engine: Provides hardware-accelerated functions such as compositing, scaling, and rotation; Dynamic refresh rate control: Supports dynamic adjustment of refresh rate (60Hz / 90Hz / 120Hz), automatically optimizing refresh rate based on content complexity and system load to reduce power consumption.

[0022] The core of this invention lies in the rendering layer. Through layered rendering, the UI and video content are rendered into separate layers, and then a compositor combines these layers into the final screen image. For example... Figure 1 As shown, the specific steps are as follows: S1. Define the image display layers and their priority order, including a background layer, a UI layer for displaying user interface elements, a video layer for displaying video content, and a system layer for displaying system-level elements; the UI layer has a higher priority than the video layer; each of the UI layer, video layer, and system layer is assigned a unique identifier and priority value, and the priority value can be adjusted. S2. Render each layer sequentially from low priority to high priority, ensuring that high-priority layers correctly cover low-priority layers. In this step, only the content within the visible area is rendered to reduce unnecessary rendering operations; only the changed areas are re-rendered to avoid full-screen redraws; for static content, it is pre-composited and cached to reduce the overhead of real-time compositing; rendering operations can be submitted asynchronously, allowing the application to continue processing other tasks before rendering is complete; at the same time, Fence objects are used to implement cross-thread and cross-process synchronization to ensure the correct order of operations. S3. A dual-track system is used to control the display order of each layer, which is determined by the stacking priority order and the local display order based on the z-index value within the same priority, and the final screen display image is synthesized; where the higher priority layer will cover the lower priority layer.

[0023] First, define several fixed priority levels: System layer (highest priority), including mouse pointers, system tools, dialog boxes, etc.; UI layer (medium priority), including the main interface, menus, and toolbars; Video layer (low priority); and Background layer (lowest priority). By default, the video layer has a lower priority than the UI layer to ensure that UI elements are not obscured by video content. Alternatively, you can adjust the layer priorities as needed, but the priority of the system layer is fixed and cannot be changed.

[0024] Step S3 also includes some other special operations. For example, by defining an alpha channel mask on the video layer, a transparent region is set on the video layer. Regions with an alpha value of 0 are considered transparent regions, allowing the content of the UI layer to be displayed through the transparent region. An improved edge collapse algorithm is adopted, introducing a multi-factor cost function: Cost(e) = w1*geometric error + w2*curvature weight + w3*texture deviation + w4*boundary influence. The geometric error uses a quadratic error metric to calculate the volume change before and after edge collapse, and the curvature weight is calculated using the angle between the normal vectors of adjacent triangles to ensure that more details are preserved in high curvature regions.

[0025] This invention employs texture streaming technology, dynamically loading and unloading textures based on view frustum analysis. It also utilizes a machine learning-based texture prefetching model to predict the user's gaze point and texture requirements. Furthermore, it employs a block-based memory allocation algorithm to optimize physical storage of texture resources according to the MIPMAP hierarchy.

[0026] This invention's texture streaming technology significantly reduces video memory usage by dynamically adjusting texture resolution and loading range, while avoiding bandwidth waste and stuttering issues caused by traditional static loading. Specifically, it employs a hierarchical bounding volume hierarchy (BVH) for rapid view frustum culling. First, the scene is divided into spatial cells using a 2m×2m×2m grid, with each cell associated with a texture asset list. A BVH tree is constructed for each cell, where parent nodes contain the joint bounding box and texture set of their child nodes. Two levels of bounding boxes are bound to each texture: object-level and texture patch-level. Next, the visible texture set and priority are output based on the view frustum to determine the minimum required MIP level. The basic MIP chain performs 2x2 or 4x4 slicing on high-resolution textures; a texture atlas is used to package small textures within the same region. The texture prefetching model uses a deep learning architecture of "spatiotemporal feature fusion + multi-task learning," employing spatial, temporal, and semantic branches for feature extraction. The output multi-tasks include: foveation prediction, texture requirement prediction, and temporal window prediction.

[0027] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the high-performance video and UI display method based on a layered rendering architecture as described above.

[0028] One application scenario of this invention is as follows: Figure 2 As shown, UI control switches for air conditioning and lighting are generated on the surveillance video. Operators can control these via mouse, and the system is linked to the actual air conditioning and lighting switches through IoT technology to achieve intelligent building control.

[0029] This invention has many other embodiments, and all technical solutions formed by equivalent transformation or equivalent transformation fall within the protection scope of this invention.

Claims

1. A high-performance video and UI display method based on a layered rendering architecture, characterized in that: Includes the following steps, S1. Define the image display layers and their priority order, including the UI layer for displaying user interface elements, the video layer for displaying video content, and the system layer for displaying system-level elements; the priority of the UI layer is set higher than that of the video layer; each of the UI layer, video layer, and system layer is assigned a unique identifier and priority value, and the priority value can be adjusted. S2. Render each layer in order from low priority to high priority, ensuring that high priority layers can correctly cover low priority layers. S3. A dual-track system is used to control the display order of each layer, which is determined by the stacking priority order and the local display order based on the z-index value within the same priority, and the final screen display image is synthesized; where the higher priority layer will cover the lower priority layer.

2. The high-performance video and UI display method based on a layered rendering architecture according to claim 1, characterized in that: It also includes a background layer.

3. The high-performance video and UI display method based on a layered rendering architecture according to claim 1, characterized in that: In step S2, only the content within the visible area is rendered to reduce unnecessary rendering operations; only the changed areas are re-rendered to avoid full-screen redrawing. For static content, it is pre-synthesized and cached.

4. The high-performance video and UI display method based on a layered rendering architecture according to claim 1, characterized in that: In step S2, the rendering operation can be submitted asynchronously, allowing the application to continue processing other tasks before rendering is complete; at the same time, the Fence object is used to achieve cross-thread and cross-process synchronization to ensure the correct order of operations.

5. The high-performance video and UI display method based on a layered rendering architecture according to claim 1, characterized in that: In step S3, a transparent area is set on the video layer by defining an alpha channel mask. Areas with an alpha value of 0 are considered transparent areas, allowing the content of the UI layer to be displayed through the transparent area.

6. The high-performance video and UI display method based on a layered rendering architecture according to claim 1, characterized in that: In step S3, an improved edge collapse algorithm is adopted, which introduces a multi-factor cost function: Cost(e) = w1*geometric error + w2*curvature weight + w3*texture deviation + w4*boundary influence. The geometric error is calculated by using a quadric error metric to calculate the volume change before and after edge collapse, and the curvature weight is calculated by the angle between the normal vectors of adjacent triangles to ensure that more details are preserved in high curvature areas.

7. The high-performance video and UI display method based on a layered rendering architecture according to claim 1, characterized in that: In step S3, texture streaming technology is used to dynamically load and unload textures based on view frustum analysis. At the same time, a texture prefetching model based on machine learning is used to predict the user's gaze point and texture requirements.

8. The high-performance video and UI display method based on a layered rendering architecture according to claim 1, characterized in that: In step S3, a block-based video memory allocation algorithm is used to optimize the physical storage of texture resources according to the MIPMAP level.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the high-performance video and UI display method based on a layered rendering architecture as described in any one of claims 1 to 9.

Images