An OpenHarmony hardware acceleration rendering method based on a native GBM

Abstract

This invention discloses a hardware-accelerated rendering method for OpenHarmony based on native GBM. Upon startup, the OpenHarmony system identifies the Mali GPU architecture and initializes memory and a zero-copy mechanism, establishing a resource management structure and hash table. The GBM module completes shader pre-compilation caching. The application side establishes an EGL rendering runtime environment and ensures multi-threaded safety. During the rendering phase, frame buffer resources are dynamically allocated, and shader scheduling is completed after memory verification, establishing a complete rendering chain to achieve direct data writing to shared memory. The display side maps shared memory to construct zero-copy layers, adapts scaling schemes according to the architecture, synchronizes timing to complete image output, and uniformly reclaims resources. The entire process achieves efficient and stable hardware-accelerated zero-copy rendering.

Description

Technical Field

[0001] This invention belongs to the field of computer software development technology, specifically relating to an OpenHarmony hardware-accelerated rendering method based on native GBM. Background Technology

[0002] On the ARM Mali GPU platform, if OpenHarmony needs to implement hardware-accelerated rendering of OpenGL ES, it relies on the Panfrost driver of the Mesa project. This driver needs to complete the allocation of frame buffers, format conversion and interaction with the kernel DRM (Direct Rendering Manager) subsystem through the buffer manager GBM (Generic BufferManager) module.

[0003] However, existing technical solutions mainly have the following problems:

[0004] First, there are compatibility and performance bottlenecks in the GBM adaptation method. Existing solutions generally adopt the approach of porting the complete Mesa source code along with its native GBM module. This leads to conflicts with the native ION memory management system of OpenHarmony. Mesa-GBM relies on the native memory management logic of the Linux system, requiring an intermediate adaptation layer to copy frame buffer data from the Mesa-GBM memory pool to the OpenHarmony ION memory pool. This adds 2 to 3 data copies to a single rendering process, increasing latency by more than 30%. The Mesa source code is over 50MB, and after porting, the OpenHarmony system image increases by 15% to 20%, which does not conform to its design positioning for lightweight devices. In addition, the version iteration and maintenance costs are high. Mesa releases a new version every quarter, and the ported code needs to be modified synchronously, which is prone to compatibility issues with the OpenHarmony kernel version.

[0005] Secondly, there is the issue of incompatibility with OpenHarmony graphics components. OpenHarmony lacks the SurfaceFlinger component for Android systems, and its graphics compositing relies on its self-developed DisplayComposer module. However, existing solutions do not have GBM interface logic designed for DisplayComposer. Instead, they require a conversion process from the GBM frame buffer to the RGB pixel stream and then to the DisplayComposer compositing buffer, resulting in the loss of native hardware information of the frame buffer, such as DRM format and physical address, leading to a 25% reduction in compositing efficiency. Furthermore, it is impossible to utilize the hardware compositing channel of DisplayComposer, forcing the use of software compositing, which further consumes CPU resources. Additionally, a zero-copy mechanism is not implemented, with the copying time of frame buffer data between GBM and DisplayComposer accounting for 40% of the total compositing time.

[0006] Third, there is a conflict in memory compatibility between Mali GPUs and lightweight devices. Mali GPUs such as the Midgard architecture T860, Bifrost architecture G52, and Valhall architecture G57, in lightweight devices with less than 512MB of memory, use a fixed-resolution allocation strategy for traditional GBM framebuffers. For example, a 1080P framebuffer occupies a fixed 4MB of memory, which easily leads to memory overflow. When multiple applications are rendering concurrently, the framebuffer occupies more than 40% of the total memory, triggering OpenHarmony's memory reclamation mechanism and causing rendering stutters. GPU resources are wasted; on low-resolution displays such as a 2.8-inch 480*320 screen loading a 1080P framebuffer, the GPU needs to perform additional scaling operations, increasing the time by 15ms per frame. Furthermore, Valhall architecture compatibility is lacking; existing solutions do not optimize framebuffer allocation strategies for the unified shader core of the Valhall architecture, resulting in an 18% reduction in rendering efficiency.

[0007] Fourthly, there are issues with Panfrost driver shader compilation latency and lack of error handling. Panfrost driver uses a real-time compilation mode for OpenGL ES shaders. On lightweight OpenHarmony devices with CPU frequencies below 1.2GHz, complex shaders, such as fragment shaders with multi-texture sampling and matrix transformations, take more than 80ms to compile, causing white screen stuttering when the application starts. There is no compilation result caching mechanism, and the application needs to be recompiled after restarting, which further degrades the user experience. The lack of a degradation strategy for frame buffer allocation failures and shader compilation errors can easily lead to application crashes.

[0008] In summary, existing technical solutions have significant shortcomings in four core areas: GBM adaptation, graphics component integration, memory resource adaptation, and driver compilation and error handling. These shortcomings not only prevent OpenGL ES hardware-accelerated rendering from meeting the performance and compatibility requirements of OpenHarmony lightweight devices, but also result in bloated system images, high maintenance costs, and insufficient application stability. These issues severely restrict the widespread adoption and implementation of OpenHarmony on ARM Mali GPU lightweight devices. Therefore, there is an urgent need to design and optimize solutions to overcome these technical bottlenecks and achieve efficient, stable, and lightweight OpenGL ES hardware-accelerated rendering. Summary of the Invention

[0009] In view of this, the present invention provides a hardware-accelerated rendering method for OpenHarmony based on native GBM. By initializing ION memory management, building a rendering environment adapted to Mali GPU, and combining frame buffer dynamic adaptation and hardware acceleration, hardware-accelerated rendering of OpenGL ES applications within OpenHarmony on the ARM Mali GPU platform is realized.

[0010] This invention provides an OpenHarmony hardware-accelerated rendering method based on native GBM, which specifically includes the following steps:

[0011] When the OpenHarmony system starts, it loads the Mali GPU driver and identifies its architecture and core parameters. It marks the unified shader core capabilities for the Valhall architecture, initializes the ION memory manager and zero-copy sharing mechanism, builds the first structure associated with DRMBO and Mali GPU to manage rendering resources, allocates cache memory according to the architecture type and records it into the structure, and establishes the first hash table to manage the shared ION framebuffer. At the same time, the GBM module completes shader pre-compilation, cache adaptation and persistent storage, and records it into the second hash table.

[0012] After the application starts, it initializes the system's default display device, filters compliant EGL configurations, creates and activates the EGL rendering context, associates and binds the GBM device with the rendering context, and enables the thread mutual exclusion mechanism.

[0013] After the application initiates an OpenGL ES rendering request, it requests and can dynamically scale the shared ION frame buffer as needed, and synchronously sends out relevant parameters; after completing the shared memory consistency check, the application passes in the shader source code and type to be used, and the GBM module completes the shader loading or real-time compilation and update according to the second hash table, and sequentially creates and binds the EGL image, OpenGL ES texture and FBO rendering link, and after verification, executes the rendering operation to write the data to the shared ION frame buffer, and after completing the GPU execution status synchronization, notifies the display compositing module to read and use it;

[0014] The display compositing module receives frame buffer-related parameters and obtains its own process identifier. It maps the shared ION frame buffer memory in read-only mode, encapsulates information to generate a zero-copy layer, and connects to the hardware compositing channel. It uses different scaling methods to adapt the screen size based on the differences in GPU architecture, binds display hardware resources and synchronizes the VSYNC signal, relies on the page flipping mechanism to complete the screen compositing output, and finally completes the release and recycling of resources throughout the entire process.

[0015] Furthermore, the method for requesting and dynamically scaling the shared ION frame buffer on demand, and synchronously sending relevant parameters, is as follows:

[0016] Request a frame buffer based on the initial display resolution. The system allocates shared ION frame buffer memory and obtains the physical address, creates a GBM frame buffer object, generates a shared ION frame buffer handle, and sends it to the display composition module. When the application window size changes, the screen rotates, the resolution is switched, or the system memory is insufficient, dynamic frame buffer scaling is triggered to determine the new frame buffer size and the effective memory access range. The shared ION frame buffer handle is updated and regenerated, and then sent to the display composition module again.

[0017] Furthermore, the dynamic frame buffer scaling method is as follows: detect window changes, screen rotation or resolution switching events, release the original frame buffer memory and management object, recalculate and allocate shared ION frame buffer memory according to the new resolution, rebuild the GBM frame buffer object and update the hash table record, and synchronously send new memory handles and parameters to the display compositing module.

[0018] Furthermore, the application inputs the shader source code and type to be used. The GBM module completes the shader loading or real-time compilation and update according to the second hash table in the following way: The GBM module preprocesses the shader source code according to the pre-compilation rules and generates a query key. It searches for matching items in the second hash table. If a match is found, the shader binary instructions are loaded from the shader shared ION memory. If no match is found, the shader source code is compiled in real time to complete the loading, and the generated binary instructions are written to the shader shared ION memory to update the second hash table.

[0019] Furthermore, the method for preprocessing according to the pre-compilation rules and generating query keys is as follows:

[0020] For the current Mali GPU architecture, we uniformly replaced architecture-related macro definitions and platform adaptation identifiers in the source code, and shielded incompatible architecture branch logic to make shader syntax and macro definitions compatible with the native GPU hardware architecture; we uniformly cleaned up redundant spaces, invalid comments, and redundant line breaks, aligned code formatting, and eliminated differences in writing format; we uniformly expanded conditional compilation branches to output standardized versions of shader source code; and based on the preprocessed standardized source code, we used a fixed hash algorithm to generate a unique hash fingerprint as the query key.

[0021] Furthermore, the method of adapting screen size by using different scaling methods based on GPU architecture differences is as follows:

[0022] When the current architecture is Valhall and the screen resolution does not match the display resolution, the display plane hardware scaling engine is invoked to complete lossless hardware scaling; if other GPU architectures are used, software scaling is used to adapt the screen size.

[0023] Furthermore, when mapping the shared ION frame buffer memory to the display compositing module process address space with read-only permissions, the minimum required memory capacity is calculated based on the frame buffer resolution and pixel format. The calculated value is compared with the actual mapped memory size. If the required capacity is not greater than the actual mapped memory capacity, the memory mapping is completed normally; otherwise, it is determined that the memory capacity is insufficient, and it is downgraded to data copy mode and an exception log is recorded.

[0024] Furthermore, the minimum memory capacity is calculated as follows: based on the width, height, and pixel format of the frame buffer, the corresponding number of bits per pixel is obtained, and the minimum memory capacity is calculated using the formula: minimum memory capacity = width * height * number of bits per pixel / 8.

[0025] Furthermore, the method for initializing the ION memory manager and zero-copy sharing mechanism is as follows: the ION memory allocator is initialized when the system starts, and the physical contiguous memory pool and cross-process sharing capability are configured; a direct physical address mapping path between the Mali GPU and ION memory is established, and the GPU cache consistency synchronization strategy is enabled.

[0026] Furthermore, the core fields of the first structure include: basic framebuffer attributes, including the width, height, and pixel format of the framebuffer; GPU architecture identifier, used to distinguish between the three architectures: Midgard, Bifrost, and Valhall; hardware parameters, including L2 cache size and the maximum rendering resolution supported by the GPU; architecture feature markers, used to identify the rendering capabilities specific to each architecture; memory-related parameters, including ION memory handles, cross-process shared ION handles, and physical memory addresses; function switches, including framebuffer slicing support status and zero-copy enabled status; and cache attributes, which are memory cache modes adapted to different architecture cache consistency capabilities.

[0027] Beneficial effects:

[0028] This invention loads the Mali GPU driver and identifies the GPU architecture when the OpenHarmony system starts. For Valhall architecture, it marks the unified shader core capabilities, initializes the ION memory manager and zero-copy sharing mechanism, and constructs a first structure and a first hash table to manage framebuffer-related resources. Simultaneously, the GBM module completes shader pre-compilation and cache persistence. After application startup, it initializes the display device and creates an EGL rendering context bound to the GBM device, establishing a zero-copy rendering environment. When the application initiates an OpenGLES rendering request, it requests and dynamically scales the shared ION framebuffer, completes memory consistency checks and shader loading, and performs zero-copy rendering through EGL image, texture, and FBO binding. After rendering is complete, it synchronizes the state and notifies the display compositing module. The display compositing module maps the shared ION framebuffer memory, adapts the scaling strategy according to the GPU architecture, binds hardware resources, and synchronizes the VSYNC signal to complete the on-screen display. Ultimately, it achieves hardware-accelerated OpenGL ES rendering for applications within OpenHarmony on the ARM Mali GPU platform, improving overall rendering efficiency and display effects. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating an OpenHarmony hardware-accelerated rendering method based on native GBM provided by the present invention. Detailed Implementation

[0030] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0031] This invention provides a hardware-accelerated rendering method for OpenHarmony based on native GBM. The core idea is as follows: When the OpenHarmony system starts, it loads the Mali GPU driver and identifies the GPU architecture. For Valhall architecture, it marks the unified shader core capabilities, initializes the ION memory manager and zero-copy sharing mechanism, and constructs a first structure and a first hash table to manage framebuffer-related resources. Simultaneously, the GBM module completes shader pre-compilation and cache persistence. After the application starts, it initializes the display device and creates an EGL rendering context bound to the GBM device, building a zero-copy rendering environment. After the application initiates an OpenGLES rendering request, it requests and dynamically scales the shared ION framebuffer, completes memory consistency checks and shader loading, and performs zero-copy rendering through EGL image, texture, and FBO binding. After rendering is complete, it synchronizes the state and notifies the display compositing module. The display compositing module maps the shared ION framebuffer memory, adapts the scaling strategy according to the GPU architecture, binds hardware resources, and synchronizes the VSYNC signal to complete the on-screen display. Ultimately, it achieves hardware-accelerated OpenGL ES rendering for applications within OpenHarmony on the ARM Mali GPU platform, improving overall rendering efficiency and display effects.

[0032] This invention provides an OpenHarmony hardware-accelerated rendering method based on native GBM, the processing flow of which is as follows: Figure 1 As shown, the specific steps include:

[0033] Step 1: The OpenHarmony system starts and loads the Mali GPU driver, reads the Mali GPU-specific registers, including the architecture identifier register and cache configuration register, identifies the current GPU architecture, cache size, and other core parameters. If the current GPU architecture is Valhall, it reads the feature control register and marks its unified shader core capabilities. It initializes the ION memory manager and zero-copy sharing mechanism, constructs an association structure between the DRM BO and the entire Mali GPU architecture, denoted as the first structure, to store the GPU architecture, memory handle, physical address, cache attributes, and zero-copy state of the application framebuffer, realizing unified management of architecture, memory, and rendering resources. When the GPU architecture is Midgard, it allocates exclusive cache memory; when it is Bifrost, it allocates shared cache memory; and when it is Valhall, it allocates cache memory with a shared tag. It obtains the physical address of the allocated cache memory and stores it in the first structure. It establishes the first hash table, using the shared ION framebuffer handle and the process ID of the display compositing module as keys, and the physical address, memory size, and data format of the shared ION framebuffer as values.

[0034] The GBM module parses the shader pre-compilation configuration file and selects shaders that are compatible with the current GPU architecture; it reads and preprocesses the shader source code, completes the pre-compilation through the underlying driver, and generates shader binary instructions that are compatible with the current architecture; using the shader source code hash value and GPU architecture identifier as cache indexes, it stores the shader binary data in the shader shared ION memory and records it in the second hash table, thus realizing shader pre-compilation and cache persistence.

[0035] Furthermore, if the input resolution exceeds the current GPU architecture's supported limit, it will be downgraded to the architecture's maximum renderable resolution, and the adjustment information will be recorded.

[0036] Among them, DRM BO (Direct Rendering Manager Buffer Object) is the core abstraction in the Linux kernel DRM subsystem used to manage GPU memory. It represents a graphics buffer that can be shared and accessed by both the CPU and GPU. It is widely used for the storage and management of graphics resources such as frame buffers, textures, and render targets.

[0037] The core fields of the first structure include: basic framebuffer attributes, including basic information such as width, height, and pixel format; GPU architecture identifier, used to distinguish between the three architectures: Midgard, Bifrost, and Valhall; hardware parameters, including L2 cache size and the maximum rendering resolution supported by the GPU; architecture characteristic markers, used to identify the rendering capabilities specific to each architecture; memory-related parameters, including ION memory handles, cross-process shared ION handles, and physical memory addresses; function switches, including framebuffer slicing support status and zero-copy enabled status; and cache attributes, including memory cache mode, adapting to different architecture cache consistency capabilities.

[0038] The initialization of the ION memory manager and zero-copy sharing mechanism is as follows: the ION memory allocator is initialized when the system starts, and the physical contiguous memory pool and cross-process sharing capabilities are configured; a direct physical address mapping path between the Mali GPU and ION memory is established, and the GPU cache consistency synchronization strategy is enabled.

[0039] The process of reading and preprocessing shader source code involves the following steps: First, the original shader source code file is read from a preset system path and loaded into a source code buffer. The maximum read capacity is limited to prevent resource overflow. Second, for the current Mali GPU architecture, architecture-related macro definitions and platform adaptation identifiers within the source code are uniformly replaced, and incompatible architecture branch logic is hidden, ensuring that shader syntax and macro definitions are adapted to the native GPU hardware architecture. Third, redundant spaces, invalid comments, and unnecessary line breaks are uniformly cleaned up, code formatting is aligned, and differences in writing style are eliminated. Fourth, conditional compilation branches are uniformly expanded to ensure that shaders with the same functionality but different syntax ultimately have the same source code form. Fifth, after architecture adaptation and format standardization, a unified standardized version of the shader source code is output. Sixth, based on the preprocessed standardized source code, a unique hash fingerprint is generated using a fixed hash algorithm. This fingerprint is then used to concatenate GPU architecture identifiers and build a shader cache retrieval index, achieving accurate cache matching.

[0040] Step 2: After the application starts, obtain the system's default display device and complete the initialization. Filter the EGL configuration that supports both zero-copy capability and standard display parameters. Create an EGL rendering context based on the selected EGL configuration and activate it as the currently globally available state. Associate and bind the GBM device with the EGL rendering context so that the rendering environment can recognize and call the shared ION frame buffer managed by the GBM. Save the display device, EGL rendering context, and EGL configuration as a unified global rendering environment and initialize a thread mutual exclusion mechanism to ensure the safety of multi-threaded reuse.

[0041] EGL is a standard low-level interface between the OpenGL ES graphics interface and the system display device. It is used to manage display devices, configure rendering parameters, create rendering contexts, associate frame buffers and display canvases, and provide the basic operating environment for OpenGL ES hardware-accelerated rendering.

[0042] Step 3: The application initiates an OpenGL ES rendering request, requests a frame buffer according to the initial display resolution, the system allocates shared ION frame buffer memory and obtains the physical address, creates a GBM frame buffer object, generates a shared ION frame buffer handle and sends it to the display composition module; when the application window size changes, the screen rotates, the resolution is switched or the system memory is insufficient, dynamic frame buffer scaling is triggered, the new frame buffer size and effective memory access range are determined, the shared ION frame buffer handle is updated and regenerated, and sent to the display composition module again;

[0043] The system performs ION memory consistency checks to verify the physical address, size, and access permissions of shared memory, ensuring zero-copy security and availability. After the check passes, the application passes in the shader source code and type to be used. The GBM module preprocesses the shader source code according to the pre-compilation rules and generates a lookup key. It then searches for a matching item in the second hash table. If a match is found, the shader binary instructions are loaded from the shader's shared ION memory. If no match is found, the shader source code is compiled in real time to complete the loading, and the generated binary instructions are written to the shader's shared ION memory, updating the second hash table.

[0044] An EGL image is created based on the GBM framebuffer object and its associated shared ION framebuffer, and the EGL image is made to point to the shared ION framebuffer memory. An OpenGL ES texture object is created, and the EGL image is bound to the texture, so that the texture data is mapped to the shared ION framebuffer managed by GBM. A framebuffer object (FBO) is created and bound to the EGL rendering context, and the texture is attached as a color attachment point. After verifying that the framebuffer object is complete, the OpenGL ES rendering instructions are executed, and the rendering data is written to the shared ION framebuffer. After the GPU rendering is completed, the application is synchronized with the GPU execution status, the framebuffer is marked as rendering complete, and the display compositing module is notified that it can be safely read and displayed on the screen.

[0045] The dynamic frame buffer scaling method is as follows: detect window changes, screen rotation or resolution switching events, release the original frame buffer memory and management object, recalculate and allocate shared ION frame buffer memory according to the new resolution, rebuild the GBM frame buffer object and update the hash table record, and synchronously send new memory handles and parameters to the display composition module to achieve dynamic adaptation of frame buffer size.

[0046] Step 4: The display compositing module receives the shared ION frame buffer handle, GBM side physical address, resolution and image format information, and obtains its own process ID; it maps the shared ION frame buffer memory to the address space of the display compositing module process with read-only permissions, encapsulates the mapped address, physical address, resolution and format into zero-copy layer data, marks the layer as zero-copy type and prioritizes its scheduling to the hardware compositing channel;

[0047] The hardware compositing channel reads the display device resolution and performs an adaptation judgment: if the current architecture is Valhall and the screen resolution does not match the display resolution, the display plane hardware scaling engine is called to complete lossless hardware scaling; if it is another GPU architecture such as Midgard or Bifrost, software scaling is used to complete the screen size adaptation.

[0048] After scaling and adaptation are completed, the display plane and CRTC hardware resources corresponding to the hardware compositing channel are associated, and the GBM frame buffer is bound to the display hardware node. The VSYNC vertical synchronization signal of the display device is synchronized, and the GBM frame buffer content is output to the display device through the page flipping mechanism to complete the layer compositing and screen display. After the display output is completed, the zero-copy memory mapping is released and the hardware compositing channel resources are released. For scenarios using software scaling, the temporary buffer is released synchronously to achieve resource consolidation and recycling throughout the entire process.

[0049] Furthermore, when mapping the shared ION frame buffer memory to the display compositing module process address space with read-only permissions, the minimum required memory capacity is calculated based on the frame buffer resolution and pixel format. The calculated value is compared with the actual mapped memory size. If the required capacity is not greater than the actual mapped memory capacity, the memory mapping is completed normally; otherwise, it is determined that the memory capacity is insufficient, and it is downgraded to data copy mode and an exception log is recorded.

[0050] Furthermore, based on the width, height, and pixel format of the frame buffer, the corresponding number of bits per pixel is obtained, and the minimum required memory capacity is calculated using the formula: Minimum required memory capacity = width * height * number of bits per pixel / 8.

[0051] Example:

[0052] This embodiment employs an OpenHarmony hardware-accelerated rendering method based on native GBM provided by this invention. By modifying the underlying OpenHarmony source code, it achieves native support for the general buffer manager GBM, allowing for integration with the Mesa Panfrost driver without relying on Mesa porting. This method enables OpenGL ES GPU rendering acceleration and is suitable for graphics rendering scenarios in lightweight IoT devices, embedded devices, and mid-range smart devices within the OpenHarmony ecosystem. The specific process includes:

[0053] S1. Resource initialization phase: Kernel layer full architecture hardware capability adaptation. Complete the collection of Mali GPU full architecture information, development of zero-copy memory sharing interface, and Valhall architecture hardware feature adaptation at the Linux kernel layer. Provide hardware-level support for frame buffer allocation and rendering execution, and solve the problems of different architecture adaptation differences and zero-copy memory safe access.

[0054] S1.1, Mali full architecture memory feature awareness interface.

[0055] S1.1.1 Structural Design:

[0056] / / The structure relating DRM BO to the entire Mali GPU architecture

[0057] struct drm_bo_mali_info {

[0058] struct drm_bo base; / / Basic DRM BO structure, containing basic framebuffer attributes

[0059] uint32_t mali_gpu_arch; / / Mali GPU architecture identifier: 0x30 = Midgard (T860), 0x40 = Bifrost (G52), 0x50 = Valhall (G610)

[0060] uint32_t mali_l2_cache_size; / / Mali L2 cache size (KB): Midgard is typically 256KB, Bifrost is 512KB, Valhall is 1024KB

[0061] uint32_t max_render_res; / / Maximum supported rendering resolution (pixels) for GPU: 1920×1080 for Midgard / Bifrost, 2560×1440 for Valhall

[0062] uint32_t arch_feature_flag; / / Architecture feature flags: 0x01 = Valhall unified shader core, 0x02 = Bifrost tiled rendering, 0x04 = Midgard legacy mode

[0063] ion_handle_t ion_hdl; / / OpenHarmony ION memory handle, pointing to the allocated physical memory block

[0064] ion_handle_t shared_ion_hdl; / / ION handle shared with DisplayComposer, pointing to the same physical memory as ion_hdl.

[0065] bool is_slice_supported; / / Whether framebuffer slicing is supported: Bifrost / Valhall enables this by default, Midgard requires manual enabling.

[0066] bool is_zero_copy_en; / / Whether to enable zero-copy mode: when enabled, ion_hdl will have the ION_FLAG_SHARED flag.

[0067] phys_addr_t ion_phys_addr; / / ION memory physical address, used for consistency checks and hardware channel binding

[0068] uint32_t cache_attr; / / Memory cache attributes: 0x01=CACHE_WB (write-back allowed), 0x02=CACHE_NONE (no cache), Valhall defaults to CACHE_WB.

[0069] };

[0070] S1.1.2, drm_bo_mali_create function implementation.

[0071] Full architecture information collection:

[0072] The call to panfrost_gpu_get_full_info(dev, &mali_full_info) retrieves full architecture information such as mali_gpu_arch and mali_l2_cache_size by reading the Mali GPU's architecture identifier register 0x000000 and cache configuration register 0x000010.

[0073] For the Valhall architecture, additionally read the feature control register 0x000020 to determine whether the unified shader core is supported, and set arch_feature_flag |= 0x01.

[0074] Differences in memory allocation strategies:

[0075] Midgard architecture: Because it does not support cache consistency, it calls ion_alloc_kernel(ION_HEAP_TYPE_DMA, size, ION_FLAG_CACHED | ION_FLAG_EXCLUSIVE, &ion_hdl) to allocate exclusive cache memory to avoid data inconsistency;

[0076] Bifrost architecture: Supports basic cache consistency, calls ion_alloc_kernel(ION_HEAP_TYPE_DMA, size, ION_FLAG_CACHED, &ion_hdl) to allocate shared cache memory;

[0077] Valhall architecture: Supports advanced cache coherency by calling ion_alloc_kernel(ION_HEAP_TYPE_DMA, size, ION_FLAG_CACHED | ION_FLAG_SHARED, &ion_hdl) to allocate cache memory with a shared flag, preparing for zero-copy.

[0078] Resolution verification and adjustment:

[0079] If the input resolution, such as 2560*1440, exceeds the current architecture's max_render_res, such as Midgard's 1920*1080, it will automatically downgrade to the maximum supported resolution, and the downgrade log will be recorded through drm_bo_mali_log, including the original resolution, the downgraded resolution, and the architectural reason.

[0080] Physical address acquisition and storage:

[0081] Calling ion_get_phys_addr(ion_hdl, &ion_phys_addr) retrieves the physical address through the OpenHarmony ION kernel interface and stores it in the ion_phys_addr field for subsequent consistency verification and hardware binding.

[0082] S1.2, Zero-copy memory sharing interface.

[0083] The drm_bo_mali_share_ion function is implemented as follows:

[0084] S1.2.1, Pre-verification of shared handle generation:

[0085] Check if mali_bo->is_zero_copy_en is true. If it is false, return -EINVAL. Non-zero copy mode does not support sharing.

[0086] Verify that target_pid is the valid DisplayComposer process ID. Confirm the existence of the process by using find_task_by_pid_ns(target_pid, &init_pid_ns) to avoid invalid sharing.

[0087] S1.2.2 Shared handle generation and access control:

[0088] Calling ion_share_handle(mali_bo->ion_hdl, target_pid, &shared_ion_hdl) generates a shared handle based on the original ION handle. This handle points to the same physical memory as the original handle, but only contains memory access permissions and no memory release permissions.

[0089] Calling ion_set_handle_permission(shared_ion_hdl, target_pid, ION_PERM_READ) grants only read permission to the DisplayComposer process and prohibits writing, thus preventing DisplayComposer from tampering with GPU rendering data;

[0090] Additional permission expiration time: Calling ion_set_handle_expire(shared_ion_hdl, 5000) will set the expiration time to 5000ms. After the timeout, the shared permissions will be automatically revoked to prevent memory leaks.

[0091] S1.2.3 Kernel Layer Consistency Record:

[0092] Initialize the kernel global hash table drm_mali_shared_ion_map, with a size of 1024, and use chaining to resolve collisions.

[0093] Construct a hash table with key-value pairs: the keys are shared_ion_hdl and target_pid, and the values ​​are struct drm_mali_shared_ion_val, which contains ion_phys_addr, mali_bo->base.size, and mali_bo->base.format;

[0094] Call hash_add(drm_mali_shared_ion_map, &val->node, key) to store the sharing relationship in a hash table for subsequent middleware layer verification and query.

[0095] S1.2.4 Shared handle return and logging:

[0096] Store the shared_ion_hdl into mali_bo->shared_ion_hdl and return it to the upper-layer GBM module;

[0097] Calling `drm_bo_mali_log("Shared ION handle %d created for pid %d, phys_addr 0x%lx", shared_ion_hdl, target_pid, mali_bo->ion_phys_addr)` records shared logs, facilitating problem localization.

[0098] S1.3, Valhall architecture plane hardware scaling adaptation.

[0099] The panfrost_valhall_plane_scale_init function is implemented as follows:

[0100] S1.3.1 Hardware scaling engine initialization:

[0101] Call drm_write_reg(dev, 0x120400, 0x00000001) to enable the DRMPLane hardware scaling engine;

[0102] Calling drm_write_reg(dev, 0x120404, 0x00000002) selects the bilinear interpolation algorithm, balancing scaling quality and speed. For higher quality, it can be set to 0x00000003 for Lanczos interpolation.

[0103] S1.3.2 Scaling parameter configuration:

[0104] Define a scaling parameter structure struct valhall_plane_scale_param, which includes source width src_width, source height src_height, target width dst_width, target height dst_height, source physical address src_phys_addr, and target physical address dst_phys_addr;

[0105] Call the source address register drm_write_reg(dev, 0x120410, param->src_phys_addr) to bind the physical address of the GBM frame buffer;

[0106] Call the source size register drm_write_reg(dev, 0x120414, param->src_width |(param->src_height << 16)) to set the source resolution;

[0107] Call the target address register drm_write_reg(dev, 0x120420, param->dst_phys_addr) to bind the physical address of the display output, such as the LCD controller frame buffer address;

[0108] Call the target size register drm_write_reg(dev, 0x120424, param->dst_width |(param->dst_height << 16)) to set the target resolution.

[0109] S1.3.3 Scaling Trigger and Synchronization:

[0110] Call the trigger register drm_write_reg(dev, 0x120430, 0x00000001) to start hardware scaling;

[0111] Call the status register drm_wait_reg(dev, 0x120434, 0x00000001, 100) to wait for scaling to complete, with a timeout set to 100ms to prevent subsequent operations from executing prematurely;

[0112] After scaling is complete, call drm_write_reg(dev, 0x120400, 0x00000000) to disable the scaling engine and save power.

[0113] S2, Frame Buffer Allocation Stage: The middleware layer GBM core module is implemented in the OpenHarmony middleware layer. It includes dynamic frame buffer allocation, ION memory consistency verification, and zero-copy mapping interface. This solves the memory conflict and zero-copy data consistency problem of lightweight devices and provides frame buffer resources for rendering execution.

[0114] S2.1 Dynamic frame buffer scaling mechanism.

[0115] The ohos_gbm_bo_resize function is implemented as follows:

[0116] S2.1.1 Parameter verification before adjustment:

[0117] Convert struct gbm_bo *bo to struct ohos_gbm_bo *ohos_bo, and obtain the current framebuffer parameters, including cur_width, cur_height, and cur_size;

[0118] Check if the new resolution new_width and new_height are valid: if new_width < 32 or new_height < 32, where 32 is the minimum resolution limit, return -EINVAL;

[0119] Calculate the new memory size new_size = new_width * new_height * drm_format_get_bpp(ohos_bo->mali_bo->base.format) / 8. If new_size > 16 * 1024 * 1024, the maximum limit is 16MB, and return -ENOMEM.

[0120] S2.1.2, Differentiation in memory adjustment strategies:

[0121] Memory shrinkage scenario, i.e., new_size < cur_size:

[0122] Calling drm_bo_mali_resize(ohos_bo->mali_bo, new_width, new_height) only updates mali_bo->base.width, mali_bo->base.height, and mali_bo->base.size in the kernel layer, without releasing the original ION memory, thus avoiding the time-consuming reallocation.

[0123] Calling ion_shrink_handle(ohos_bo->mali_bo->ion_hdl, new_size) shrinks the effective access range of ION memory, marking the portion exceeding new_size as unused, but without releasing physical memory;

[0124] Memory expansion scenarios, i.e., new_size > cur_size:

[0125] Check if the current ION memory block supports expansion: Call ion_check_expandable(ohos_bo->mali_bo->ion_hdl, new_size). If it supports expansion (i.e., the physical memory is contiguous and the subsequent space is not occupied), execute ion_expand_handle(ohos_bo->mali_bo->ion_hdl, new_size) to expand the memory size.

[0126] If extension is not supported, call ion_realloc_kernel(ohos_bo->mali_bo->ion_hdl, new_size, ohos_bo->mali_bo->is_zero_copy_en ? ION_FLAG_SHARED : 0, &new_ion_hdl) to allocate a new ION memory block;

[0127] Call memcpy(ion_map_kernel(new_ion_hdl), ion_map_kernel(ohos_bo->mali_bo->ion_hdl), cur_size) to copy the original framebuffer data to the new memory block;

[0128] Call ion_free_kernel(ohos_bo->mali_bo->ion_hdl) to release the old memory block and update mali_bo->ion_hdl and mali_bo->ion_phys_addr.

[0129] S2.1.3, Sharing relationships and DRM information synchronization:

[0130] If zero-copy is enabled, call drm_bo_mali_share_ion(ohos_bo->mali_bo, ohos_bo->ohos_dev->dc_pid) to regenerate the shared handle, ensuring that DisplayComposer accesses the latest memory;

[0131] Call drmModeRmFB(ohos_bo->ohos_dev->drm_fd, ohos_bo->drm_fb_id) to delete the old DRM frame buffer;

[0132] Call drmModeAddFB2(ohos_bo->ohos_dev->drm_fd, new_width, new_height,ohos_bo->mali_bo->base.format, &ohos_bo->mali_bo->base.plane[0].fb_stride, &ohos_bo->mali_bo->ion_phys_addr, &new_drm_fb_id, 0) to register a new DRM frame buffer and update ohos_bo->drm_fb_id.

[0133] S2.1.4 Adjustment Result Return and Log:

[0134] Update ohos_bo->base.width = new_width and ohos_bo->base.height = new_height, and return 0 to indicate success;

[0135] Call ohos_gbm_log("GBM BO resized from %dx%d(%dB) to %dx%d(%dB), zero-copy: %d", cur_width, cur_height, cur_size, new_width, new_height, new_size,ohos_bo->is_zero_copy) to record the adjustment log.

[0136] S2.2, ION memory consistency verification module.

[0137] The ohos_gbm_ion_verify function is implemented as follows:

[0138] S2.2.1 Middleware layer verification preparation:

[0139] Initialize the middleware layer hash table ohos_gbm_shared_ion_verify to have the same structure as the kernel layer hash table for easy synchronization;

[0140] The ohos_gbm_sync_kernel_hash(drm_fd, &ohos_gbm_shared_ion_verify) method synchronizes the drm_mali_shared_ion_map data from the kernel layer using the DRM_IOCTL_MALI_GET_SHARED_ION_MAP instruction, ensuring that the middleware layer and the kernel layer have consistent information.

[0141] S2.2.2 Input parameter validation:

[0142] Receive the shared handle shared_ion_hdl from the DisplayComposer side, the DisplayComposer process ID target_pid, and the physical address mapped by DisplayComposer dc_phys_addr;

[0143] If shared_ion_hdl == 0 or target_pid == 0, return -EINVAL, indicating that the parameter is invalid.

[0144] S2.2.3 Hash table lookup and comparison:

[0145] Construct the query key key = shared_ion_hdl + target_pid, consistent with the kernel layer;

[0146] Call hash_find(ohos_gbm_shared_ion_verify, &val, key) to query the middleware layer hash table and obtain the physical address gbm_phys_addr and the memory size gbm_size of the GBM side record;

[0147] Triple comparison logic:

[0148] Physical address comparison: If dc_phys_addr != gbm_phys_addr, return -EINVAL, indicating address mismatch, possibly due to a shared handle error;

[0149] Memory size comparison: Call ion_get_handle_size(shared_ion_hdl, &dc_size). If dc_size != gbm_size, return -EINVAL. The memory sizes are inconsistent, and the memory may have been tampered with.

[0150] Permission verification: Call ion_get_handle_permission(shared_ion_hdl, target_pid, &dc_perm). If dc_perm != ION_PERM_READ, return -EPERM, indicating a permission error and possible unauthorized modification.

[0151] S2.2.4, Verification passed and subsequent processing:

[0152] If all three comparisons pass, a return value of 0 indicates successful verification.

[0153] If any alignment fails, call ohos_gbm_zero_copy_recover(ohos_bo, &dc_buf) to trigger a zero-copy downgrade to copy mode and record an error log, including the alignment failure item, the actual value, and the expected value.

[0154] S3, Shader Loading Phase: Panfrost Shader Pre-compilation Caching. The shader pre-compilation caching mechanism is implemented in the GBM module. Through pre-compilation, caching, and fast loading, it solves the problem of Panfrost driver shader compilation delay and provides efficient shader resources for rendering execution.

[0155] S3.1, Pre-compilation stage.

[0156] The ohos_gbm_shader_precompile_init function is implemented as follows:

[0157] S3.1.1, Precompiled configuration file parsing:

[0158] Open the precompile configuration file ` / etc / gbm_shader_precompile.conf`. The file format is schema identifier:shader path:shader type. Example:

[0159] 0x50: / system / shaders / ui_vertex.glsl:VERTEX

[0160] 0x50: / system / shaders / ui_fragment.glsl:FRAGMENT

[0161] 0x40: / system / shaders / legacy_ui_fragment.glsl:FRAGMENT

[0162] Parse the file line by line: Call strtok(line, ":") to split the fields and get the architecture identifier arch, shader path shader_path, shader type shader_type, where the shader type is either VERTEX or FRAGMENT;

[0163] Filter configurations that do not match the current architecture: If arch != ohos_bo->ohos_dev->mali_arch, skip the configuration to avoid invalid pre-compilation.

[0164] S3.1.2 Shader source code reading and preprocessing:

[0165] Call fopen(shader_path, "r") to open the shader file and read the source code into the shader_source buffer, which has a maximum size of 64KB.

[0166] Preprocessing source code: Call ohos_gbm_shader_preprocess(shader_source, &processed_source) to replace the architecture macro definitions in the source code, such as #define VALHALL 1, to ensure adaptation to the current architecture.

[0167] S3.1.3 Shader pre-compilation and binary generation:

[0168] The Panfrost driver compilation interface is invoked by calling panfrost_shader_compile(processed_source, shader_type, ohos_bo->ohos_dev->mali_arch, &shader_bin, &bin_len) to generate binary instructions for the corresponding architecture.

[0169] If compilation fails, such as due to a syntax error, call ohos_gbm_log("Precompile shader %s failed,error: %s", shader_path, panfrost_get_error()) to log the error and skip the shader.

[0170] S3.1.4, Cache storage, ION shared cache:

[0171] Calculate the source code hash value: Call ohos_gbm_shader_hash(processed_source, &source_hash), and use the SHA-256 algorithm to generate a 64-bit hash value to ensure uniqueness;

[0172] Build the cache index: cache_key = source_hash + ohos_bo->ohos_dev->mali_arch, which specifies the architecture and hash value to ensure cross-architecture cache isolation;

[0173] Allocate ION cache memory: Call ion_alloc(ION_HEAP_TYPE_DMA, bin_len, ION_FLAG_CACHED | ION_FLAG_SHARED, &cache_ion_hdl) to allocate ION memory specifically for storing shader cache;

[0174] Write cached data: Call ion_map(cache_ion_hdl, &cache_addr) to map memory to user space, and write the binary instruction shader_bin, length bin_len, and type shader_type to the cache address;

[0175] Record cache index: Call hash_add(ohos_bo->ohos_dev->shader_cache_map, &cache_node, cache_key) to associate the cache ION handle cache_ion_hdl with cache_key and store it in the cache hash table.

[0176] S3.2 Loading phase.

[0177] The ohos_gbm_shader_cache_load function is implemented as follows:

[0178] S3.2.1, Processing the application shader source code:

[0179] Receive the shader source code app_source and type app_shader_type passed in by the application;

[0180] Preprocessing source code: Calls ohos_gbm_shader_preprocess(app_source, &processed_app_source), which is consistent with the preprocessing logic in the pre-compilation stage;

[0181] Calculate the application source code hash value: Call ohos_gbm_shader_hash(processed_app_source, &app_source_hash) to generate a hash value consistent with the pre-compilation stage.

[0182] S3.2.2, Cache Query and Matching:

[0183] Construct the query key: query_key = app_source_hash + ohos_bo->ohos_dev->mali_arch;

[0184] Call hash_find(ohos_bo->ohos_dev->shader_cache_map, &cache_val, query_key) to query the cache hash table;

[0185] Cache hit handling:

[0186] If a hit occurs, obtain cache_ion_hdl = cache_val->ion_hdl and cache the ION handle;

[0187] Call ion_map(cache_ion_hdl, &cache_addr) to map cache memory to user space;

[0188] Read the binary instruction shader_bin, length bin_len, and type shader_type from the cache address, and verify that shader_type == app_shader_type to ensure type matching;

[0189] Returns shader_bin and bin_len to the application; loading time is no more than 1ms.

[0190] Cache miss handling:

[0191] The `panfrost_shader_compile(processed_app_source, app_shader_type, ohos_bo->ohos_dev->mali_arch, &shader_bin, &bin_len)` function is called to compile the shader in real time.

[0192] The ohos_gbm_shader_cache_store(processed_app_source, shader_bin, bin_len, app_shader_type) function stores the compilation results in the cache for use the next time they are loaded.

[0193] Returns the compiled shader_bin and bin_len. Real-time compilation takes no more than 80ms for complex shaders.

[0194] S3.2.3 Cache Expiration Management:

[0195] Each time the cache is loaded, call ion_set_handle_expire(cache_ion_hdl, 86400000) to set the cache expiration time to 24 hours, thus extending the cache validity period.

[0196] Call ohos_mem_get_usage(&mem_usage). If mem_usage > 90%, it means that the system memory is insufficient. Call ohos_gbm_shader_cache_cleanup(30%) to clean up 30% of the least used cache and release memory.

[0197] S4. In the rendering execution phase, the OpenHarmony Render Service is modified to achieve zero-copy rendering context creation, GBM frame buffer binding, and direct writing of OpenGL ES instructions to shared ION memory, thus solving the rendering data copying latency problem and providing efficient rendering results for composite display.

[0198] S4.1 Zero-copy rendering context creation.

[0199] The RenderService::CreateZeroCopyContext function is implemented as follows:

[0200] S4.1.1, EGL Display and Configuration Selection:

[0201] Call eglGetDisplay(EGL_DEFAULT_DISPLAY) to get the EGL default display, corresponding to the DRM device;

[0202] Call eglInitialize(dpy, NULL, NULL) to initialize the EGL display and get the EGL version. Only versions 1.5 and above support zero-copy extensions.

[0203] Define a list of EGL configuration properties to support zero-copy:

[0204] const EGLint config_attrs[] = {

[0205] EGL_SURFACE_TYPE, EGL_PBUFFER_BIT,

[0206] EGL_RED_SIZE, 8,

[0207] EGL_GREEN_SIZE, 8,

[0208] EGL_BLUE_SIZE, 8,

[0209] EGL_ALPHA_SIZE, 8,

[0210] EGL_DEPTH_SIZE, 24,

[0211] EGL_STENCIL_SIZE, 8,

[0212] EGL_RENDERABLE_TYPE, EGL_OPENGL_ES3_BIT,

[0213] EGL_ZERO_COPY_HINT_KHR, EGL_TRUE, / / Enable zero-copy hints

[0214] EGL_NONE

[0215] };

[0216] Call eglChooseConfig(dpy, config_attrs, &config, 1, &num_config) to select an EGL configuration that supports zero-copy.

[0217] S4.1.2 Zero-copy EGL context creation:

[0218] Define the EGL context property list to support OpenGL ES 3.1:

[0219] const EGLint context_attrs[] = {

[0220] EGL_CONTEXT_CLIENT_VERSION, 3,

[0221] EGL_CONTEXT_OPENGL_ES3_BIT_KHR,

[0222] EGL_CONTEXT_ZERO_COPY_ENABLE_KHR, EGL_TRUE, / / Enable zero-copy

[0223] EGL_NONE

[0224] };

[0225] Call eglCreateContext(dpy, config, EGL_NO_CONTEXT, context_attrs, &egl_ctx) to create a zero-copy EGL context;

[0226] Call eglMakeCurrent(dpy, EGL_NO_SURFACE, EGL_NO_SURFACE, egl_ctx) to activate the context and prepare for subsequent framebuffer binding.

[0227] S4.1.3, GBM device and context binding:

[0228] Calling ohos_egl_set_gbm_device(egl_ctx, gbm_dev) binds the GBM device handle to the EGL context through the EGL extension interface, enabling EGL to recognize the GBM framebuffer.

[0229] Call eglQueryContext(dpy, egl_ctx, EGL_CONTEXT_ZERO_COPY_STATUS_KHR, &zero_copy_status) to verify whether zero copy is enabled. Zero_copy_status == EGL_TRUE indicates success.

[0230] S4.1.4 Context Storage and Management:

[0231] Store egl_dpy, egl_ctx, and egl_config into the ZeroCopyContext structure of RenderService as a global context for all rendering threads to reuse;

[0232] Call pthread_mutex_init(&context_lock, NULL) to initialize the context mutex lock and avoid multi-threaded conflicts.

[0233] S4.2, GBM framebuffer binding and OpenGL ES rendering.

[0234] The implementation of the GlRenderer::BindZeroCopyBO function is as follows:

[0235] S4.2.1, Obtaining GBM frame buffer attributes:

[0236] Obtain the physical address ion_phys_addr, width, height, and format from ohos_bo;

[0237] Call gbm_bo_get_plane_stride(ohos_bo, 0, &stride) to get the framebuffer line span, in bytes per line.

[0238] S4.2.2, EGL Image Creation, Zero-Copy Binding:

[0239] Define an EGL image attribute list and associate it with a GBM framebuffer:

[0240] const EGLint image_attrs[] = {

[0241] EGL_IMAGE_PRESERVED_KHR, EGL_TRUE,

[0242] EGL_IMAGE_BIND_TO_TEXTURE_RGBA8_OES, EGL_TRUE,

[0243] EGL_IMAGE_OFFSET_KHR, 0,

[0244] EGL_IMAGE_PITCH_KHR, stride,

[0245] EGL_NONE

[0246] };

[0247] Call eglCreateImageKHR(egl_dpy, egl_ctx, EGL_NATIVE_PIXMAP_KHR, ohos_bo, image_attrs) to create an EGL image, which is directly associated with the shared ION memory of the GBM framebuffer without data copying.

[0248] S4.2.3, OpenGL ES texture binding:

[0249] Call glGenTextures(1, &tex_id) to generate an OpenGL ES texture ID;

[0250] Call glBindTexture(GL_TEXTURE_2D, tex_id) to bind the texture;

[0251] Calling glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, egl_image) binds the EGL image to the GBM framebuffer as a texture. At this time, the texture data directly points to the shared ION memory.

[0252] Set texture parameters to avoid stretching distortion:

[0253] glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);

[0254] glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

[0255] glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);

[0256] glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

[0257] S4.2.4 Frame Buffer Object (FBO) Binding and Rendering:

[0258] Call glGenFramebuffers(1, &fbo_id) to generate the FBO ID;

[0259] Call glBindFramebuffer(GL_FRAMEBUFFER, fbo_id) to bind the FBO;

[0260] Call glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, tex_id, 0) to attach the texture-associated GBM framebuffer to the color attachment point of the FBO;

[0261] Calling glCheckFramebufferStatus(GL_FRAMEBUFFER) verifies the integrity of the FBO and should return GL_FRAMEBUFFER_COMPLETE;

[0262] Execute OpenGL ES rendering instructions: Call glViewport(0, 0, width, height) to set the viewport, and call glDrawArrays(GL_TRIANGLES, 0, 3) to perform drawing. The rendering data is written directly to the shared ION memory without intermediate copying.

[0263] S4.2.5, Rendering complete synchronization:

[0264] Call glFinish() and wait for the GPU to finish rendering to ensure that the data is written to the shared ION memory;

[0265] Call eglWaitClient() to synchronize the EGL client and the GPU, preventing subsequent operations from being executed prematurely.

[0266] Calling ohos_gbm_bo_set_render_done(ohos_bo, true) marks the completion of framebuffer rendering, making it available for DisplayComposer to read.

[0267] S5. In the compositing and display stage, OpenHarmony DisplayComposer is modified to achieve zero-copy memory mapping, hardware compositing channel binding, and Valhall architecture plane hardware scaling, solving the problem of compositing data copy latency and efficiently outputting the rendering results to the display device.

[0268] S5.1, Zero-copy memory mapping and consistency verification.

[0269] The DisplayComposer::MapZeroCopyBO function is implemented as follows:

[0270] S5.1.1 Shared Parameter Reception and Preprocessing:

[0271] Receive the shared ION handle shared_ion_hdl, GBM side physical address gbm_phys_addr, width, height, and format from the Render Service;

[0272] Call ohos_dev_get_self_pid(&dc_pid) to obtain the DisplayComposer process ID itself.

[0273] S5.1.2, Zero-copy memory mapping:

[0274] Call ion_map(shared_ion_hdl, &dc_map_addr, &dc_map_size) to map the shared ION memory to the DisplayComposer process address space, with the mapping permission set to read-only;

[0275] Check the mapping size: If dc_map_size < width * height * drm_format_get_bpp(format) / 8, return -ENOMEM, the mapping size is insufficient.

[0276] S5.1.3, Physical Address Acquisition and Consistency Verification:

[0277] Call ion_get_phys_addr(shared_ion_hdl, &dc_phys_addr) to get the physical address mapped on the DisplayComposer side;

[0278] Calling `ohos_gbm_ion_verify(shared_ion_hdl, dc_pid, dc_phys_addr, gbm_phys_addr, width, height, format)` triggers middleware layer consistency checks.

[0279] Verify the physical address: dc_phys_addr must be exactly the same as gbm_phys_addr;

[0280] Verify memory size: dc_map_size must be greater than or equal to the actual size of the framebuffer;

[0281] Verification format: The format must be consistent with the GBM frame buffer format;

[0282] If the verification fails, call ohos_gbm_zero_copy_recover(shared_ion_hdl, dc_map_addr, &dc_buf) to trigger a downgrade to copy mode and log the error.

[0283] S5.1.4, Mapping Result Storage and Preparation for Synthesis:

[0284] The mapping address dc_map_addr, physical address dc_phys_addr, width, height, and format are stored in the ZeroCopyLayer structure as composite layer data;

[0285] Calling dc_layer_set_type(layer, DC_LAYER_TYPE_ZERO_COPY) marks the layer type as zero copy, prioritizing its scheduling to the hardware compositing channel.

[0286] S5.2, Zero-Copy Hardware Compositing and Valhall Hardware Scaling.

[0287] The DisplayComposer::ComposeZeroCopyLayer function is implemented as follows:

[0288] S5.2.1, Selection and Priority Scheduling of Synthesis Channels:

[0289] Call dc_hw_compose_channel_get(DC_HW_COMPOSE_TYPE_GBM_ZERO_COPY, &channel) to obtain the zero-copy dedicated hardware compositing channel;

[0290] Calling dc_hw_compose_channel_set_priority(channel, 0) sets the channel priority to the highest (0 for highest, 10 for lowest) to ensure that zero-copy layers are composited first.

[0291] If the channel is occupied, call dc_hw_compose_channel_wait(channel, 10) to wait for 10ms, and switch to the backup channel after the timeout.

[0292] S5.2.2, Valhall architecture hardware scaling judgment and execution:

[0293] Call dc_display_get_resolution(&display_width, &display_height) to get the display device resolution;

[0294] Determine if scaling is needed: If width != display_width || height != display_height, and ohos_bo->ohos_dev->mali_arch == 0x50, it indicates a Valhall architecture, and hardware scaling is performed;

[0295] Hardware scaling execution:

[0296] The hardware scaling engine is initialized by calling panfrost_valhall_plane_scale_init(ohos_bo->ohos_dev->drm_fd, dc_phys_addr, width, height, display_width, display_height, format);

[0297] The panfrost_valhall_plane_scale_execute(ohos_bo->ohos_dev->drm_fd, &scale_result) is called to perform scaling and obtain the scaled physical address scaled_phys_addr.

[0298] Calling drm_write_reg(ohos_bo->ohos_dev->drm_fd, 0x120430, 0x00000001) triggers scaling, waits for completion, and times out to 100ms.

[0299] Non-Valhall architecture handling: For Midgard / Bifrost architecture, call dc_sw_scale(dc_map_addr, width, height, display_width, display_height, format, &scaled_buf) to perform software scaling, which takes less than 15ms.

[0300] S5.2.3 Hardware synthesis channel binding and display output:

[0301] To obtain DRM Plane and CRTC information, call dc_hw_compose_get_plane_crtc(channel, &plane_id, &crtc_id);

[0302] Binding the framebuffer to the Plane: Call drmModeSetPlane(ohos_bo->ohos_dev->drm_fd,plane_id, crtc_id, ohos_bo->drm_fb_id, 0, 0, 0, width, height, 0, 0, width <<16, height << 16), which is replaced with scaled_phys_addr if hardware scaling is performed;

[0303] Synchronize the VSYNC signal: Call dc_vsync_wait(&vsync_ts) to wait for the display device's VSYNC signal;

[0304] Trigger display update: Call drmModePageFlip(ohos_bo->ohos_dev->drm_fd, crtc_id,ohos_bo->drm_fb_id, DRM_MODE_PAGE_FLIP_EVENT, NULL) to output the frame buffer to the display device and complete the composite display.

[0305] S5.2.4 Resource Release and Cleanup:

[0306] Call ion_unmap(dc_map_addr) to unmap the zero-copy memory.

[0307] Call dc_hw_compose_channel_release(channel) to release the hardware composition channel;

[0308] If software scaling is performed, call free(scaled_buf) to release the software scaling buffer.

[0309] S6. Error handling phase: Implement GBM error degradation mechanism. For exceptions such as frame buffer allocation failure, shader compilation error, zero-copy sharing failure, etc., trigger multiple rounds of degradation strategy to ensure that the application does not crash and maintains basic rendering capabilities.

[0310] S6.1, Frame buffer allocation failure, downgrade.

[0311] The ohos_gbm_bo_recover_mem function is implemented as follows:

[0312] S6.1.1 Error Detection and Root Cause Analysis:

[0313] It receives error_code, cur_width, cur_height, cur_format, and remaining memory_mem. An error_code of 1 indicates insufficient memory.

[0314] Calculate the memory required for the current framebuffer: cur_size = cur_width * cur_height * drm_format_get_bpp(cur_format) / 8;

[0315] Analyze the degree of insufficient memory: If remaining_mem < cur_size * 0.5, the remaining memory is less than 50% of the current requirement, triggering a double downgrade of resolution and format; if remaining_mem ≥ cur_size * 0.5, only resolution downgrade is triggered.

[0316] S6.1.2, Multi-round resolution degradation strategy:

[0317] Define the resolution downgrade sequence: 1920*1080 to 1280*720 to 800*480 and then to 480*320, each time reducing the resolution by approximately 40%;

[0318] Round-by-round allocation attempt:

[0319] Round 1: Try 1280*720, calculate new_size = 1280*720*drm_format_get_bpp(cur_format) / 8. If new_size ≤ remaining_mem*0.8, reserve 20% of memory and call drm_bo_mali_create(..., 1280, 720, ...) to attempt allocation;

[0320] If it fails, try 800*480 in the second round and repeat the above logic;

[0321] If it still fails, the third round will try the lowest resolution of 480*320. If it still fails, the format will be downgraded.

[0322] S6.1.3 If resolution downgrading still fails, then the downgrade format is:

[0323] Define a format degradation sequence, from low to high memory requirements: RGB565 to RGB888 and then to ARGB8888;

[0324] Try it round by round:

[0325] Trying RGB565: Calculate new_size = 480*320*2, which is 307.2KB. If new_size ≤ remaining_mem * 0.8, then call drm_bo_mali_create(..., 480, 320, DRM_FORMAT_RGB565, ...).

[0326] If it fails, try RGB888 (460.8KB) and repeat the logic.

[0327] Downgrade result verification and return:

[0328] If the allocation is successful, return the new frame buffer new_ohos_bo, new_width, new_height, and new_format, and record the downgrade log, including the original parameters, the downgraded parameters, and the remaining memory;

[0329] If it still fails, return -ENOMEM and call ohos_gbm_log("GBM BO recover failed, noenough memory (remaining: %dB)", remaining_mem), triggering application-layer basic rendering, such as solid color filling.

[0330] S6.2, Zero-copy sharing failure downgrade.

[0331] The ohos_gbm_zero_copy_recover function is implemented as follows:

[0332] S6.2.1 Error Detection and Cause Classification:

[0333] Receive error_type, shared_ion_hdl, gbm_bo, dc_buf, where 1 indicates insufficient permissions, 2 indicates inconsistent physical addresses, and 3 indicates expired shared handle;

[0334] Perform preprocessing for different error types:

[0335] Insufficient permissions: Attempt to regenerate the shared handle by calling drm_bo_mali_share_ion(gbm_bo->mali_bo, gbm_bo->ohos_dev->dc_pid). If this still fails, downgrade.

[0336] Physical address mismatch: Call ohos_gbm_log("Zero-copy verify failed: GBM phys0x%lx, DC phys 0x%lx", gbm_bo->ion_phys_addr, dc_phys_addr) to directly downgrade;

[0337] Shared handle expired: Regenerate the shared handle; if this fails, downgrade it.

[0338] S6.2.2, Copy mode downgrade execution:

[0339] Call gbm_bo_map(gbm_bo, 0, 0, gbm_bo->base.width, gbm_bo->base.height, GBM_BO_TRANSFER_READ, &stride, &gbm_map_addr) to map the GBM framebuffer to user space;

[0340] Allocate a DisplayComposer copy buffer: call malloc(gbm_bo->base.width * gbm_bo->base.height * drm_format_get_bpp(gbm_bo->mali_bo->base.format) / 8) to generate dc_buf;

[0341] Perform data copy: Call memcpy(dc_buf, gbm_map_addr, gbm_bo->base.width *gbm_bo->base.height * drm_format_get_bpp(gbm_bo->mali_bo->base.format) / 8) to copy the GBM framebuf data to dc_buf;

[0342] Call gbm_bo_unmap(gbm_bo) to unmap the GBM frame buffer.

[0343] S6.2.3, Copy Mode Composition and Cleanup:

[0344] The software composition is performed by calling dc_sw_compose(dc_buf, gbm_bo->base.width, gbm_bo->base.height, gbm_bo->mali_bo->base.format, &composed_buf);

[0345] Call dc_display_output(composed_buf, gbm_bo->base.width, gbm_bo->base.height) to output the composite result to the display device;

[0346] Call `free(dc_buf)` and `free(composed_buf)` to release the copy buffer;

[0347] Record degradation logs: Call ohos_gbm_log("Zero-copy recover to copy mode, errortype: %d", error_type) for easy subsequent optimization.

[0348] Experiments have verified that this invention has the following technical effects:

[0349] Performance advantages include: a zero-copy mechanism that eliminates data copying from rendering to compositing, reducing compositing time by 40% and end-to-end latency from 18ms to 9ms; Valhall architecture's Plane hardware scaling reducing scaling time from 15ms to 2ms and bandwidth requirements by 40%; a shader pre-compilation cache reducing complex shader loading time from 80ms to 10ms and improving application startup speed by 87.5%; and dynamic framebuffer scaling reducing resource adjustment time from 5ms to 0.8ms and improving frame rate stability for concurrent rendering of multiple applications by 30%.

[0350] With advantages in resources and stability, dynamic frame buffering and zero-copy reduce frame buffer usage on lightweight devices from 4MB to 1.2MB, reducing memory usage by 70% and avoiding memory overflow; the error degradation mechanism reduces the application crash rate by 99%, and can maintain basic rendering even in the event of hardware anomalies such as insufficient memory or permission errors; the native GBM library, including the full architecture and zero-copy logic, is only 480KB in size, reducing resource usage by 99% compared to porting Mesa, and is adapted to lightweight devices.

[0351] It boasts compatibility advantages, supporting all Mali architectures including Midgard, Bifrost, and Valhall, adapting to different GPU models such as T860, G52, and G610 without modifying the core logic; it follows the GBM standard interface, allowing for integration without modifying the Panfrost driver; it supports all features of OpenGL ES 3.1, including instantiated rendering and high dynamic range; and it adapts to all OpenHarmony devices, supporting resolutions from 480*320 to 2560*1440, and more than 15 formats.

[0352] Advantages in maintainability and scalability: Modular design, adding new architectures only requires extending the full architecture interfaces of the kernel layer and the adaptation logic of the middleware layer, reducing maintenance costs by 60%; Error logs and degradation records contain detailed parameters, such as physical address, resolution, and architecture, facilitating problem localization and subsequent optimization; Modules such as zero-copy consistency verification and hardware scaling can be reused in other cross-process memory sharing scenarios, such as video decoding to display, demonstrating strong scalability.

[0353] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hardware-accelerated rendering method for OpenHarmony based on native GBM, characterized in that, Specifically, the following steps are included: When the OpenHarmony system starts, it loads the Mali GPU driver and identifies its architecture and core parameters. It marks the unified shader core capabilities for the Valhall architecture, initializes the ION memory manager and zero-copy sharing mechanism, builds the first structure associated with the DRM BO and MaliGPU to manage rendering resources, allocates cache memory according to the architecture type and records it into the structure, and establishes the first hash table to manage the shared ION framebuffer. At the same time, the GBM module completes shader pre-compilation, cache adaptation and persistent storage, and records it into the second hash table. After the application starts, it initializes the system's default display device, filters compliant EGL configurations, creates and activates the EGL rendering context, associates and binds the GBM device with the rendering context, and enables the thread mutual exclusion mechanism. After an application initiates an OpenGL ES rendering request, it can request and dynamically scale a shared ION frame buffer as needed, and synchronously send out relevant parameters. After completing the shared memory consistency check, the application passes in the shader source code and type to be used. The GBM module loads the shader or compiles and updates it in real time according to the second hash table, and sequentially creates and binds the EGL image, OpenGL ES texture and FBO rendering link. After verification, the rendering operation is performed and the data is written to the shared ION frame buffer. After the GPU execution status is synchronized, the display compositing module is notified to read and use it. The display compositing module receives frame buffer-related parameters and obtains its own process identifier. It maps the shared ION frame buffer memory in read-only mode, encapsulates information to generate a zero-copy layer, and connects to the hardware compositing channel. It uses different scaling methods to adapt the screen size based on the differences in GPU architecture, binds display hardware resources and synchronizes the VSYNC signal, relies on the page flipping mechanism to complete the screen compositing output, and finally completes the release and recycling of resources throughout the entire process.

2. The OpenHarmony hardware-accelerated rendering method according to claim 1, characterized in that, The method for requesting and dynamically scaling the shared ION frame buffer on demand, and synchronously sending relevant parameters, is as follows: Request a frame buffer based on the initial display resolution. The system allocates shared ION frame buffer memory and obtains the physical address, creates a GBM frame buffer object, generates a shared ION frame buffer handle, and sends it to the display composition module. When the application window size changes, the screen rotates, the resolution is switched, or the system memory is insufficient, dynamic frame buffer scaling is triggered to determine the new frame buffer size and the effective memory access range. The shared ION frame buffer handle is updated and regenerated, and then sent to the display composition module again.

3. The OpenHarmony hardware-accelerated rendering method according to claim 2, characterized in that, The dynamic frame buffer scaling method is as follows: detect window changes, screen rotation or resolution switching events, release the original frame buffer memory and management object, recalculate and allocate shared ION frame buffer memory according to the new resolution, rebuild the GBM frame buffer object and update the hash table record, and synchronously send the new memory handle and parameters to the display compositing module.

4. The OpenHarmony hardware-accelerated rendering method according to claim 1, characterized in that, The application inputs the shader source code and type to be used. The GBM module completes the shader loading or real-time compilation and update according to the second hash table as follows: The GBM module preprocesses the shader source code according to the pre-compilation rules and generates a query key. It then searches for a matching item in the second hash table. If a match is found, the shader binary instructions are loaded from the shader shared ION memory. If no match is found, the shader source code is compiled in real time to complete the loading, and the generated binary instructions are written to the shader shared ION memory, updating the second hash table.

5. The OpenHarmony hardware-accelerated rendering method according to claim 4, characterized in that, The method for preprocessing according to pre-compilation rules and generating query keys is as follows: For the current Mali GPU architecture, we uniformly replaced the architecture-related macro definitions and platform adaptation identifiers in the source code, and shielded the incompatible architecture branch logic, so that the shader syntax and macro definitions are adapted to the native GPU hardware architecture. Clean up redundant spaces, invalid comments, and extra line breaks; align code formatting; eliminate differences in writing style; expand conditional compilation branches uniformly; and output standardized versions of shader source code. Based on the preprocessed standardized source code, a fixed hash algorithm is used to generate a unique hash fingerprint as the query key.

6. The OpenHarmony hardware-accelerated rendering method according to claim 1, characterized in that, The method of adapting screen size by using different scaling methods based on GPU architecture differences is as follows: When the current architecture is Valhall and the screen resolution does not match the display resolution, the display plane hardware scaling engine is invoked to complete lossless hardware scaling; if other GPU architectures are used, software scaling is used to adapt the screen size.

7. The OpenHarmony hardware-accelerated rendering method according to claim 1, characterized in that, When mapping the shared ION framebuffer memory to the display compositing module process address space with read-only permissions, the minimum required memory capacity is calculated based on the framebuffer resolution and pixel format. The calculated value is compared with the actual mapped memory size. If the required capacity is not greater than the actual mapped memory capacity, the memory mapping is completed normally. Otherwise, if the memory capacity is insufficient, the system will downgrade to data copy mode and log the exception.

8. The OpenHarmony hardware-accelerated rendering method according to claim 7, characterized in that, The minimum memory capacity is calculated as follows: based on the width, height, and pixel format of the frame buffer, the corresponding number of bits per pixel is obtained, and the minimum memory capacity is calculated using the formula: minimum memory capacity = width * height * number of bits per pixel / 8.

9. The OpenHarmony hardware-accelerated rendering method according to claim 1, characterized in that, The method for initializing the ION memory manager and zero-copy sharing mechanism is as follows: the ION memory allocator is initialized when the system starts, and the physical contiguous memory pool and cross-process sharing capability are configured; a direct physical address mapping path between the Mali GPU and ION memory is established, and the GPU cache consistency synchronization strategy is enabled.

10. The OpenHarmony hardware-accelerated rendering method according to claim 1, characterized in that, The core fields of the first structure include: basic framebuffer attributes, including the width, height, and pixel format of the framebuffer; GPU architecture identifier, used to distinguish between the three architectures: Midgard, Bifrost, and Valhall; hardware parameters, including L2 cache size and the maximum rendering resolution supported by the GPU; architecture feature markers, used to identify the rendering capabilities specific to each architecture; memory-related parameters, including ION memory handles, cross-process shared ION handles, and physical memory addresses; and function switches, including framebuffer slicing support status and zero-copy enabled status. The cache attribute is set to a memory caching mode, adapting to different architectures and ensuring cache consistency.

Images