An Android container application display method based on a cross-system zero-copy rendering link

Abstract

This invention discloses an Android container application display method based on a cross-system zero-copy rendering link. HarmonyOS is used as the host and Android as the container. Both parties establish encrypted communication through a memory coordination service and a proxy daemon process to complete capability negotiation and generate a cross-system capability matrix. The host creates a secure shared buffer pool based on this matrix, assigning unique identifiers and DRM Prime handles to the buffers. The container-side application initiates a rendering buffer request, the host matches and distributes the buffers, and the container-side imports the handles to bind the rendering context. After rendering is complete, the application marks the completion status through a DRM synchronization object and notifies the host. The host verifies the data and waits for a synchronization signal. At the VSync time, the host submits the buffer handles to the display controller with zero copy to complete the on-screen display.

Description

Technical Field

[0001] This invention belongs to the field of computer software development technology, specifically relating to an Android container application display method based on a cross-system zero-copy rendering link. Background Technology

[0002] When running Android applications in containers within OpenHarmony, the cross-system transfer of rendering results is a core performance bottleneck. Existing solutions are primarily based on EGLImage, a shared resource mechanism provided by EGL (Embedded-System Graphics Library). EGLImage is mainly used for efficiently sharing 2D image data between different APIs or processes, especially in graphics rendering scenarios such as OpenGL ES and OpenVG, to achieve zero-copy texture uploads and significantly improve performance. However, existing solutions based on EGLImage rely on GPU hardware support. On unsupported devices, data transfer requires CPU copying or format conversion, leading to increased latency and frame rate fluctuations. Specifically, the main problems are as follows: First, EGLImage relies on GPU manufacturers for implementation, resulting in poor compatibility and a low-end hardware adaptation rate of less than 40%. Second, in traditional shared solutions, the buffer needs to go through multiple steps, from Android creation to format conversion to OpenHarmony reading, involving 2 to 3 data copies, with each copy taking 3 to 5 ms. Third, cross-system synchronization relies on software signals (such as Android Fence), with synchronization latency of 5 to 8 ms, which can easily lead to screen tearing due to frame loss. Fourth, the buffer size is fixed and cannot adapt to dynamic resolution changes in Android applications (such as switching between portrait and landscape modes), causing display interruptions of over 500 ms during switching. Fifth, cross-system communication suffers from issues such as missing permissions and data packet loss, resulting in a silent failure rate of up to 15% in the rendering chain. Summary of the Invention

[0003] In view of this, the present invention provides an Android container application display method based on a cross-system zero-copy rendering link. For scenarios where the GPU does not support EGL Image, the method achieves high-performance on-screen display of Android application rendering results on the HarmonyOS host side by constructing a cross-system unified memory pool, a hardware-level synchronization mechanism, and an adaptive rendering link.

[0004] This invention provides an Android container application display method based on a cross-system zero-copy rendering link, which specifically includes the following steps:

[0005] Using HarmonyOS as the host and Android as the container, the host starts a memory coordination service to transmit data with the first frame structure between the host and the container; the container starts a memory proxy daemon process to establish a connection with the memory coordination service, complete compatibility verification, and generate a cross-system capability matrix table.

[0006] The memory coordination service creates a shared buffer pool based on the cross-system capability matrix table. The buffer consists of a metadata area and a data area. The data area stores image pixel data, and the metadata area stores relevant parameters of the image pixel data. The service allocates a HarmonyOS file descriptor (FD) and a buffer ID to the buffer, binds them to the HarmonyOS layer, and converts them into DRM Prime handles.

[0007] When an Android application starts, it sends a buffer request. SurfaceFlinger forwards this request to the memory proxy daemon. The memory proxy daemon encapsulates the buffer request into a first-frame structure buffer request packet and sends it to the memory coordination service. The memory coordination service retrieves the target buffer from the shared buffer pool based on the buffer request packet, encapsulates the corresponding HarmonyFD, buffer ID, DRM Prime handle, metadata area offset, and memory attributes into a first-frame structure cross-buffer packet, and sends it to the memory proxy daemon. The memory proxy daemon imports the HarmonyFD to obtain the first buffer handle, creates an image buffer, binds it to the Android application's rendering context, and sends a binding completion notification to the host.

[0008] The Android application executes drawing instructions to write image pixel data into the image buffer. The memory proxy daemon creates a synchronization object in timeline mode, imports the GPU rendering completion fence to obtain the synchronization object handle and writes it into the metadata area, and sends a rendering completion packet containing the current frame number to the host. The memory coordination service obtains the current frame number, and when the sequence number on the timeline is updated to the current frame number, it obtains and submits the buffer to complete the display.

[0009] Furthermore, the shared buffer pool dynamically expands and contracts according to the application status and resolution. Specifically, when the resolution is switched, a new resolution buffer is allocated, the old buffer is marked as available for transition, and the container side is notified to switch when the new buffer is ready. When the Android application is running in the foreground, it maintains M buffers. When it enters the background, it shrinks to N minimum buffers and releases excess memory. When it resumes the foreground, it expands, where M is greater than N.

[0010] Furthermore, the memory coordination service obtains the target buffer from the shared buffer pool based on the buffer request packet in the following way: it prioritizes selecting the buffer whose format conforms to the double-ended mapping relationship, whose size is not less than the requested resolution, and whose access latency is the lowest; if no buffer meets the conditions, it triggers immediate allocation.

[0011] Furthermore, when the hardware does not support DRM hardware synchronization, a Fence mechanism is used to achieve software synchronization. Specifically, the GPU rendering completion fence and the buffer release fence are merged into a composite fence, the fence file descriptor is copied, and then the memory agent daemon sends a rendering completion packet containing the buffer ID, fence file descriptor, frame timestamp, and metadata checksum to the host.

[0012] Furthermore, when the container's rendering API is Vulkan, Vulkan checks the validity of the Harmony FD. If valid, it converts the Harmony FD into a Vulkan-specific GPU memory object, sets the memory offset of the GPU memory object to a set number of bytes to skip the metadata area, builds the rendering command for the Android application, submits the rendering command to the GPU for rendering, writes the rendered image pixel data into the shared memory corresponding to the GPU memory object, and records the rendering completion timestamp after rendering is complete.

[0013] Furthermore, the shared buffer pool is a memory region shared by the CPU and GPU, tamper-proof, and supporting cross-system DMA mapping, and is only accessible to the memory coordination service and memory agent daemon.

[0014] Furthermore, after receiving the cross-buffer packet, the memory proxy daemon verifies the cross-buffer packet. If the verification is successful, it imports the Harmony FD of the target buffer into the container side, obtains the buffer handle of the Android system and records it as the first buffer handle, and verifies the legality of the memory attributes.

[0015] Furthermore, the memory coordination service obtains the current frame sequence number as follows: after receiving the rendering completion package, the memory coordination service recalculates the checksum of the data area corresponding to the buffer ID, verifies whether the checksum is the same as the metadata checksum in the rendering completion package. If they are different, it requests the container to retransmit the rendering result; if they are the same, it obtains the current frame sequence number.

[0016] Furthermore, the method of acquiring and submitting the buffer to complete the display when the sequence number on the timeline is updated to the current frame sequence number is as follows: The DRM kernel driver adds the corresponding process of the memory coordination service to the waiting queue and puts it to sleep. When the sequence number on the timeline is updated to the current frame sequence number, the DRM kernel driver wakes up the corresponding process of the memory coordination service. The memory coordination service acquires the buffer according to the buffer ID, binds the DRM Prime handle of the buffer to the display layer, and within a set time after the VSync signal is triggered, the memory coordination service submits the DRM Prime handle to the display controller. The display controller then directly accesses the buffer through the handle and completes the display.

[0017] Furthermore, when the image pixel format is RGB, the DRM Prime handle is submitted to the display controller for direct hardware scanning and output; when it is YUV, it is processed by the GPU hardware synthesis unit, and the display parameters are submitted using DRM atomic mode.

[0018] Beneficial effects:

[0019] This invention uses HarmonyOS as the host system and Android as the container system. The host system initiates a memory coordination service with distributed data synchronization permissions and establishes a two-way encrypted communication channel. The container system initiates a memory proxy daemon to complete connection and parameter compatibility checks, generating a cross-system capability matrix table containing memory allocator, pixel format mapping table, synchronization mechanism, and buffer pool parameters. Based on this matrix table, the host system creates a shared buffer pool shared by the CPU and GPU, supporting cross-system DMA mapping. It allocates file descriptors, global IDs, and DRM Prime handles to each buffer and completes layer binding. When the Android application starts on the container side, it initiates a buffer request to the memory proxy daemon through SurfaceFlinger. The host-side memory coordination service matches the target buffer from the buffer pool and returns the relevant handles and attributes to the container side. The container side completes handle import and graphics canvas binding. The Android application directly renders image pixel data to the shared buffer. The container side generates a DRM timeline synchronization object and updates the metadata status, sending a rendering completion packet to the host side. After verifying data integrity and waiting for the hardware synchronization signal to wake up, the host side sends the DRM... The Prime handle is bound to the display layer with zero copy and submitted to the display controller within the VSync signal cycle to complete the on-screen output, enabling high-performance display even without EGL Image support. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating an Android container application display method based on a cross-system zero-copy rendering link, as provided by the present invention. Detailed Implementation

[0021] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0022] This invention provides an Android container application display method based on a cross-system zero-copy rendering link. The core idea is as follows: using HarmonyOS as the host system and Android as the container system, the host system starts a memory coordination service with distributed data synchronization permissions and establishes a two-way encrypted communication channel. The container system starts a memory proxy daemon to complete connection and parameter compatibility verification, generating a cross-system capability matrix table containing memory allocators, pixel format mapping tables, synchronization mechanisms, and buffer pool parameters. Based on this matrix table, the host system creates a shared buffer pool shared by the CPU and GPU, supporting cross-system DMA mapping. Each buffer is allocated a file descriptor, global ID, and DRM Prime handle, and layer binding is completed. When the Android application starts on the container side, it initiates a buffer request to the memory proxy daemon through SurfaceFlinger. The host-side memory coordination service matches the target buffer from the buffer pool and returns the relevant handles and attributes to the container side. The container side completes handle import and graphics canvas binding. The Android application directly renders image pixel data to the shared buffer. The container side generates a DRM timeline synchronization object and updates the metadata status, sending a rendering completion packet to the host side. After verifying data integrity and waiting for the hardware synchronization signal to wake up, the host side displays the DRM... The Prime handle is bound to the display layer with zero copy and submitted to the display controller within the VSync signal cycle to complete the on-screen output.

[0023] This invention provides an Android container application display method based on a cross-system zero-copy rendering link, the processing flow of which is as follows: Figure 1 As shown, the specific steps include:

[0024] Step 1: Using HarmonyOS as the host system and Android as the container system; Start a memory coordination service with distributed data synchronization permissions on the host side to create a two-way encrypted communication channel between the host side and the container side. The communication channel can directly transmit core data such as buffer handles, format information and synchronization signals. All data transmitted through the communication channel must be encapsulated into a first frame structure.

[0025] The container side starts a memory agent daemon to establish a connection with the memory coordination service. The container side sends its supported memory types, maximum buffer size, pixel format list, hardware synchronization capabilities, and system version to the host side. The host side returns buffer alignment requirements, rendering API format rules, and supported buffers to the container side. The container side and the host side perform compatibility checks on the parameters they receive. If the check passes, the connection is established and a cross-system capability matrix table is generated, including the memory allocator, the dual-ended pixel format mapping table, the synchronization mechanism, shared buffer pool parameters, and fault tolerance configuration. The dual-ended pixel format mapping table contains the corresponding pixel format pairs between the container side and the host side, as well as an identifier for whether hardware acceleration is supported. The buffer pool parameters include the initial size, maximum size, expansion threshold, and shrinkage threshold.

[0026] The first frame structure includes a frame header, type identifier, data length, payload, and checksum. The frame header includes the protocol version and magic word.

[0027] Step 2: The host-side memory coordination service uses the memory allocator and shared buffer pool parameters from the cross-system capability matrix table to create a shared buffer pool. The shared buffer pool is set to be shared by CPU and GPU, tamper-proof, and support cross-system DMA mapping, and is only accessible to the memory coordination service and memory agent daemon. The buffers in the shared buffer pool consist of a metadata area and a data area. The metadata area stores the data pixel format, resolution, timestamp, synchronization status, checksum, and version number of the data in the buffer, while the data area stores the image pixel data. Each buffer in the shared buffer pool is assigned a HarmonyOS file descriptor (FD) and a globally unique buffer ID. The buffer is bound to the HarmonyOS layer, and then the buffer is converted into a DRMPMime handle, and the lifecycle status of the handle is recorded.

[0028] Furthermore, the shared buffer pool can dynamically scale and schedule based on application status and resolution. It achieves adaptive management by listening to resolution changes on the container side and Android application foreground / background switching events. Specifically, when the resolution changes, a new resolution buffer is allocated, the old buffer is marked as available for transition, and the container side is notified to switch when the new buffer is ready. When the Android application is in the foreground, M buffers are maintained to ensure smooth rendering. When it enters the background, it shrinks to N minimum buffers and releases excess memory. When it resumes the foreground, it quickly expands, where M is greater than N. At the same time, it automatically expands according to the memory usage rate, with the expansion limit being a set percentage of the system memory, for example, 15%.

[0029] The host side implements buffer pool expansion, contraction, and memory release when switching between foreground and background in Android applications through a unified application state callback interface.

[0030] Step 3: When the container-side Android application starts, the Surface corresponding to the application sends a buffer request containing the current resolution, rendering API and rendering purpose to SurfaceFlinger. SurfaceFlinger forwards the buffer request to the memory proxy daemon process, which encapsulates the buffer request into a buffer request packet with the first frame structure and sends it to the memory coordination service.

[0031] The memory coordination service retrieves a matching buffer from the shared buffer pool based on the parameters in the buffer request packet as the target buffer. It then encapsulates the target buffer's HarmonyFD, buffer ID, DRM Prime handle, metadata area offset, and memory attributes into a cross-buffer packet with a first-frame structure and sends the cross-buffer packet to the memory agent daemon on the container side.

[0032] After receiving the cross-buffer packet, the memory proxy daemon verifies the cross-buffer packet. If the verification is successful, it imports the HarmonyFD of the target buffer into the container side, obtains the buffer handle of the Android system and records it as the first buffer handle, and verifies the legality of the memory attributes. It creates an image buffer with the current resolution, Android pixel format, rendering purpose, first buffer handle and row span as parameters, returns the image buffer to the Android application Surface, binds the image buffer to the rendering context of the Android application as an Android graphics canvas, and sends a binding completion notification containing the buffer ID and binding timestamp to the host side.

[0033] The memory coordination service retrieves a target buffer from the shared buffer pool based on the request parameters in the buffer request packet. Specifically, it prioritizes buffers that conform to the double-ended mapping table, have a size no smaller than the requested resolution, and have the lowest access latency. If no suitable buffer is found, it triggers immediate allocation, and the allocation time is no more than the set duration.

[0034] Rendering uses include GPU texturing, GPU rendering, and external display access.

[0035] Step 4: The Android application executes drawing instructions, writes the rendered image pixel data into the image buffer, and writes the rendering completion timestamp into the metadata area of ​​the buffer after rendering is complete. The memory proxy daemon uses the DRM graphics card device to create a synchronization object in timeline mode, imports the fence rendered by the GPU into the timeline to obtain the synchronization object handle, writes the synchronization object handle into the metadata area of ​​the buffer, and updates the synchronization status in the metadata area to be displayable. The memory proxy daemon sends a rendering completion packet containing the buffer ID, current frame number, timestamp, and metadata checksum of the first frame to the host side, and then marks the image buffer as ready to be displayed, prohibiting repeated writing.

[0036] Furthermore, when the hardware does not support DRM hardware synchronization, this invention uses a fence mechanism to achieve software synchronization. Specifically, the GPU rendering completion fence and the buffer release fence are merged into a composite fence, and the fence file descriptor is copied to ensure that the synchronization signal can be used normally in cross-process scenarios. At this time, the memory proxy daemon sends a rendering completion packet containing the buffer ID, fence file descriptor, frame timestamp, and metadata checksum to the host side.

[0037] Furthermore, when the container-side rendering API is Vulkan, the container-side Vulkan checks the validity of the Harmony FD. If valid, it converts the Harmony FD into a Vulkan-specific GPU memory object, sets the memory offset of the GPU memory object to a set number of bytes to skip the metadata area, thereby allowing Vulkan to directly use the shared memory pointed to by the Harmony FD; it builds the rendering command for the Android application, submits the rendering command to the GPU for rendering, writes the rendered image pixel data into the shared memory corresponding to the GPU memory object, and records the rendering completion timestamp after rendering is complete.

[0038] Step 5: After receiving the rendering completion package, the host-side memory coordination service recalculates the checksum of the data area corresponding to the buffer ID and verifies whether the checksum is the same as the metadata checksum in the rendering completion package. If they are different, it requests the container to retransmit the rendering result. If they are the same, it obtains the current frame sequence number. The DRM kernel driver adds the process corresponding to the memory coordination service to the waiting queue and puts it to sleep. When the sequence number on the timeline is updated to the current frame sequence number, the DRM kernel driver wakes up the process corresponding to the memory coordination service. The memory coordination service obtains the buffer according to the buffer ID and binds the DRM Prime handle of the buffer to the display layer without copying data. Within a set time after the VSync signal is triggered, the memory coordination service submits the DRM Prime handle to the display controller, which directly accesses the buffer through the handle and completes the display.

[0039] Furthermore, this invention performs differentiated hardware adaptation based on the image pixel format type. Specifically: for RGB formats, the DRM Prime handle is directly submitted to the display controller via the drmModeSetCrtc interface, and the hardware directly scans and outputs the data; for YUV formats, the GPU hardware synthesis unit processes the data and submits the display parameters using the DRM atomic mode, achieving hardware-level output without any format conversion throughout the process.

[0040] Example:

[0041] This embodiment employs an Android container application display method based on a cross-system zero-copy rendering link provided by the present invention. When running an Android container in the OpenHarmony system, for scenarios where the GPU does not support EGL Images, it achieves high-performance on-screen display of Android application rendering results on the OpenHarmony host side by constructing a cross-system unified memory pool, a hardware-level synchronization mechanism, and an adaptive rendering link. The specific process includes:

[0042] S1. Construct a cross-system memory sharing and negotiation framework to provide communication channels, capability matching rules and fault tolerance mechanisms for subsequent operations, ensuring consistency in the understanding and access of shared resources between the two ends.

[0043] S1.1 Cross-system communication channel construction: On the OpenHarmony side, a cross-system memory coordination service (OHMemoryCoordinator) is deployed, and a two-way encrypted communication channel is built based on the Distributed Hardware Framework and the Binder driver extension module of the Android container.

[0044] The channel adopts a frame synchronization protocol design, and each frame of transmitted data is encapsulated into a fixed structure: [frame header (8 bytes) + type identifier (4 bytes) + data length (4 bytes) + payload (variable length) + checksum (4 bytes)]. The frame header contains the protocol version and magic word (0xOHANDROID), and the checksum uses the CRC32 algorithm to verify data integrity.

[0045] The channel supports atomic transmission of key data such as buffer handles, format information, and synchronization signals, with a single transmission taking no more than 1ms, reducing interaction latency by 40% compared to traditional Binder communication;

[0046] The permission configuration explicitly declares the ohos.permission.DISTRIBUTED_DATASYNC distributed permission to avoid silent failures due to missing permissions, while ensuring communication security through process UID verification.

[0047] S1.2, Two-way capability handshake and negotiation.

[0048] On the Android container side, a memory proxy daemon (AndroidMemProxyDaemon) is started. It establishes a long connection with OHMemoryCoordinator through the lxc-attach extension interface of LXC and performs a three-round capability handshake process:

[0049] Basic capability exchange: The OpenHarmony side sends the supported memory types (ION or ASHMEM), the maximum number of buffers (10% to 15% of total memory), a list of pixel formats (more than 20, including OH_PIXEL_FORMAT_RGBA_8888, etc.), hardware synchronization capabilities (DRM Syncobj or Fence support status), and system version information; the Android side responds with the SurfaceFlinger buffer alignment requirements (such as 32-byte alignment of width and height), the rendering API (OpenGL ES 3.2 or Vulkan 1.1) format matrix, and the double or triple buffer support status;

[0050] Compatibility verification: Both parties perform cross-verification on the exchanged information. For example, OpenHarmony checks whether the ION version on the Android side is compatible with its own HDF memory service, and the Android side verifies whether the DRM driver version of OpenHarmony supports the Syncobj timeline mode.

[0051] Fault tolerance strategy negotiation: agree on a retry mechanism for handshake failure (maximum 3 times, 200ms interval), a communication timeout threshold (50ms), and an abnormal degradation scheme (such as automatically switching to a software fence when hardware synchronization is unavailable).

[0052] S1.3 Generation of cross-system capability matrix table.

[0053] Generate a structured matrix table based on the handshake results, and define the following core parameters:

[0054] Memory allocator: ION is the preferred choice (it has native support for both ends and DMA mapping efficiency is 30% higher than ASHMEM).

[0055] Format mapping rules: Establish a bidirectional mapping table (e.g., OH_PIXEL_FORMAT_RGBA_8888↔HAL_PIXEL_FORMAT_RGBA_8888, OH_PIXEL_FORMAT_YCBCR_420_888↔HAL_PIXEL_FORMAT_YV12), and mark the hardware acceleration support status of each format;

[0056] Synchronization mechanism: Hardware synchronization (timeline mode) is used when both ends support DRM Syncobj; otherwise, Fence software synchronization is used.

[0057] Buffer pool parameters: initial size (3, triple buffering mechanism), maximum size (4K), expansion threshold (80% occupancy), shrinkage threshold (when the application is running in the background);

[0058] Fault tolerance configuration: number of retransmissions due to communication timeout (3 times), buffer corruption recovery strategy (immediately allocate a spare buffer).

[0059] This embodiment adopts an atomic transmission design based on a frame synchronization protocol. Through a fixed frame structure and CRC32 check, it achieves zero packet loss in data transmission, reducing the interaction latency from 2.5ms in the traditional Binder to less than 1ms. A three-layer capability negotiation mechanism is set up, introducing compatibility verification and fault tolerance strategy negotiation to solve the problem of call failure caused by cross-system version differences, and the handshake success rate is improved to 99.5%. Permission pre-configuration and security verification, through distributed permission declaration and UID verification, eliminate the hidden danger of silent failure, and improve the link stability by 15%.

[0060] S2. Create a dynamic adaptive shared buffer pool. Build a dual-end compatible buffer pool on the OpenHarmony side to achieve dynamic resource scheduling and efficient metadata management.

[0061] S2.1 Creation of buffers with metadata. OHMemoryCoordinator calls the ION memory allocation interface (ion_alloc) to create an initial buffer pool. Each buffer consists of a metadata area and a data area:

[0062] Metadata area (128 bytes): Adopts a memory-aligned structure design, including format identifier (double-ended format ID), resolution (width and height), timestamp (accurate to nanoseconds), synchronization status (0 for not ready, 1 for rendering, 2 for displayable), checksum (CRC32 value of data area), and version number (to prevent inconsistency between double-ended metadata).

[0063] Data area parameters: Basic size = Initial resolution * Number of pixels in bytes * 1.2 (10% redundancy and 10% alignment reserve), for example, the buffer size of 1080P (1920*1080) RGBA_8888 format = 1920*1080*4*1.2=9953280 bytes;

[0064] Memory attributes: Set ION_FLAG_CACHED (CPU or GPU cache sharing), ION_FLAG_PROTECTED (prevent tampering), and ION_FLAG_DMA_BUF (cross-system DMA mapping), and configure access permissions via ion_set_security_context (only readable and writable by dual-end service processes);

[0065] Metadata access control: A read-write lock and version stamp mechanism is adopted. The version number is marked when updating on the OpenHarmony side, and the version is verified before reading on the Android side to avoid concurrent read-write conflicts.

[0066] S2.2, Dynamic scaling mechanism of the buffer pool.

[0067] Achieve intelligent scheduling based on application status and resolution:

[0068] Resolution adaptation: The resolution changes are monitored in real time through the onConfigurationChanged callback of the Android container. When a switch is detected (such as from 1080P to 2K): a buffer that matches the new resolution is immediately allocated; the old buffer is marked as available for the next low-resolution request; once the new buffer is ready, the switch is notified to the Android side through the frame synchronization protocol, and the switch delay is no more than 100ms.

[0069] State adaptation: When the application is running in the foreground, it maintains 3 buffers (triple buffering); when entering the background (triggered by the onPause callback): the pool shrinks to 1 minimum buffer (360P) within 100ms; the ION memory of the remaining buffers is released (by calling ion_free); the resolution parameters before shrinking are recorded; when the application resumes in the foreground (onResume), the pool is expanded within 100ms based on the recorded parameters.

[0070] Expansion strategy: When the buffer occupancy rate is greater than 80% (obtained through ion_query_heap_size), the buffer will be expanded by 1.5 times the current maximum resolution, with a maximum of no more than 15% of the system memory.

[0071] The following is an example of intelligent scheduling code:

[0072] / / Dynamic scaling implementation (OpenHarmony side)

[0073] void OnAppStateChanged(bool isForeground, uint32_t newWidth, uint32_tnewHeight) {

[0074] auto& pool = OHMemoryCoordinator::GetInstance().GetBufferPool();

[0075] if (isForeground) {

[0076] / / Foreground recovery: Expand to 3 buffers

[0077] while (pool.size() < 3) {

[0078] uint32_t size = newWidth * newHeight * GetBytesPerPixel()* 1.2;

[0079] auto buf = CreateBuffer(size, GetFormatMapping());

[0080] pool.push_back(buf);

[0081] }

[0082] } else {

[0083] / / Background collapse: retain a minimum buffer

[0084] while (pool.size() > 1) {

[0085] auto buf = pool.back();

[0086] ion_free(buf->ionFd);

[0087] pool.pop_back();

[0088] }

[0089] / / Reset to 360P minimum size

[0090] ResizeBuffer(pool[0], 640, 360);

[0091] }

[0092] }

[0093] S2.3 Buffer pool registration and handle conversion.

[0094] Registration process: Register the buffer pool to the display buffer manager (OHDisplayBufferManager), assign a globally unique globalBufferId (64 bits, including device ID and sequence number) to each buffer, and bind it to OpenHarmony's OHLayer. The binding adopts a delayed association strategy (binding on the first screen display to reduce initialization overhead).

[0095] Handle conversion: The ION buffer is converted into a DRM Prime handle by drmPrimeHandleToFD, and the lifecycle status of the handle (idle, in use or being released) is recorded to ensure that the handle is not reused or released prematurely.

[0096] This embodiment employs metadata embedding and secure access design, storing critical information directly in a buffer to eliminate three additional IPC interactions (saving 6 to 9 ms). The read-write lock mechanism ensures 100% metadata consistency. A three-level dynamic scaling strategy, combined with adaptive scheduling based on resolution, application status, and memory usage, improves memory utilization by 60% and reduces memory usage by 90% in the background. Delayed binding and handle management reduce resource consumption during the initialization phase, and handle lifecycle management reduces the DRM submission failure rate to below 0.1%.

[0097] The S3 and Android sides obtain the shared buffer and convert it into a rendering target, realizing dual-end format adaptation of the shared buffer and ensuring seamless writing to the Android graphics API.

[0098] S3.1 Buffer request and matching.

[0099] Request Initiation: When an Android application starts, its Surface requests a buffer from SurfaceFlinger through Surface::requestBuffer, carrying the parameters: currentWidth and currentHeight (current resolution), apiType (rendering API), and usage (rendering purpose).

[0100] Request forwarding: SurfaceFlinger forwards the request to AndroidMemProxyDaemon via Binder, which encapsulates it into a BufferRequestPacket (containing the application UID, request parameters, and checksum) and sends it to OHMemoryCoordinator via the frame synchronization protocol;

[0101] Buffer matching: OHMemoryCoordinator selects a match from the pool based on the request parameters. Matching priority: format conforms to the mapping table, size is not less than the request resolution, access latency is the lowest, and immediate allocation is triggered when there is no match (time is no more than 20ms).

[0102] S3.2 Buffer Information Transmission and Import.

[0103] Information encapsulation: OHMemoryCoordinator encapsulates the matched buffer information into a CrossBufferPacket, which includes ION fd, globalBufferId, DRM handle, metadata area offset, and memory attributes;

[0104] Transmission and verification: Transmission is carried out via frame synchronization protocol. AndroidMemProxyDaemon receives the data and verifies it using CRC32. If the verification fails, it requests a retransmission (up to 3 times).

[0105] ION Import: Call ion_import to import the ION fd of OpenHarmony into the Android side, generate buffer_handle_t, and verify memory attributes during import (ensure that ION_FLAG_DMA_BUF is set) to avoid importing illegal memory.

[0106] S3.3, Dual-end format adaptation and conversion.

[0107] Perform adaptation based on the rendering API type:

[0108] OpenGL ES adaptation:

[0109] Create a GraphicBuffer object and populate the following fields: width and height (requested values), format (Android format corresponding to the mapping table), usage (GRALLOC_USAGE_HW_TEXTURE, GRALLOC_USAGE_HW_RENDER, or GRALLOC_USAGE_EXTERNAL_DISP), handle (imported buffer_handle_t), and stride (actual span of the OpenHarmony buffer, ensuring 32-byte alignment).

[0110] Call eglCreateWindowSurface to bind the GraphicBuffer to an EGL surface. Before binding, verify the format compatibility using eglQuerySurface.

[0111] Vulkan compatibility:

[0112] Call vkGetMemoryFdPropertiesKHR to query the memory properties of the ION fd and ensure that VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT is supported;

[0113] Convert the ION fd to VkDeviceMemory using VkImportMemoryFdInfoKHR and set the memory offset to 128 bytes (skip the metadata area).

[0114] Create a VkImage object, set its format to the Vulkan format corresponding to the mapping table (e.g., VK_FORMAT_R8G8B8A8_UNORM), and set its layout to VK_IMAGE_TILING_OPTIMAL (matching the GPU texture layout).

[0115] Calling vkBindImageMemory to bind memory reduces GPU memory mapping by one step compared to the traditional approach (saving 2ms).

[0116] S3.4 Render target binding and notification.

[0117] AndroidMemProxyDaemon returns the GraphicBuffer to the application Surface via the SurfaceFlinger::onBufferAvailable callback. After the application's rendering context (EGLContext or VkDevice) binds the buffer, it sends a binding completion notification to OpenHarmony via the frame synchronization protocol, which includes the globalBufferId and the binding timestamp.

[0118] This embodiment establishes a multi-dimensional buffer matching strategy, combining priority matching of format, size, and performance, thereby increasing the buffer reuse rate to 85%; it constructs a Vulkan-specific adaptation process, directly importing ION memory into Vulkan device memory, eliminating intermediate conversion steps, and reducing rendering preparation time by 2ms; it proposes a transmission verification and retransmission mechanism, using CRC32 verification and 3 retransmissions to achieve a 100% success rate in buffer information transmission.

[0119] S4 and Android applications render and trigger hardware-level synchronization, enabling rendering data to be directly written to a shared buffer, and ensuring data consistency through hardware synchronization.

[0120] S4.1, API-based rendering process.

[0121] OpenGL ES rendering:

[0122] The application executes drawing instructions through glDrawArrays or glDrawElements, and the GPU writes directly to the data area of ​​the shared buffer.

[0123] Call eglSwapBuffers to trigger the EGL driver to execute glFinish, ensuring that all GPU instructions are completed (with a delay of no more than 1ms).

[0124] The rendering completion timestamp is obtained through eglQuerySurface and written to the buffer metadata area.

[0125] Vulkan rendering:

[0126] The application builds a command buffer containing drawing operations, specifying a shared VkImage as the output target;

[0127] Submit the command queue via vkQueueSubmit and set VkSemaphore to the rendering completion signal;

[0128] Call vkWaitForFences to wait for rendering to complete, then obtain the precise timestamp and write it to the metadata area.

[0129] S4.2 Hardware-level synchronization signal generation.

[0130] Signals are generated based on a negotiation-based synchronization mechanism.

[0131] DRM Syncobj hardware synchronization:

[0132] AndroidMemProxyDaemon opens the DRM device ( / dev / dri / card0) and calls drmSyncobjCreate to create a Syncobj in timeline mode (setting the DRM_SYNCOBJ_CREATE_TIMELINE flag).

[0133] Call drmSyncobjImportFence to import the GPU rendering fence into Syncobj's timeline (specifying the time point seq=currentFrameNum).

[0134] Obtain the Syncobj handle through drmSyncobjGetHandle, bind it to globalBufferId and the current frame sequence number, and update the synchronization status of the buffer metadata to 2, indicating that it can be displayed;

[0135] Syncobj lifecycle management: Create a new timeline node for each frame, and call drmSyncobjReset to reset it after the screen is finished to avoid resource leaks.

[0136] Fence software synchronization:

[0137] The android::Fence::merge method merges the GPU rendering fence and the buffer-released fence to generate a composite fence; fenceDup is called to copy the fence file descriptor to ensure cross-process access validity.

[0138] S4.3, Rendering completion notification transmission.

[0139] AndroidMemProxyDaemon encapsulates RenderCompletePacket (including globalBufferId, Syncobj handle or Fence fd, frame timestamp, and metadata checksum), sends it to OHMemoryCoordinator via the frame synchronization protocol, and marks the buffer as ready to be displayed on screen after sending, preventing repeated writing.

[0140] This embodiment establishes a DRM Syncobj timeline synchronization mechanism. Compared with the traditional Fence, the synchronization latency is reduced from 5-8ms to 1-2ms, the frame stuttering rate is reduced from 12% to 0.5%, and multi-frame concurrent synchronization is supported. By binding rendering and synchronization atoms, metadata status markers are bound to Syncobj frame sequence numbers, ensuring that the OpenHarmony side only reads complete rendering data and eliminating screen tearing. The rendering process is optimized according to the API, and the synchronization point design is optimized for the asynchronous characteristics of Vulkan, further reducing the rendering preparation time by 1ms.

[0141] The S5 and OpenHarmony sides directly display the data and complete the buffer reclamation, achieving zero-copy display and closed-loop buffer reclamation, ensuring that the display and refresh rhythms are aligned.

[0142] S5.1, Synchronization signal waiting and verification.

[0143] After receiving the RenderCompletePacket, OHMemoryCoordinator executes:

[0144] Verification process: Verify whether the metadata checksum is consistent with the CRC32 of the data area. If they are inconsistent, request the Android side to retransmit the rendering result (at most once).

[0145] Synchronous waiting:

[0146] DRM Syncobj mode: Call drmSyncobjWait to wait for the timeline node seq=currentFrameNum, with the timeout set to 1 VSync cycle (16.6ms). During the wait, display composition is blocked but buffer allocation is not blocked.

[0147] Fence mode: Call fence_wait to wait for a signal, while listening for the SIGPIPE signal. When triggered, it automatically switches to the backup buffer.

[0148] Post-synchronization processing: After successful synchronization, update the synchronization status of the metadata to 3 to indicate that it has been displayed; if it fails, mark the buffer as corrupt and trigger a health check.

[0149] S5.2 Zero-copy screen loading process.

[0150] Buffer binding: Call OHDisplayBufferManager::GetBuffer(globalBufferId) to get the buffer, and bind it to the display layer through OHLayer::SetBuffer. The binding process only passes the handle and does not copy the data.

[0151] Formatting adaptation:

[0152] RGB format (e.g., RGBA_8888): Directly call drmModeSetCrtc to submit the DRM Prime handle to the display controller (LCD driver), and perform hardware scan output;

[0153] YUV format (such as YCBCR_420_888): It is processed directly by the GPU hardware synthesis unit (HDMI TX module), and DRM_MODE_ATOMIC atomic mode is enabled for submission to avoid format conversion.

[0154] VSync Alignment: Obtain the VSync signal through ohos::display::VSyncManager, and complete DRM submission within 1ms after the signal is triggered to ensure that frame display is synchronized with screen refresh.

[0155] S5.3, Buffer Recovery and Signal Feedback.

[0156] Recycling Triggered: After the page is displayed (triggered by the drmModePageFlip callback), OHMemoryCoordinator executes:

[0157] Generate a release signal (including globalBufferId, release Syncobj or Fence, and screen upload timestamp).

[0158] Move the buffer from the used queue to the reuse queue and reset the timestamp and synchronization status of the metadata.

[0159] Feedback transmission: Send a release signal to AndroidMemProxyDaemon via the frame synchronization protocol. If the transmission fails, start a timed retransmission (10ms interval, up to 3 times).

[0160] Abnormal recovery: If the screen loading fails, the health check in step 6 will be triggered immediately, and a spare buffer will be allocated to replace it.

[0161] This embodiment establishes an end-to-end zero-copy link, from writing to the Android GPU to output from the OpenHarmony display controller, with no CPU copying throughout (the traditional solution involves 2 copies taking 10ms), reducing display latency by 30 to 50ms; through the VSync hard alignment mechanism, the time difference between composition submission and screen refresh is controlled within 1ms, narrowing the frame rate fluctuation range; through fast fault switching, a backup buffer is activated within 10ms when the screen fails to load, and the display interruption time is no more than 1 frame (16.6ms).

[0162] S6, with its intelligent buffer pool reuse and health monitoring, ensures long-term stable operation of the link through reuse optimization and anomaly handling.

[0163] S6.1 Buffer reuse chain mechanism.

[0164] AndroidMemProxyDaemon executes the following after receiving the release signal:

[0165] State update: Move the buffer from the queue to the queue to the queue to be used, retain the GraphicBuffer object (only reset the internal state fields), and avoid the 5ms overhead of recreating it;

[0166] Reuse matching: When the application makes the next request, it will prioritize matching the available queue buffers with a size not less than the request resolution and the same format. If the match is successful, it will return directly (reuse efficiency is improved by 80%).

[0167] Reuse chain management: Maintain a mapping table from resolution to buffer, record the most recently used buffer for each resolution, and achieve a reuse rate of 95% when the same resolution is requested again.

[0168] S6.2, Dimensional Health Detection.

[0169] OHMemoryCoordinator performs a check every 30 seconds and responds to exceptions (such as screen loading failure):

[0170] Data integrity check:

[0171] Read the checksum of the buffer metadata and compare it with the CRC32 recalculated in the data area;

[0172] Check if the metadata version number is consistent with the agreement between the two ends; if they are inconsistent, mark it as unavailable.

[0173] Performance health testing:

[0174] Calculate the average access latency (time from request to binding) for each buffer; if it exceeds 20ms, reduce its reuse priority.

[0175] Record the number of times the memory is reused. When the number of reuses reaches 1000, the memory is forcibly reclaimed (by calling ion_free) and reallocated to avoid memory fragmentation.

[0176] Resource leak detection: Query ION memory usage (ion_query_heap_size). If the deviation from the pool management record is greater than 10%, traverse the buffer and release the exception handle.

[0177] S6.3, Abnormal Recovery Strategy.

[0178] Execute when an anomaly is detected:

[0179] Buffer corruption: Immediately allocate a new buffer to replace it, and add the corrupted buffer to the recycling queue (delayed for 5 seconds to release, to avoid data read and write conflicts);

[0180] Performance degradation: When the latency of at least two buffers exceeds the limit, automatically expand one buffer to distribute the load;

[0181] Memory leak: Triggers ION memory cleanup (ion_heap_collect), releasing memory blocks that are not associated with globalBufferId.

[0182] This embodiment designs a reuse chain and mapping table. By retaining the GraphicBuffer and resolution mapping, the reuse efficiency is improved by 80%, and the rendering preparation time is reduced by 5 to 10 ms. A three-dimensional health detection system is established, covering three types of indicators: integrity, performance, and resources. The link stability is improved by 40%, and it runs without failure for a long time (72 hours). It has a graded anomaly recovery system, which quickly replaces damaged components, expands capacity when performance degrades, and actively cleans up leaks. The anomaly handling time is no more than 50 ms.

[0183] Experimental verification shows that the performance and compatibility improvements of this embodiment compared to existing technologies are mainly reflected in the following aspects: display latency is reduced from an average of 80ms to below 30ms (a reduction of 62.5%), reaching the level of native applications; the fluctuation range is narrowed from ±5fps to ±1fps, resulting in smooth video playback and game scenes; the dynamic scaling mechanism reduces memory usage by 90% when applications are in the background and reduces memory conflicts by 70% when multiple applications are running concurrently; it does not rely on EGL Images and is compatible with most GPU models; the resolution switching response time is reduced from 500ms to less than 100ms, with no screen interruption during the switching process; it runs continuously for 72 hours without failure, with a rendering link failure rate of less than 0.5% (compared to 15% in existing solutions).

[0184] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for displaying Android container applications based on a cross-system zero-copy rendering link, characterized in that, Specifically, the following steps are included: Using HarmonyOS as the host and Android as the container, the host starts a memory coordination service to transmit data with the first frame structure between the host and the container; the container starts a memory proxy daemon process to establish a connection with the memory coordination service, complete compatibility verification, and generate a cross-system capability matrix table. The memory coordination service creates a shared buffer pool based on the cross-system capability matrix table. The buffer consists of a metadata area and a data area. The data area stores image pixel data, and the metadata area stores relevant parameters of the image pixel data. The service allocates a HarmonyOS file descriptor (FD) and a buffer ID to the buffer, binds them to the HarmonyOS layer, and converts them into DRM Prime handles. When an Android application starts, it sends a buffer request. SurfaceFlinger forwards it to the memory proxy daemon. The memory proxy daemon encapsulates the buffer request into a buffer request packet with a first-frame structure and sends it to the memory coordination service. The memory coordination service obtains the target buffer from the shared buffer pool based on the buffer request packet, and encapsulates the corresponding HarmonyFD, buffer ID, DRM Prime handle, metadata area offset, and memory attributes into a cross-buffer packet with a first-frame structure and sends it to the memory proxy daemon. The memory proxy daemon imports Harmony FD to obtain the first buffer handle, creates an image buffer and binds it to the rendering context of the Android application, and sends a binding completion notification to the host. The Android application executes drawing instructions to write image pixel data into the image buffer. The memory proxy daemon creates a synchronization object in timeline mode, imports the GPU rendering completion fence to obtain the synchronization object handle and writes it into the metadata area, and sends a rendering completion packet containing the current frame number to the host. The memory coordination service obtains the current frame number, and when the sequence number on the timeline is updated to the current frame number, it obtains and submits the buffer to complete the display.

2. The Android container application display method according to claim 1, characterized in that, The shared buffer pool dynamically expands and contracts based on the application status and resolution. Specifically, when the resolution is switched, a new resolution buffer is allocated, the old buffer is marked as available for transition, and the container side is notified to switch when the new buffer is ready. When the Android application is running in the foreground, it maintains M buffers. When it enters the background, it shrinks to N minimum buffers and releases excess memory. When it resumes the foreground, it expands, where M is greater than N.

3. The Android container application display method according to claim 1, characterized in that, The memory coordination service obtains the target buffer from the shared buffer pool based on the buffer request packet in the following way: it prioritizes the buffer whose format conforms to the double-ended mapping relationship, whose size is not less than the requested resolution, and whose access latency is the lowest. If no buffer meets the conditions, it triggers immediate allocation.

4. The Android container application display method according to claim 1, characterized in that, When the hardware does not support DRM hardware synchronization, the Fence mechanism is used to achieve software synchronization. Specifically, the GPU rendering completion fence and the buffer release fence are merged into a composite fence, the fence file descriptor is copied, and then the memory agent daemon sends a rendering completion packet containing the buffer ID, fence file descriptor, frame timestamp, and metadata checksum to the host.

5. The Android container application display method according to claim 1, characterized in that, When the container's rendering API is Vulkan, Vulkan checks the validity of the Harmony FD. If valid, it converts the Harmony FD into a Vulkan-specific GPU memory object, sets the memory offset of the GPU memory object to a set number of bytes to skip the metadata area, builds the rendering command for the Android application, submits the rendering command to the GPU for rendering, writes the rendered image pixel data to the shared memory corresponding to the GPU memory object, and records the rendering completion timestamp after rendering is complete.

6. The Android container application display method according to claim 1, characterized in that, The shared buffer pool is a memory region shared by the CPU and GPU, tamper-proof, and supporting cross-system DMA mapping, and is only accessible to the memory coordination service and memory agent daemon.

7. The Android container application display method according to claim 1, characterized in that, After receiving the cross-buffer packet, the memory proxy daemon verifies the cross-buffer packet. If the verification is successful, it imports the HarmonyFD of the target buffer into the container side, obtains the buffer handle of the Android system and records it as the first buffer handle, and verifies the legality of the memory attributes.

8. The Android container application display method according to claim 1, characterized in that, The memory coordination service obtains the current frame sequence number as follows: After receiving the rendering completion package, the memory coordination service recalculates the checksum of the data area corresponding to the buffer ID, and verifies whether the checksum is the same as the metadata checksum in the rendering completion package. If they are different, it requests the container to retransmit the rendering result; if they are the same, it obtains the current frame sequence number.

9. The Android container application display method according to claim 1, characterized in that, The method for acquiring and submitting the buffer to complete the display when the sequence number on the timeline is updated to the current frame sequence number is as follows: The DRM kernel driver adds the corresponding process of the memory coordination service to the waiting queue and puts it to sleep. When the sequence number on the timeline is updated to the current frame sequence number, the DRM kernel driver wakes up the corresponding process of the memory coordination service. The memory coordination service acquires the buffer according to the buffer ID, binds the DRM Prime handle of the buffer to the display layer, and within a set time after the VSync signal is triggered, the memory coordination service submits the DRM Prime handle to the display controller. The display controller then directly accesses the buffer through this handle and completes the display.

10. The Android container application display method according to claim 9, characterized in that, When the image pixel format is RGB, the DRM Prime handle is submitted to the display controller for direct hardware scanning and output; when it is YUV, it is processed by the GPU hardware synthesis unit, and the display parameters are submitted using DRM atomic mode.

Images