A Linux container application display method based on a virtual synthetic node in a Hongmeng platform

Abstract

This invention discloses a method for displaying Linux container applications on the HarmonyOS platform based on virtual compositing nodes. Using HarmonyOS as the host and Linux as the container, the host-side Vulkan resource management hub pre-checks GPU characteristics, allocates shared Vulkan resources, creates rendering channels, and initializes global timeline semaphores to build a unified resource pool. After the container starts, it requests resources from the host to import shared video memory objects, and the container compositor registers virtual nodes. During the rendering phase, the application uses shared resources to complete parallel rendering and sends notifications to the container compositor. The container compositor, based on the global display state, uses Vulkan instructions to perform window blending, transformation, and other processing, encapsulates global compositing units, and submits them to the HarmonyOS global compositing scheduler. The scheduler reads the global compositing units and, when the timeline semaphore meets the on-screen conditions, completes the global compositing of the container application window and the host's native window, driving the display service to output the screen.

Description

Technical Field

[0001] This invention belongs to the field of computer software development technology, specifically relating to a method for displaying Linux container applications on the HarmonyOS platform based on virtual composite nodes. Background Technology

[0002] When running a Linux system in a container on OpenHarmony, the existing graphics display pipeline suffers from significant performance bottlenecks due to disordered process sequence and fragmented stages. The main problems are as follows: First, resources and rendering are disconnected. Traditional solutions start application rendering first, then temporarily request cross-domain resources, resulting in a delay of over 10ms while waiting for resource allocation during rendering. Second, compositing and display are misaligned. After compositing within the container, raw data is transmitted directly without verifying resource validity, leading to format incompatibility requiring re-conversion and incurring a loss of 15 to 20ms. Third, state adaptation is lagging. Global states (such as refresh rate changes) do not take effect in real time during rendering or compositing, resulting in a mismatch between the compositing result and display requirements, with invalid resources accounting for over 30%. Fourth, there is no coordination across the entire pipeline. Rendering, compositing, and display all run independently without unified scheduling, resulting in a screen tearing rate exceeding 5% and accumulated latency exceeding 30ms. Summary of the Invention

[0003] In view of this, the present invention provides a method for displaying Linux container applications on the HarmonyOS platform based on virtual synthesis nodes, which realizes low-latency and high-fidelity display of Linux applications in containers on the HarmonyOS platform, and is suitable for high-performance graphics interaction scenarios such as 3D modeling, real-time video editing, and high frame rate games.

[0004] This invention provides a method for displaying Linux container applications on the HarmonyOS platform based on virtual synthetic nodes, specifically including the following steps:

[0005] Using HarmonyOS as the host and Linux as the container, the host starts a Vulkan resource management hub to obtain supported Vulkan features, pre-allocates shared Vulkan resources for different Linux application types, pre-creates rendering channels and binds them to the frame buffer, initializes global timeline semaphores, and forms a resource pool by the ID, type, file descriptor FD, rendering channel, frame buffer and timeline semaphore of the shared Vulkan resources;

[0006] After receiving a resource request from the container's Vulkan driver, the Vulkan resource management hub selects a shared Vulkan resource from the resource pool and sends the shared Vulkan resource's ID and file descriptor (FD) to the container. The container's Vulkan driver imports the FD to obtain a Vulkan memory object, binds it to the resource within the container, and then submits a node registration request to the host's synthesis node management service to obtain the virtual node ID assigned to it by the host's synthesis node management service.

[0007] When a Linux application creates an image resource, the Vulkan driver binds the corresponding shared Vulkan resource to the application rendering context, creates a rendering process using the corresponding rendering channel, and the fragment shader executes multiple rendering targets in parallel, writing color values, depth values, and normal vectors into different channels of the shared Vulkan resource. After rendering is completed, the global timeline semaphore is updated, the obtained image pixel data is written into the shared Vulkan resource, and a rendering completion notification is sent to trigger the container compositor to obtain the global display state of HarmonyOS through the virtual node ID. The compositor then performs window blending, rotation, and scaling in the corresponding framebuffer to complete the compositing, and submits the global compositing unit containing the shared Vulkan resource ID, window display parameters, and compositing progress markers to the HarmonyOS global compositing scheduler.

[0008] The HarmonyOS global synthesis scheduler updates the resource reference count, notifies the Vulkan resource management center to record the mapping between shared Vulkan resources and the on-screen window, and completes the global synthesis of the host native window and the container application window when the global timeline semaphore is satisfied. The display service is driven by the global timeline semaphore to complete the on-screen display.

[0009] Furthermore, the container compositor dynamically adjusts the composing cycle based on the system's global refresh rate and reduces the composing frequency of background windows based on the window's focus state.

[0010] Furthermore, the HarmonyOS global compositing scheduler extracts the attributes of the external memory handle in the global compositing unit and verifies whether the attributes match the formats supported by the HarmonyOS GPU.

[0011] Furthermore, the HarmonyOS global synthesis scheduler queries the current layout of the shared Vulkan resources, compares the current layout with the layout required for the screen, and if they match, the verification is passed; otherwise, GPU instructions are used to accelerate the layout conversion.

[0012] Furthermore, the HarmonyOS global synthesis scheduler extracts the permission tokens bound in the global synthesis unit and verifies whether the permission tokens have not expired and whether the operation type is screen-ready.

[0013] Furthermore, when the container and the host use different GPUs, the queue families of the GPUs used by the container and the host are obtained respectively. The queue family that supports both graphics rendering and data transfer is selected, and the shared Vulkan resources and corresponding memory handles are allocated to the memory pool corresponding to the queue family. For shared Vulkan resources that do not support external memory export, asynchronous image copying is performed through the queue family to complete the format conversion. After the asynchronous image copy is completed, the resource transfer is marked as complete and set to the ready state by updating the timeline semaphore.

[0014] Furthermore, the resource request includes Linux application type and Vulkan resource requirements.

[0015] Furthermore, the rendering channel is a chain of sub-channels formed by sub-channels.

[0016] Furthermore, after the fragment shader performs multi-target rendering in parallel and writes the color value, depth value, and normal vector into different channels of the shared Vulkan resource, the sub-channel reads the attachment channel of the shared Vulkan resource through the input attachment to complete the rendering of the corresponding stage. After the rendering of the corresponding stage of the sub-channel is completed, the global timeline semaphore is updated, and the rendered image pixel data is written into the memory area corresponding to the shared Vulkan resource.

[0017] Furthermore, the registration request includes a container ID and compositing capabilities, wherein the compositing capabilities include support for high dynamic range, the maximum number of windows that can be opened simultaneously, and supported resolutions and formats.

[0018] Beneficial effects:

[0019] This invention uses HarmonyOS as the host and Linux as the container. The host-side Vulkan resource management hub pre-completes GPU characteristic checks, shared Vulkan resource allocation, rendering channel creation, and global timeline semaphore initialization to build a unified resource pool. After the container starts, it requests resources from the host and imports shared video memory objects. The container compositor completes virtual node registration. During the rendering phase, the Linux application uses shared resources to perform MRT parallel rendering. Sub-channels complete multi-stage rendering by inputting attachments. After the frame rendering is complete, a notification is sent to the container compositor. The container compositor uses Vulkan instructions to perform window blending, transformation, and other processing based on the global display state. It encapsulates global compositing units and submits them to the HarmonyOS global compositing scheduler. The scheduler performs legality, resource status, and permission verification on the global compositing units, maintains resource reference counts and mapping relationships, and completes global compositing of the container application window and the host native window when the timeline semaphore meets the on-screen conditions, ultimately driving the display service to output the screen. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a method for displaying Linux container applications on the HarmonyOS platform based on virtual synthetic nodes, provided by the present invention. Detailed Implementation

[0021] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0022] This invention provides a method for displaying Linux container applications on the HarmonyOS platform based on virtual compositing nodes. The core idea is as follows: using HarmonyOS as the host and Linux as the container, the host-side Vulkan resource management hub pre-completes GPU characteristic checks, shared Vulkan resource allocation, rendering channel creation, and global timeline semaphore initialization to build a unified resource pool. After the container starts, it requests resources from the host and imports shared video memory objects; the container compositor registers virtual nodes. During the rendering phase, the Linux application uses shared resources to perform MRT parallel rendering, and sub-channels complete multi-stage rendering through input attachments. After frame rendering is complete, a notification is sent to the container compositor. The container compositor uses Vulkan instructions to perform window blending, transformation, and other processing based on the global display state, encapsulates global compositing units, and submits them to the HarmonyOS global compositing scheduler. The scheduler verifies the legality, resource status, and permissions of the global compositing units, maintains resource reference counts and mapping relationships, and completes global compositing of the container application window and the host's native window when the timeline semaphore meets the on-screen conditions, ultimately driving the display service to output the screen.

[0023] This invention provides a method for displaying Linux container applications on the HarmonyOS platform based on virtual composite nodes. The processing flow is as follows: Figure 1 As shown, the specific steps include:

[0024] Step 1: Using HarmonyOS as the host and Linux as the container, start the Vulkan resource management hub within the host. The Vulkan resource management hub checks the Vulkan features supported by the GPU, loads the first mapping describing the correspondence between Linux application types and Vulkan resources, pre-allocates shared Vulkan resources corresponding to different Linux application types in the first mapping, pre-creates rendering channels and binds them to the frame buffer. The rendering channel is a chain of sub-channels formed by sub-channels. A global timeline semaphore with an initial value of 0 is created to mark the application's display execution progress. A resource pool is formed by the shared Vulkan resource ID, shared Vulkan resource type, shared Vulkan resource file descriptor FD, the rendering channel containing the sub-channel chain, the frame buffer, and the timeline semaphore.

[0025] After the container starts, the Vulkan driver inside the container sends a resource request containing the Linux application type and Vulkan resource requirements to the Vulkan resource management hub; the Vulkan resource management hub selects the corresponding shared Vulkan resource from the resource pool according to the resource request, and sends the shared Vulkan resource ID and shared Vulkan resource FD to the container;

[0026] The container's Vulkan driver imports the shared Vulkan resource FD into a Vulkan memory object usable by the container, completing the resource binding within the container. The container synthesizer submits a node registration request to the host synthesizer node management service. The registration request includes the container ID and synthesizer capabilities. After receiving the node registration request, the host synthesizer node management service assigns a virtual node ID to the container synthesizer.

[0027] Compositing capabilities include whether it supports High Dynamic Range (HDR), the maximum number of windows that can be opened, supported resolutions, and formats.

[0028] Step 2: Inside the container, when the Linux application starts and creates image resources using a shared Vulkan resource ID, the Vulkan driver binds the shared Vulkan resource corresponding to the shared Vulkan resource ID to the application's rendering context, and completes the rendering process using the rendering channel corresponding to the shared Vulkan resource ID.

[0029] Within the container, fragment shaders execute multiple render target (MRT) rendering in parallel, simultaneously outputting color values, depth values, and normal vectors, and writing them to different channels of the shared Vulkan resource. Sub-channels read the attachment channels of the shared Vulkan resource through input attachments to complete the rendering of their corresponding stages. After completing the rendering of the corresponding stage of the sub-channel, the global timeline semaphore is updated, and the rendered image pixel data is written to the memory area corresponding to the shared Vulkan resource. After the current frame is rendered, the rendering completion semaphore is triggered, sending a rendering completion notification containing the shared Vulkan resource ID and rendering timestamp to the container compositor, and initiating the window compositing process.

[0030] Step 3: After receiving the rendering completion notification, the container compositor reads the shared Vulkan resource ID and rendering timestamp. Using its own virtual node ID, it obtains the global display status of HarmonyOS from the compositor management service within the host. The global display status includes the current display refresh rate, global window hierarchy table, screen split ratio, etc. The resource status obtained from the shared Vulkan resource ID includes image layout, resolution, etc.

[0031] Based on the global display state, the container compositor uses Vulkan instructions to perform window blending, rotation, and scaling in the framebuffer corresponding to the shared Vulkan resource ID. After the compositing is completed, a global compositing unit containing the shared Vulkan resource ID, window display parameters, and compositing progress markers will be submitted to the HarmonyOS global compositing scheduler with a set delay.

[0032] Furthermore, the container synthesizer dynamically adjusts the synthesis cycle based on the system's global refresh rate and reduces the synthesis frequency of background windows according to the window focus state, thereby achieving dynamic optimization of synthesis load and power consumption.

[0033] Step 4: After receiving the global synthesis unit, the HarmonyOS global synthesis scheduler sequentially completes the external memory handle validity verification, Vulkan resource layout status check and automatic conversion, and permission token validity verification. After successful verification, it updates the resource reference counts of the container and the host respectively, and notifies the Vulkan resource management center to record the mapping relationship between shared Vulkan resources and the on-screen window. When the Linux application closes the window, the synthesis node management service in the host notifies the Vulkan resource management center to decrement the reference count. When the reference count is 0, the shared Vulkan resources are released.

[0034] Furthermore, the external memory handle validity verification method is as follows: the HarmonyOS global synthesis scheduler extracts the attributes of the external memory handle in the global synthesis unit and verifies whether the attribute matches the format supported by the HarmonyOS GPU.

[0035] Furthermore, the resource status consistency check is performed as follows: the HarmonyOS global synthesis scheduler queries the current layout of the shared Vulkan resources, compares the current layout with the layout required for the screen display, and if they match, the verification is passed; otherwise, GPU instructions are used to accelerate the layout conversion.

[0036] Furthermore, the permission token verification method is as follows: the HarmonyOS global synthesis scheduler extracts the permission token bound in the global synthesis unit and verifies whether the permission token has not expired and whether the operation type is screenable.

[0037] Furthermore, when the container and the host use different GPUs, this invention obtains the queue families of the GPUs used by the container and the host respectively, selects the queue family that supports both graphics rendering and data transmission, and allocates the shared Vulkan resources and corresponding memory handles to the memory pool corresponding to the queue family to reduce cross-queue synchronization overhead. For shared Vulkan resources that do not support external memory export, asynchronous image copying is performed through the queue family to complete the format conversion. After the asynchronous image copy is completed, the resource transmission is marked as complete and set to the ready state by updating the timeline semaphore.

[0038] Step 5: When the global timeline semaphore meets the on-screen conditions, the HarmonyOS global compositing scheduler completes the global compositing of the host native window and the container application window in the shared Vulkan resources in the GPU to obtain the final image. The display service is then driven by the global timeline semaphore to output the final image to the screen.

[0039] Example:

[0040] This embodiment employs a method for displaying Linux container applications on the Harmony platform based on virtual compositing nodes, provided by this invention. Targeting the graphical display scenario of Linux containers in the OpenHarmony system, it reconstructs the technical chain according to the process of rendering, compositing, verification, and then displaying on screen. It transforms the Wayland compositor within the container into an OpenHarmony virtual compositing node, deeply reusing advanced Vulkan features to achieve the display of Linux applications within the container on the OpenHarmony side. The specific process includes:

[0041] S1. Building a cross-domain resource pool that supports deep Vulkan features provides on-demand cross-domain Vulkan resources for Linux application rendering, avoiding waiting during rendering. This is the foundation for all subsequent steps. Specifically, this includes:

[0042] S1.1 Deploy the Vulkan Advanced Resource Management Hub (V-ARM) on the OpenHarmony side. After V-ARM starts, it will prioritize completing three core tasks:

[0043] Feature-aware resource pre-allocation: Detects Vulkan features supported by the OpenHarmony-side GPU (such as VK_KHR_multiview, VK_EXTERNAL_MEMORY_HANDLE_TYPE_OPAQUE_FD_BIT), and pre-allocates dedicated VkImages according to a pre-defined application-resource mapping table (such as multi-channel resources for 3D applications and YCbCr format resources for video applications).

[0044] Assign a 3-channel MRT-VkImage (color: VK_FORMAT_R8G8B8A8_UNORM, depth: VK_FORMAT_D32_SFLOAT, normal: VK_FORMAT_R16G16B16A16_SFLOAT) to the 3D application.

[0045] Assign a hardware-accelerated YCbCr-VkImage (VK_FORMAT_G8B8G8R8_422_UNORM) to video applications.

[0046] All resources generate Vulkan standard external memory handles (in file descriptor form) to ensure cross-GPU architecture (ARM Mali / Intel Xe) compatibility.

[0047] Sub-pass resource pooling: Pre-create VkRenderPasses with fixed dependencies (such as sub-pass chains from geometry rendering, lighting calculation to post-processing) and bind them to dedicated VkFramebuffers to reduce the overhead of application creating RenderPasses (saving 2 to 3ms per frame).

[0048] Timeline semaphore initialization: Create a global timeline semaphore (VkSemaphoreType_TIMELINE) with an initial value of 0. This semaphore is used to mark the progress of subsequent end-to-end stages (e.g., value 1 indicates rendering is complete, value 3 indicates compositing is complete, and value 5 indicates on-screen display is complete).

[0049] S1.2 Resource negotiation and virtual node pre-registration during Linux container initialization: After the Linux container starts, resource docking is completed first instead of starting the application directly.

[0050] Resource requirement reporting: The Vulkan driver (Mesa driver) inside the container reports the application type (such as Blender 3D modeling) and resource requirements (such as 10 1920x1080 MRT-VkImages) to V-ARM through the virtio-vsock cross-domain channel.

[0051] Resource matching and handle transfer: V-ARM selects resources from the pre-allocation pool according to the reported requirements, and passes the resource list (including CGUID) and external memory handle to the container driver through memfd cross-domain mapping. The container driver completes resource binding through vkImportMemoryFdKHR.

[0052] Virtual node pre-registration: The Wayland compositor (V-Weston) inside the container synchronously submits a registration request to the OpenHarmony Compositing Node Management Service (CNS), carrying the container UUID and compositing capabilities (such as HDR support, maximum number of windows). The CNS pre-assigns virtual node IDs to prepare for subsequent compositing.

[0053] S2, a Linux application, leverages advanced Vulkan features to perform efficient rendering (core image generation). Based on provided cross-domain resources, it directly generates reusable rendering results across domains, serving as the source of on-screen data. Specifically, this includes:

[0054] S2.1 Application access to cross-domain Vulkan resources.

[0055] After Linux applications (such as Blender and Kdenlive video editors) start, they connect to resources in the following ways:

[0056] When calling vkCreateImage to create an image resource, the external resource identifier is specified as the returned CGUID in VkImageCreateInfo. After the container driver recognizes the identifier, it directly binds the shared VkImage allocated by V-ARM to the application rendering context without creating a new resource in the container's private memory.

[0057] When calling vkCreateRenderPass, the pre-created sub-pass chain can be reused directly through the extended interface vkCreateRenderPassWithPredefinedKHR, avoiding the time-consuming operation of repeatedly building RenderPass.

[0058] S2.2 Optimize the rendering process using advanced Vulkan features.

[0059] The application improves rendering efficiency through features such as MRT and sub-passes. The specific process is as follows:

[0060] MRT Parallel Rendering: Specify three color attachments (corresponding to the three channels of MRT) in VkPipelineRenderingCreateInfo. The fragment shader simultaneously outputs color values, depth values, and normal vectors to different channels of the shared VkImage, reducing drawing commands by 40% compared to traditional single-channel multi-round rendering.

[0061] Seamless subpass switching: Based on the pre-created subpass chain, the application does not need to explicitly switch frame buffers. After completing geometry rendering (subpass 0), it automatically enters lighting calculation (subpass 1). Through the frame buffer sharing feature between subpasses, resource binding overhead is reduced (saving 2ms per frame).

[0062] Stage progress marker: After each sub-pass is completed, the global timeline semaphore is updated via vkSignalSemaphore (e.g., set to 1 when sub-pass 0 is completed, and set to 2 when all rendering is completed), so that subsequent stages can perceive the rendering progress in real time.

[0063] S2.3, Rendering results are directly written to shared resources.

[0064] After the rendering command (vkCmdDraw) is submitted to the container's Vulkan logical device, the device directly writes the rendering result to the allocated shared VkImage memory area (since this area has been mapped across domains, there is no need to copy it).

[0065] After full-frame rendering is complete, the application triggers R-Sem (rendering complete semaphore) and sends a rendering complete notification (including CGUID and timestamp) to V-Weston via the Wayland protocol, triggering subsequent compositing.

[0066] S3, within the container, V-Weston acts as a virtual compositing node, generating global compositing units (window data processing). It receives rendering results and processes them into GCUs that can be directly used by OpenHarmony global compositing, serving as a crucial bridge connecting rendering and on-screen display.

[0067] S3.1, V-Weston receives the rendering results and obtains the global state.

[0068] V-Weston synchronizes data and status in the following ways:

[0069] Receive the rendering completion notification sent by the application, extract the CGUID and rendering timestamp, and obtain the OpenHarmony global state (such as the current display refresh rate of 60Hz, global window hierarchy table, and split-screen ratio of 1:1) from CNS through the pre-registered virtual node ID.

[0070] Call the cross-domain resource interface to query the current status of the shared VkImage (such as image layout and resolution) to ensure that the composition parameters match the resource.

[0071] S3.2, Window composition within the container and GCU encapsulation.

[0072] V-Weston performs synthesis based on global state and generates a standardized GCU:

[0073] Efficient compositing: For overlapping windows within a container, pixel-level blending is performed directly in shared memory using vkCmdBlitImage (e.g., when the upper window's alpha=0.8, the result is calculated as upper layer * 0.8 + lower layer * 0.2); for windows that need to be rotated or scaled, VkSampler is called to set bilinear filtering, and the transformation is completed using vkCmdTransformImage, avoiding CPU involvement.

[0074] GCU Standardized Wrapper: After compositing, instead of generating a final image, the Global Composition Unit (GCU) is encapsulated, containing:

[0075] Resource information: External memory handle and CGUID shared with VkImage;

[0076] Display parameters: window coordinates (x, y, w, h) in the OpenHarmony global coordinate system, transparency (alpha value), and level (z-order);

[0077] Progress marker: Current value of global timeline semaphore (set to 3 to indicate synthesis is complete).

[0078] GCUs are submitted to the OpenHarmony Global Synthesis Scheduler (GCS) via the virtio-vsock channel, with a submission latency of less than 1ms.

[0079] S3.3, Dynamic adjustment of synthesis strategy.

[0080] V-Weston optimizes the synthesis strategy in real time based on the global state:

[0081] If the global refresh rate drops from 60Hz to 30Hz, the compositing period is automatically extended from 16.67ms to 33.33ms, reducing compositing resource consumption by 50%. If the container window loses focus (by obtaining focus status from GCS), the compositing priority is reduced, changing from compositing every frame to compositing every 3 frames, reducing invalid calculations.

[0082] S4. Cross-domain resource verification and permission control (legality check) based on Vulkan external specifications: Verify the legality of the generated GCU resources to avoid retries or conversions caused by resource issues during the screen upload stage. This is a key check before screen upload.

[0083] S4.1 Three-layer resource validity verification.

[0084] After receiving the GCU, the GCS verifies it through the following process:

[0085] External handle attribute verification: Call vkGetMemoryFdPropertiesKHR to query the attributes of the external memory handle in the GCU (such as memory type, size, supported operations) to confirm that it matches the format supported by the GPU on the OpenHarmony side (such as checking whether it supports layout conversion of VK_FORMAT_R8G8B8A8_UNORM).

[0086] Resource status consistency check: Call vkGetImageSubresourceLayout to query the current layout of the shared VkImage (such as VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL), compare it with the layout required for the screen (VK_IMAGE_LAYOUT_PRESENT_SRC_KHR), and if they are inconsistent, automatically complete the hardware-accelerated conversion through vkCmdPipelineBarrier (time less than 1ms);

[0087] Permission token verification: Extract the permission token bound in GCU (generated by V-ARM, containing owner ID, operation type, and validity period), verify that the token has not expired and the operation type is updatable, and reject unauthorized access (such as a container attempting to modify read-only resources).

[0088] S4.2 Queue family matching optimization.

[0089] If the container and the OpenHarmony side use different GPUs, improve resource access efficiency in the following ways: query the queue family information of both GPUs and select a shared queue family that supports both graphics and transmission capabilities; allocate the memory of the shared VkImage to the memory pool of this queue family to reduce the synchronization overhead of cross-queue access; for the very few formats that do not support external memory, perform asynchronous transmission through the transmission channel (vkCmdCopyImage) of the shared queue, and set the timeline semaphore to 4 after the transmission is completed to mark the resource as ready.

[0090] S4.3 Resource lifecycle synchronization.

[0091] After GCS verification is passed, the resource reference count (container-side reference and OpenHarmony-side reference) is updated synchronously, and V-ARM is notified to record the mapping relationship between the resource and the on-screen window. If the application closes the window, V-Weston notifies V-ARM to decrement the reference count through CNS. When the count is 0, the resource is released to avoid memory leaks.

[0092] S5 and Vulkan timeline semaphore-driven end-to-end collaborative scheduling binds the rhythm of rendering, compositing, and verification, achieving seamless integration of rendering, compositing, verification, and on-screen display, and is the ultimate guarantee for on-screen display.

[0093] S5.1, End-to-End Stage Rhythm Binding.

[0094] Based on global timeline semaphores, define the triggering conditions and progress markers for each stage:

[0095] Before GCS starts global synthesis, it waits for the semaphore to reach values ​​3 and 4 via vkWaitForSemaphores to prevent the screen from being started when synthesis is incomplete or resources are invalid.

[0096] After global compositing (merging the container GCU with the native window) is completed, GCS sets the semaphore to 5 via vkSignalSemaphore to notify the on-screen service (OH-Present).

[0097] After the OH-Present waits for the semaphore to reach 5, it calls vkQueuePresentKHR to perform the on-screen operation. After the on-screen operation is completed, it resets the semaphore to 0 and triggers the rendering of the next frame.

[0098] S5.2 Dynamic Delay Compensation and Parameter Feedback.

[0099] When a delay occurs in a certain step, the preceding steps can be compensated for and optimized in the following ways:

[0100] Latency detection: GCS calculates the time taken for the semaphore value to change from 2 to 3 (i.e., from rendering to compositing) and from 3 to 4 (i.e., from compositing to verification) in real time. If the time taken for two consecutive frames exceeds the threshold (e.g., 8ms), it is judged as latency.

[0101] Compensation strategy: If the rendering latency (the time taken to change from 1 to 2 exceeds 5ms), GCS notifies V-Weston through CNS to reduce the application's rendering load (e.g., from 1080P to 720P), and V-Weston then notifies the application to adjust the rendering parameters through the Wayland extended interface; if the compositing latency (the time taken to change from 2 to 3 exceeds 8ms), V-Weston automatically turns off anti-aliasing, blurring and other effects.

[0102] Parameter feedback: After the data is displayed, OH-Present will feed back the display time and display device status (such as refresh rate changes) to V-Weston and the application through CNS. V-Weston will adjust the compositing cycle and the application will optimize the rendering strategy for the next frame (such as locking the rendering frame rate at 30Hz when the refresh rate drops to 30Hz).

[0103] S5.3 Fault tolerance and screen display protection.

[0104] If any component fails (such as a V-Weston crash), ensure uninterrupted screen display by using the following methods:

[0105] After CNS detects that V-Weston is unresponsive, it immediately starts a backup synthesis agent, which temporarily takes over the synthesis task based on the verified resources, with a switching time of less than 50ms.

[0106] If a shared resource is damaged, V-ARM automatically schedules a backup resource from the pre-allocated pool and rebinds the application through the interface to ensure that rendering is not interrupted and the on-screen latency increases by less than 2ms.

[0107] This embodiment establishes a Vulkan advanced feature-coupled cross-domain resource direct write mechanism, deeply integrating Vulkan multi-layer rendering (MRT), sub-pass resource reuse, and cross-domain sharing of VkImage. This allows Linux applications to directly write multi-channel rendering results to pre-allocated resources on the OpenHarmony side, eliminating the efficiency losses associated with traditional data copying and single-channel rendering. It also transforms the container compositor into an OpenHarmony virtual compositor node, specifically targeting Wayland within the container. The compositor (V-Weston), by registering as an OpenHarmony virtual compositor node, accesses the global compositing context and outputs globally compatible compositing units (GCUs), breaking down the fragmented link from independent compositing to proxy forwarding and then to secondary compositing. It achieves cross-domain resource standardization management based on the Vulkan external memory handle specification, constructing a three-layer management system including format verification, state synchronization, and permission tokens. This enables zero-conversion reuse of resources across different GPU architectures (such as ARM or Intel) while preventing unauthorized access. It also achieves globally state-aware dynamic adaptation of container rendering and compositing, using the OpenHarmony Global State Broadcast Service (GSBS) to match the container's rendering and compositing parameters (frame rate, resolution) with the system's display status (refresh rate, load) in real time, avoiding unnecessary resource consumption. Finally, it implements Vulkan timeline semaphore-driven end-to-end collaborative scheduling, using Vulkan timeline semaphores to define the end-to-end stage values ​​from rendering and compositing to on-screen display, achieving precise rhythm binding and dynamic latency compensation for each stage, completely resolving screen tearing and latency accumulation issues.

[0108] This invention significantly improves process efficiency. Following a workflow design from resource gathering, rendering, compositing, verification to on-screen display, it eliminates ineffective steps such as resource waiting and format conversion in traditional solutions. Display latency is reduced from 30ms to less than 10ms, reaching native application levels. Rendering performance is close to native; features such as MRT and sub-passes improve in-container application rendering efficiency by 30% to 40%, 3D modeling scene frame rate increases from 30fps to over 50fps, and video editing and export speed is increased by 25%. Resource utilization is optimized; cross-domain resource pre-allocation and lifecycle management reduce container memory usage by 40% to 50% (200MB reduction for 1080P scenes) and ineffective resource consumption by 60%. On-screen stability is ensured; end-to-end collaborative scheduling reduces screen tearing rate from 5%. The failure rate is reduced to less than 0.01%, the fault switching time is less than 50ms, and the on-screen interruption rate is less than 0.1%. It enhances compatibility and scalability, requires no modification to Linux application source code, has greater than 95% compatibility with Vulkan and Wayland applications, supports high-end scenarios such as 4K resolution and 120Hz refresh rate, and can be adapted to multi-architecture GPUs such as ARM and x86.

[0109] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for displaying Linux container applications on the HarmonyOS platform based on virtual composite nodes, characterized in that, Specifically, the following steps are included: Using HarmonyOS as the host and Linux as the container, the host starts a Vulkan resource management hub to obtain supported Vulkan features, pre-allocates shared Vulkan resources for different Linux application types, pre-creates rendering channels and binds them to the frame buffer, initializes global timeline semaphores, and forms a resource pool by the ID, type, file descriptor FD, rendering channel, frame buffer and timeline semaphore of the shared Vulkan resources; After receiving a resource request from the container's Vulkan driver, the Vulkan resource management hub selects a shared Vulkan resource from the resource pool and sends the shared Vulkan resource's ID and FD to the container. The container Vulkan driver imports the FD to obtain the Vulkan video memory object, binds it to the resources in the container, submits a node registration request to the host synthesis node management service, and obtains the virtual node ID assigned to it by the host synthesis node management service; When a Linux application creates an image resource, the Vulkan driver binds the corresponding shared Vulkan resource to the application rendering context, creates a rendering process using the corresponding rendering channel, and the fragment shader executes multiple rendering targets in parallel, writing color values, depth values, and normal vectors into different channels of the shared Vulkan resource. After rendering is completed, the global timeline semaphore is updated, the obtained image pixel data is written into the shared Vulkan resource, and a rendering completion notification is sent to trigger the container compositor to obtain the global display state of HarmonyOS through the virtual node ID. The compositor then performs window blending, rotation, and scaling in the corresponding framebuffer to complete the compositing, and submits the global compositing unit containing the shared Vulkan resource ID, window display parameters, and compositing progress markers to the HarmonyOS global compositing scheduler. The HarmonyOS global synthesis scheduler updates the resource reference count, notifies the Vulkan resource management center to record the mapping between shared Vulkan resources and the on-screen window, and completes the global synthesis of the host native window and the container application window when the global timeline semaphore is satisfied. The display service is driven by the global timeline semaphore to complete the on-screen display.

2. The method for displaying Linux container applications under the HarmonyOS platform according to claim 1, characterized in that, The container compositor dynamically adjusts the composing cycle based on the system's global refresh rate and reduces the composing frequency of background windows based on the window's focus state.

3. The method for displaying Linux container applications under the HarmonyOS platform according to claim 1, characterized in that, The HarmonyOS global compositing scheduler extracts the attributes of the external memory handle in the global compositing unit and verifies whether the attributes match the format supported by the HarmonyOS GPU.

4. The method for displaying Linux container applications under the HarmonyOS platform according to claim 1, characterized in that, The HarmonyOS global synthesis scheduler queries the current layout of shared Vulkan resources, compares the current layout with the layout required for the screen, and if they match, the verification is passed; otherwise, GPU instructions are used to accelerate the layout conversion.

5. The method for displaying Linux container applications under the HarmonyOS platform according to claim 1, characterized in that, The HarmonyOS global synthesis scheduler extracts the permission tokens bound in the global synthesis unit and verifies whether the permission tokens have not expired and whether the operation type is "can be displayed".

6. The method for displaying Linux container applications under the HarmonyOS platform according to claim 1, characterized in that, When the container uses a different GPU than the host, the queue families of the GPUs used by the container and the host are obtained respectively. The queue family that supports both graphics rendering and data transfer is selected, and the shared Vulkan resources and corresponding memory handles are allocated to the memory pool corresponding to the queue family. For shared Vulkan resources that do not support external memory export, asynchronous image copying is performed through the queue family to complete the format conversion. After the asynchronous image copy is completed, the resource transfer is marked as completed and set to the ready state by updating the timeline semaphore.

7. The method for displaying Linux container applications under the HarmonyOS platform according to claim 1, characterized in that, The resource request includes the Linux application type and Vulkan resource requirements.

8. The method for displaying Linux container applications under the HarmonyOS platform according to claim 1, characterized in that, The rendering channel is a chain of sub-channels formed by sub-channels.

9. The method for displaying Linux container applications under the HarmonyOS platform according to claim 8, characterized in that, After the fragment shader performs multi-target rendering in parallel and writes color values, depth values ​​and normal vectors into different channels of the shared Vulkan resource, the sub-channel reads the attachment channel of the shared Vulkan resource through the input attachment to complete the rendering of the corresponding stage. After the rendering of the corresponding stage of the sub-channel is completed, the global timeline semaphore is updated, and the rendered image pixel data is written into the memory area corresponding to the shared Vulkan resource.

10. The method for displaying Linux container applications under the HarmonyOS platform according to claim 1, characterized in that, The registration request includes a container ID and compositing capabilities, including support for high dynamic range, the maximum number of windows that can be opened simultaneously, and supported resolutions and formats.

Images