Containerized-microkernel camera efficient access method based on cross-frame shared page table

Abstract

The application discloses a containerized Hongmeng camera efficient access method based on cross-frame shared page table, takes Linux as a host, takes OpenHarmony as a container, and a V4L2 drive pre-grants a container process logical device node access permission when the container starts; when a container application applies to open a camera, the drive creates a logical device node bound to the container and returns a response; when an application queries a camera list, the container reads hardware capability and converts it into container standard device information, and completes cooperative device registration; after the application issues a flow configuration request, the drive completes format adaptation, buffer quantity calculation, and allocation of DMA shared memory divided according to the flow; after the application starts collection, the drive configures DMA hardware to write camera images into shared memory, and after data is ready, the container distributes data to corresponding business modules according to a flow identifier and a priority, marks a buffer as idle after processing is completed, and returns the buffer to the drive for recycling and reuse, and efficient access of a camera in a container is completely realized.

Description

Technical Field

[0001] This invention belongs to the field of computer software development technology, specifically relating to a containerized HarmonyOS camera high-efficiency access method based on cross-framework shared page tables. Background Technology

[0002] OpenHarmony (hereinafter referred to as OH), as a distributed operating system for all scenarios, has four core features in its camera software framework: distributed device management, multi-stream concurrent scheduling, advanced function standardization, and ION centralized memory management. The underlying layer relies on the Stream Pipeline architecture of Camera HAL 3.0, which supports parallel multi-stream preview, shooting, and recording and strictly follows priority scheduling, that is, the priority decreases from preview to shooting to recording. The middle layer provides a unified application call interface through CameraFramework, which encapsulates advanced control logic such as autofocus AF, auto exposure AE, white balance AWB, and high dynamic range imaging HDR, and integrates a distributed device discovery protocol. The underlying layer relies on the ION memory manager to realize the unified allocation, management, and reclamation of buffers, ensuring the security and efficiency of memory access. The Linux V4L2 driver framework is characterized by device node management, ioctl command set interaction, buffer queue transfer, and native DMA support. Its physical cameras are abstracted as device nodes, and device configuration, data acquisition, and control operations are realized through a standardized ioctl command set. It adopts a producer-consumer pattern to manage image data through buffer queues, supports efficient data paths such as mmap memory mapping and direct DMA transfer, and achieves fine-grained control of camera hardware through the ctrl subsystem.

[0003] When OH is deployed as a container on a Linux host, the fundamental differences between the two frameworks will lead to intractable technical bottlenecks in camera access. Existing adaptation solutions do not fully utilize the characteristics of both frameworks, mainly due to the following technical problems: First, there is a lack of interface coordination. Because OH's distributed device discovery mechanism cannot recognize V4L2's device node management mode, an additional independent device enumeration middleware needs to be developed, resulting in high device recognition latency. Due to the semantic mismatch between V4L2's ioctl command set and OH Camera HAL 3.0's stream configuration interface, parameter negotiation requires three conversions from the OH framework layer to the middle layer and then to the V4L2 driver, thus introducing redundant overhead. Furthermore, the significant differences in the data structure definitions between the two require additional parsing and conversion, further reducing execution efficiency. Second, data transmission efficiency is low. OH uses the ION memory manager for buffer allocation, while V4L2 relies on kernel-mode buffer queues. The memory spaces of the two are isolated from each other, and data needs to pass through the V4L2 driver kernel buffer to the user-mode middle layer buffer and then to OH. The ION buffer requires three copies, and the conversion between the commonly used NV21 / RGBA format in OH and the native YUYV, MJPG, and H.264 formats of V4L2 relies on software. In high-resolution scenarios such as 4K, the frame rate is less than 20fps, which cannot meet the requirements of high-definition and low latency. Third, there are conflicts in container resource isolation. Linux containers isolate device resources through namespaces. The current method requires manually mounting nodes and configuring corresponding user group permissions, which cannot be adapted to OH's distributed permission management model. In addition, the CPU scheduling strategy of the container is out of sync with the interrupt response mechanism of the V4L2 driver, resulting in poor real-time performance of camera data transmission. In particular, when multiple containers request camera resources at the same time, there is a lack of an effective contention arbitration mechanism, which can easily lead to crashes caused by resource preemption. Fourth, there is a break in the mapping of advanced functions. Advanced functions supported by OH, such as distributed multi-camera collaboration and HDR dynamic range adjustment, are not implemented in the standard control items of V4L2. The current method can only map basic control functions and lacks an automatic function compatibility adaptation mechanism, which makes it impossible to enable some advanced features of camera hardware.

[0004] In summary, existing adaptation solutions rely solely on a middleware layer for simple switching without deep integration of the two frameworks. This fails to fundamentally address issues such as slow device recognition, multiple data copies, poor real-time performance, resource conflicts among multiple containers, and the inability to use advanced features. Consequently, these solutions cannot meet the usage requirements of containerized OH (High-Definition, Low-Latency, Multi-Camera Collaboration) systems. Summary of the Invention

[0005] In view of this, the present invention provides a containerized HarmonyOS camera high-efficiency access method based on cross-framework shared page tables. Through layered interaction between the OH container and the Linux host V4L2 driver, parameter negotiation, zero-copy transmission of shared memory, hardware capability adaptation, dynamic function control and full-process resource reclamation mechanism, it realizes end-to-end closed-loop camera processing capabilities, including camera device initialization, multi-stream concurrent acquisition, advanced image quality control and security resource management.

[0006] This invention provides a method for efficient access to containerized HarmonyOS cameras based on cross-frame shared page tables, specifically including the following steps:

[0007] Using Linux as the host and OpenHarmony as the container, the V4L2 driver within the host generates camera capabilities. When the container starts, the host grants the container process access permissions to the V4L2 logical device node. When the container application initiates a request to open the camera, the container generates a resource request message and sends it to the host, triggering the V4L2 driver to create a container logical device node bound to the container ID, granting the container process camera-related permissions and access permissions to the container logical device node and shared memory, allocating an interrupt channel for the container camera data, and generating a resource request response to return to the container.

[0008] The container application initiates a camera list query request. The container V4L2 device adaptation unit obtains the container logical device node, triggers the V4L2 driver to query camera capabilities to form a device data packet, converts it into container device information, and the container registers the camera and marks it as a V4L2 collaborative device. The container application sends a stream configuration request, which is encapsulated by the container into a container stream configuration request and sent to the V4L2 driver. The V4L2 driver is triggered to match the format, determine the frame rate and resolution list, convert the container application priority into the host buffer scheduling priority, calculate the number of buffers, and send a stream configuration response to trigger the container to initialize the stream channel, buffer management queue, and data distribution logic. The V4L2 driver allocates shared memory and divides it into buffers corresponding to the data streams, and maintains the buffer state queue. The V4L2 driver associates the shared memory with its own buffer queue and binds it to the container handle. The container attaches the shared memory handle to its own stream queue.

[0009] The container application sends a request to start image acquisition. The container generates and sends a data stream start message to trigger the V4L2 driver to configure the DMA hardware. The image data acquired by the camera is written to shared memory via the DMA hardware. Then, a data ready notification is sent to trigger the container to distribute the image data to the corresponding processing module according to the stream identifier and priority. After processing, the corresponding buffer is marked as idle. The V4L2 driver adds it to the idle queue to complete the access.

[0010] Furthermore, the container sends a container stream configuration request containing basic stream information, format and resolution, buffer requirements, and stream combination relationships, triggering the V4L2 driver to match the format. Specifically, the V4L2 driver queries the format support list of the camera capability, compares the requested format identifier with the V4L2 native format code, and records the format identifier if the formats match. Otherwise, if the host supports hardware-accelerated conversion, it records the format conversion rules and the corresponding hardware conversion channel ID. If the host does not support hardware-accelerated conversion, it selects the compatible format that is closest to the requested format.

[0011] Furthermore, the container application sends a camera control request. The container determines the V4L2 control item ID and parameter value, encapsulates the container application's function command, parameter value, operation type, and maximum waiting time into a container control request, and sends it to the V4L2 driver. After receiving the request, the V4L2 driver converts the parameter value according to the mapping conversion relationship, completes the configuration writing of the camera sensor and ISP hardware registers, and then encapsulates the operation execution result, actual hardware parameters, running status, and command time into response data and sends it back to the container. After receiving the response data, the container reverse converts the original hardware parameters into standard values ​​that the application can recognize, synchronously sends an operation result notification to the container application, and updates the locally stored camera running status.

[0012] Furthermore, for multi-camera collaborative control commands, the V4L2 driver synchronously distributes the container control request to other camera drivers to execute unified configuration, and then encapsulates the execution results of all cameras into response data and sends it back to the container.

[0013] Furthermore, when the container application sends a camera shutdown request or the application process exits, the container generates a container data stream stop request based on all running data stream identifiers, sends the request to the V4L2 driver, synchronously notifies the container to delete the address mapping relationship of shared memory, reclaims the container memory management handle and marks the corresponding buffer as to be reclaimed, sends control instructions to the V4L2 driver to restore the camera hardware parameters to the default values, and sends a resource release request to the V4L2 driver.

[0014] After receiving a request to stop the container's data stream, the V4L2 driver terminates hardware data transmission and buffer queue scheduling, waits for all ongoing data processing tasks to finish, and releases resources in sequence upon receiving a resource release request. Then, it sends a cleanup notification to the container. The container verifies whether all local resources have been released. After verification and release, it shuts down the camera. If there are any unreleased resources, it activates a forced reclamation mechanism.

[0015] Furthermore, when the V4L2 driver receives multiple container camera resource requests simultaneously, it reads the priority and resource requirements of each container's request, stores all requests in the container request queue and sorts them according to arrival time; then it reads the resource occupancy status table to obtain the real-time occupancy information of the cameras, determines the resource allocation method according to the driver's preset arbitration configuration mode, and forms an arbitration result; based on the arbitration result, the container that receives the authorization feedback obtains the camera resources, and other containers receive a notification of failure to compete.

[0016] Furthermore, the step of determining the resource allocation method and forming the arbitration result based on the driver's preset arbitration configuration mode specifically involves:

[0017] In preemptive mode, all requests in the container request queue are traversed, and resources are allocated to high-priority containers first. If the resource occupancy status table shows that the camera is currently occupied by a low-priority container, a preemption notification is sent to the low-priority container to make it relinquish the camera. The container that receives the preemption notification stops image acquisition and saves its running status, reports to the host that the resource release is complete, and after the host confirms that the resources are cleared, the camera resources are allocated to a high-priority container.

[0018] In fair scheduling mode, the arbitration module continuously reads the occupied time recorded in the resource occupancy status table. After the current occupied container's usage time expires, the resource is released, and then subsequent containers in the container request queue are scheduled in sequence.

[0019] Furthermore, when a container application needs multiple cameras to collect data synchronously, the container initiates a batch device query request to the V4L2 driver. After receiving the request, the V4L2 driver returns the device file paths corresponding to all currently bound cameras. The container sets the master-slave identity identifier and synchronization clock identifier of each camera based on the returned information, and establishes a distributed collaborative relationship between multiple cameras.

[0020] Furthermore, when the container collects operational metrics data from the camera, it acquires frame rate, data transmission latency, response time to control commands, format conversion overhead, memory usage, processor load, buffer usage percentage, data transmission bandwidth, and success rate of various functions at fixed intervals. The container uses a circular storage method to save the collected data. It sets the normal threshold range for each metric according to the system configuration. When the operating values ​​exceed the limit range, it sends performance optimization requests and system alarm information. At the same time, it periodically sends the monitoring data to the V4L2 driver monitoring node.

[0021] The container adaptation and tuning unit determines the cause of the performance bottleneck based on the anomaly information and driver logs, and sends the corresponding optimization and adjustment instructions to the V4L2 driver. The driver receives and executes the instructions to complete the optimization. The container monitors changes in indicators, and records the current optimization configuration when the running data is within a reasonable range. If the indicators do not return to normal, the optimization strategies are activated sequentially until the problem is fixed.

[0022] Furthermore, when the container detects abnormal operation information from the camera, the exception handling unit within the container obtains the call return status, buffer operation flag, and instruction execution timeout event, records the time of the exception occurrence, fault type, error flag, and on-site operating parameters, and performs tiered processing according to three fault levels: severe, normal, and minor. In the event of a severe fault, all data stream transmission is terminated, the current camera operating configuration is saved, and the device fault information is pushed to the container application, while a fault alarm message is sent to the V4L2 driver. In the event of a normal fault, buffer reset and image format path switching repair operations are performed first; once the repair is successful, normal business operation is restored and a notification is sent. If the application anomaly has been resolved, and the repair fails, the operating indicators will be reduced or the image function will be stopped. When a minor fault occurs, only the anomaly record is retained, the business process continues to operate normally and the command is retried periodically. At the same time, the complete fault scene data is reported to the system log service. After receiving the container fault notification, the V4L2 driver shuts down data transmission, queue scheduling and hardware interrupt response behavior, resets various internal operating data structures of the driver, reclaims the system resources occupied by the fault, and completes the repair operations of hardware device reinitialization, memory block reallocation and transmission path reconfiguration according to the fault type. After processing, the repair results and updated resource configuration data are returned to the container.

[0023] Beneficial effects:

[0024] In this invention, a camera architecture using Linux as the host and OH as the container, the host V4L2 driver reads the camera hardware capabilities and generates a capability cache during initialization. After the container application initiates a camera opening request, the container generates a signature request carrying identity, permissions, and resource information and reports it to the host. After the host completes multi-dimensional verification, it creates a dedicated logical device node for the container, grants corresponding permissions, allocates an independent transmission interrupt, and returns resource configuration information. The container queries the driver hardware capabilities through the dedicated device node, converts it into standard container device information, and completes local device registration. After the container issues a stream configuration request containing specification parameters, buffer requirements, and stream combination relationships, the driver verifies and adapts the image format, resolution, and frame rate, completes priority mapping, buffer calculation, and bandwidth allocation, and returns the final effective parameters to complete the container streaming process. Service link initialization; the driver uses DMA mechanism to create a cross-framework shared memory pool and realize page table sharing, the container completes user space memory mapping, and both parties bind buffer queues bidirectionally to build a zero-copy data interaction channel; after the container starts image acquisition, the camera acquires images according to negotiated parameters and writes them to shared memory via DMA. The driver updates buffer metadata through interrupts and pushes data ready notifications. After the container verifies the data, it completes multi-stream service distribution. The buffer that has been read is reclaimed and reused by both ends in a collaborative manner; when the container sends a camera control request, both parties complete parameter mapping, hardware configuration, status feedback and multi-camera collaborative control to ensure real-time synchronization of image quality parameters; the container initiates camera shutdown and resource release requests, the driver orderly stops transmission and reclaims all software and hardware resources, and the container cleans up local resources and supports forced reclamation. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the containerized HarmonyOS camera high-efficiency access method based on cross-frame shared page tables provided by the present invention. Detailed Implementation

[0026] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0027] The core idea of ​​this invention, which provides a containerized HarmonyOS camera access method based on cross-framework shared page tables, is as follows: In a camera architecture with Linux as the host system and OH as the container system, the host V4L2 driver reads the camera hardware capabilities and generates a capability cache during initialization. After the container application initiates a camera opening request, the container generates a signature request carrying identity, permissions, and resource information and reports it to the host. After the host completes multi-dimensional verification, it creates a dedicated logical device node for the container, grants corresponding permissions, allocates an independent transmission interrupt, and returns resource configuration information. The container queries the driver hardware capabilities through the dedicated device node, converts it into standard container device information, and completes local device registration. After the container issues a stream configuration request containing specification parameters, buffer requirements, and stream combination relationships, the driver verifies and adapts the image format, resolution, and frame rate, and completes priority mapping, buffer calculation, and bandwidth calculation. The process involves allocating and sending back the final effective parameters to initialize the container streaming service link. The driver uses a DMA mechanism to create a cross-framework shared memory pool and achieve page table sharing. The container completes user space memory mapping, and both parties bind buffer queues bidirectionally to build a zero-copy data interaction channel. After the container application starts image acquisition, the camera acquires images according to the negotiated parameters and writes them to the shared memory via DMA. The driver updates the buffer metadata through interrupts and pushes data ready notifications. After verifying the data, the container completes multi-stream service distribution. The buffers that have been read are reclaimed and reused collaboratively by both ends. When the container sends a camera control request, both parties complete parameter mapping, hardware configuration, status feedback, and multi-camera collaborative control to ensure real-time synchronization of image quality parameters. When the container initiates a camera shutdown and resource release request, the driver orderly stops transmission and reclaims all software and hardware resources. The container cleans up local resources and supports forced reclamation.

[0028] The present invention provides a containerized HarmonyOS camera high-efficiency access method based on cross-frame shared page tables, the processing flow of which is as follows: Figure 1 As shown, the specific steps include:

[0029] Step 1: Using Linux as the host system and OH as the container system, during the initialization phase of the host V4L2 driver, the host reads the camera hardware registers to obtain hardware capabilities and generates a camera capability cache. When the container starts, the host adds the user identity ID of the container process to the host video user group. When the container application initiates a request to open the camera, the container collects the container identifier, functional requirements, required resource quotas and permission declarations, generates a standard resource request message, and sends the resource request message to the host.

[0030] After receiving the request, the host sequentially completes identity verification, hardware capability verification, and resource quota verification. Once the verification is successful, the V4L2 driver creates a container-specific logical device node bound to the container ID; grants the container process camera acquisition permissions, real-time scheduling permissions, and memory locking permissions; allows the container process to access the container-specific logical device node, execute V4L2-related commands, and access shared memory; allocates an independent interrupt channel for container-related camera data transmission; generates a resource request response containing the logical device node path, resource quota allocation results, a list of supported advanced features, and supported hardware conversion channels, and sends it to the container.

[0031] The container identifier includes the container process ID, container ID, container distributed device ID, and the application signature to which the container belongs. Functional requirements include the target application scenario, resolution range, frame rate requirements, image format requirements, advanced functional requirements, and real-time performance level. Required resource quotas include the number of CPU cores, memory size, number of buffers, and interrupt channel requirements. The permission declaration includes the requested device access permission type and permission validity period. The target application scenario is previewing, taking photos, recording, or multi-stream concurrency.

[0032] Furthermore, when multiple containers simultaneously request the same camera resource, the resource authorization module built into the host V4L2 driver detects resource contention and initiates the resource arbitration module. The resource arbitration module determines the resource allocation based on the container request queue, resource occupancy status table, and configurable dual-scheduling mode, forming an arbitration result. The container's built-in resource contention processing unit receives the arbitration result from the host and executes the corresponding processing, specifically:

[0033] When the host V4L2 driver receives multiple container camera resource requests simultaneously, the resource arbitration module reads the priority and resource requirements of each container request, stores all requests in the container request queue, and sorts them according to their arrival order. It then reads the resource occupancy status table to obtain real-time camera occupancy information and allocates resources based on the driver's preset arbitration configuration modes. In preemptive mode, it iterates through all requests in the container request queue, prioritizing resource allocation to high-priority containers. If the resource occupancy status table shows that a camera is currently being used by a low-priority container, a preemption notification is sent to that low-priority container, instructing it to relinquish the camera. In fair scheduling mode, the arbitration module continuously reads the occupied duration recorded in the resource occupancy status table. Once the currently occupied container's usage time expires, it releases the resources and then schedules subsequent containers in the container request queue in order. This process forms the arbitration result.

[0034] According to the arbitration result, the container that acquires the camera resources will receive authorization feedback, while the other containers will receive a competition failure notification, with the reason for failure and an alternative solution to switch to other cameras; for the container that receives the preemption notification, it will stop image acquisition and save the running state, report to the host that the resource release is complete, and after the host confirms that the resources are cleared, the camera resources will be allocated to high-priority containers.

[0035] The resource occupancy status table is used to record in real time whether the camera is currently occupied, the currently occupied container, the duration of occupation, and the resource load.

[0036] Step 2: When the container application initiates a camera list query request, the V4L2 device adaptation unit inside the container's device manager traverses the dedicated logical device nodes authorized for this container to obtain the corresponding device file path; based on the device file path, it sends an ioctl query command to the V4L2 driver. The driver reads information from the camera capability cache, preprocesses and encapsulates it into a device data packet, and the V4L2 device adaptation unit maps and converts the device data packet into a container standard device information structure; the container registers this camera to the container's local distributed device list based on this device information structure, marks the device type as a V4L2 collaborative device, and binds and records information such as the device file path and allocated resource quotas.

[0037] The implementation of initiating an ioctl query command to the V4L2 driver based on the device file path is as follows: The internal enumeration processing logic of the V4L2 driver is optimized, and a new device information query interaction command is added within the reserved extended command range. This includes: a command to output a complete basic camera information data packet, which is of fixed length and sequentially contains device identification parameters, resolution and frame rate parameters, image encoding format mapping relationships, container collaboration attributes, and hardware function bitmaps such as ISP and DMA according to fixed byte offsets; a command to feed back the synchronous operation of multiple camera data streams, including available stream combination types, bandwidth limits for each group, and scheduling priority limits for various data streams; and a command to output adjustable camera control parameters, including various control command numbers, parameter adjustment ranges, command response times, and real-time dynamic adjustment and adaptation status. When the container initiates a query interaction based on the device file path, the driver can directly output structured data packets.

[0038] Furthermore, when a container application needs multiple cameras to collect data synchronously, the container device manager initiates a batch device query request to the host driver through the V4L2 device adapter unit; the host driver returns the dedicated device file paths corresponding to all currently mutually bound physical cameras, and the container device manager sets the master-slave identity identifier and synchronization clock identifier of each camera according to the returned information, establishing a distributed collaborative relationship between multiple cameras.

[0039] Step 3: When the container application sends a stream configuration request containing parameters such as resolution, frame rate, and image format, the container's stream manager encapsulates the stream configuration request into a container stream configuration request containing basic stream information, format and resolution parameters, buffer requirements, and stream combination relationships. Then, the container stream configuration request is sent to the V4L2 driver. After receiving the request, the V4L2 driver queries the format support list in the camera capability cache, compares the format identifier of the request with the V4L2 native format code, and records the format identifier if the format matches. Otherwise, if the host supports hardware-accelerated conversion, it records the format conversion rules and the corresponding hardware conversion channel ID. If the host does not support hardware-accelerated conversion, it selects the compatible format that is closest to the requested format.

[0040] The V4L2 driver verifies the validity of the requested frame rate and resolution based on the sensor's maximum resolution and corresponding frame rate limit in the camera's capability cache, and sends the confirmed list of frame rates and resolutions to the container; it converts the application priority defined by the container into a buffer scheduling priority that the host kernel can recognize, and distinguishes the scheduling weight of preview, photo taking, and recording data streams according to the set mapping rules; and then calculates the number of buffers based on the resolution, frame rate, and transmission bandwidth indicators.

[0041] The V4L2 driver sends a stream configuration response to the container, which includes the actual resolution, frame rate, format, number of buffers, format conversion method, stream priority mapping result, and bandwidth quota for each stream. This triggers the container to initialize the stream channels, buffer management queues, and data distribution logic.

[0042] Specifically, the legality of the requested frame rate and resolution is verified, and the determined frame rate and resolution are sent to the container. If the requested frame rate is not greater than the hardware's rated limit, the original configuration is used; otherwise, it is corrected to the maximum frame rate supported by the hardware, and the corrected frame rate value is sent back to the container. The legality of the resolution is verified synchronously. If the configured resolution is not within the range supported by the hardware, a list of all available resolutions is pushed to the container for selection.

[0043] Among them, the stream combination relationship is a set of association rules that describe the combination, resource allocation, and operational constraints of multiple data streams when they share the same camera hardware.

[0044] Step 4: Based on the determined buffer parameters, the V4L2 driver calculates and allocates physical contiguous memory with redundant space as shared memory using the DMA mechanism. It divides the memory into independent buffer partitions belonging to different data streams, maintains the buffer state queue, and modifies the kernel page table attributes corresponding to the shared memory to achieve page table sharing. The container initiates a memory mapping request to the V4L2 driver and completes the container user space address mapping based on the shared memory and page table information issued by the V4L2 driver. The V4L2 driver associates the shared memory with its own buffer queue and binds it to the container handle. The container attaches the shared memory handle to its own stream queue and establishes a data interaction channel with the host.

[0045] Step 5: When the container application sends an image acquisition start request, the container generates a data stream start message containing the identifiers of each data stream and the corresponding buffer queue identifier, and sends the message to the V4L2 driver. After receiving the message, the V4L2 driver configures the start and end addresses of the DMA hardware and the amount of data transmitted in a single transmission. The camera acquires raw image data according to the determined frame rate, and the image data is written to the shared memory by the DMA hardware. After the writing is completed, an interrupt signal is triggered, which triggers the driver to increase the data length, acquisition timestamp, data verification information and ready status flag to update the shared memory, and pushes a data ready notification to the container with the data stream number, buffer number and metadata storage address.

[0046] After capturing the data ready notification, the container completes the shared memory data integrity verification. Based on the recorded stream identifier and priority, it distributes the image data to the corresponding processing modules for preview rendering, static image encoding, and video encoding. After the container application completes the image data processing, it marks the corresponding buffer in the shared memory as idle. The container sends the idle buffer number to the V4L2 driver, and the driver adds the buffer to the idle queue to wait for a new round of data writing.

[0047] Furthermore, the V4L2 driver relies on a priority scheduling mechanism to manage the DMA transmission resource allocation of multiple data streams in real time. High-priority preview data streams can preempt transmission resources to ensure stable frame rates, while low-priority data streams such as recording perform transmission tasks during resource idle periods, thereby meeting the frame rate requirements when multiple camera data streams run concurrently.

[0048] Furthermore, this invention adds a data receiving and processing unit to the data stream processing module within the container, which directly reads the original image data in the shared memory according to the mapped memory access address; at the same time, it calls hardware acceleration resources to complete the in-situ format conversion in the original shared memory space according to the conversion channel identifier of the shared memory, and opens the business usage permission of the buffer after the conversion verification is qualified; the container manages the shared memory read and write operations according to the access permissions of the container application process, and synchronously completes the reclamation and release of buffer resources according to the application running status.

[0049] Furthermore, the V4L2 driver optimizes buffer queue management and DMA transfer, distinguishes the transfer order of different data streams according to preemptive priority scheduling rules, improves the data writing efficiency of batch buffers by adopting DMA aggregation transfer characteristics, and realizes real-time synchronous updates of data status with interrupt notification mechanism.

[0050] Step 6: When the container application sends a camera control request, the container determines the corresponding V4L2 control item ID and parameter value, encapsulates the container application's function command, parameter value, operation type, and maximum waiting time into a container control request, and sends it to the V4L2 driver. After receiving the request, the V4L2 driver converts the parameter value according to the mapping conversion relationship, completes the configuration writing for the camera sensor and ISP hardware registers, and then encapsulates the operation execution result, actual hardware parameters, running status, and command time into response data and sends it back to the container. After receiving the response data, the container reverse converts the original hardware parameters into standard values ​​that the application can recognize, synchronously sends an operation result notification to the container application, and updates the locally stored camera running status.

[0051] If it is a multi-camera collaborative control command, the V4L2 driver will synchronously distribute the container control request to other camera drivers to execute unified configuration, and then encapsulate the execution results of all cameras into response data and send it back to the container.

[0052] The V4L2 driver pre-stores the one-to-one correspondence between container camera functions and host hardware control items in the form of configuration files. It distinguishes between three types of mapping rules: regular image quality control, advanced image capabilities, and multi-device collaborative control. Each rule is labeled with the conversion formula of the parameter ranges at both ends, instruction response indicators, hardware dependency conditions, and compatibility status. During the driver startup phase, the configuration content is read and saved to the kernel running memory. After the configuration file is updated, the internal cache can be refreshed in real time. The container management module adds an adaptation unit, which loads the mapping rules issued by the synchronous driver during the process startup phase and caches them persistently. The driver adds an instruction parsing unit, which distinguishes between standard hardware control instructions and custom extended control instructions to complete register configuration. When hardware write exceptions are captured, fault logs are recorded and the corresponding error identifier is returned.

[0053] The camera control requests include operations such as focus, exposure, white balance, HDR, digital zoom, and multi-camera simultaneous focus.

[0054] Furthermore, after receiving the request, the V4L2 driver verifies whether the current hardware module has the corresponding control capability and prerequisite operating conditions according to the control mapping lookup table. If the hardware does not support the function, it performs function degradation processing according to preset rules.

[0055] Step 7: When the container application sends a camera shutdown operation request or the application process exits, the container generates a container data stream stop request based on all running data stream identifiers, sends the request to the V4L2 driver, synchronously notifies the container to delete the address mapping relationship of shared memory, reclaims the container memory management handle and marks the corresponding buffer as to be reclaimed, sends control instructions to the V4L2 driver to restore the camera hardware parameters to the default values, and sends a resource release request to the V4L2 driver.

[0056] After receiving a request to stop the container's data stream, the V4L2 driver terminates hardware data transmission and buffer queue scheduling, waits for all ongoing data processing tasks to finish, and then releases resources in sequence upon receiving a resource release request. It then sends a cleanup notification to the container. The container verifies whether all resources, such as local memory handles, buffer queues, and data stream pipelines, have been released. After verification and release, the camera is turned off. If any unreleased resources exist, a forced reclamation mechanism is activated.

[0057] Furthermore, the V4L2 driver completes resource release in the following way: reclaiming DMA contiguous physical memory layer by layer, destroying the buffer queue structure corresponding to each data stream, removing the container-specific logical device file node, removing the user group access permissions corresponding to the container and uninstalling the security control policy, reclaiming the allocated hardware interrupt resources and unbinding the processor relationship, modifying all resources released this time to an idle state, and resetting the internal container adaptation field, hardware capability cache, resource quota table and other runtime data of the driver.

[0058] Furthermore, when the container collects operational data from the camera, the container system monitoring module acquires various operational information at fixed intervals, including frame rate, data transmission latency, response time to control commands, format conversion overhead, memory usage, processor load, buffer usage percentage, data transmission bandwidth, and success rate of various functions. The collected data is stored in a circular manner. Normal threshold ranges for each indicator are set according to system configuration. If the operational values ​​exceed the defined range, performance optimization requests and system alarms are sent. Simultaneously, monitoring data is periodically sent to the V4L2 driver monitoring node. The V4L2 driver outputs raw operational information, providing data for container fault location and performance analysis.

[0059] The container adaptation and tuning unit determines the cause of the performance bottleneck based on the anomaly information and driver logs, and sends the corresponding optimization and adjustment instructions to the V4L2 driver. The driver receives and executes the instructions to complete optimization operations such as increasing the number of buffers, adjusting processor core binding and scheduling weights, improving interrupt response level, and switching image format conversion paths. The container monitors changes in indicators, and records the current optimization configuration when the running data is within a reasonable range. If the indicators do not return to normal, optimization strategies are activated sequentially until the problem is fixed.

[0060] Furthermore, when the container detects an operational anomaly in the camera link, the anomaly handling unit within the container obtains the call return status, buffer operation flag, and instruction execution timeout event. It simultaneously records the time of the anomaly, fault type, error flag, and on-site operating parameters, and performs tiered processing according to pre-defined three fault levels: severe, moderate, and minor. In the event of a severe fault, all data stream transmission is immediately terminated, the current camera operating configuration is saved, and device fault information is pushed to the container application. Simultaneously, a fault alarm message is sent to the V4L2 driver. In the event of a moderate fault, priority is given to performing repair operations such as buffer reset and switching image format paths. Once the repair is successful, normal business operation is restored, and a notification is sent. If the application anomaly has been resolved, and the repair fails, the operating indicators will be reduced and the image function will be stopped. When a minor fault occurs, only the anomaly record is retained, the business process continues to operate normally and the command is retried periodically. At the same time, the complete fault scene data is reported to the system log service. After receiving the container fault notification, the V4L2 driver shuts down data transmission, queue scheduling and hardware interrupt response behavior, resets various internal operating data structures of the driver, reclaims system resources such as memory, buffers and interrupts occupied by the fault, and performs repair actions such as hardware device reinitialization, memory block reallocation and transmission path reconfiguration according to the fault type. After processing, the repair results and updated resource configuration data are returned to the container.

[0061] Example:

[0062] This embodiment employs the containerized HarmonyOS camera high-efficiency access method based on cross-frame shared page tables provided by this invention. Using Linux as the host system and OH as the container system, it establishes a processing flow from container resource request, multi-container resource arbitration, device discovery and enumeration, multi-stream parameter negotiation, memory and buffer initialization, data acquisition and zero-copy transmission, control function interaction, and resource release. This enables the containerized HarmonyOS application to use the host camera. The specific process includes:

[0063] S1. Dynamic Container Resource Request and V4L2 Driver-Driven Co-authorization: A V4L2 resource request submodule has been added to the ContainerManager module of OH. This submodule integrates OH distributed permission management logic that conforms to the OH permission control framework. When an OH application initiates a camera access request through CameraManager.openCamera(), it collects core request parameters and encapsulates them in a standardized manner. The encapsulated information includes:

[0064] Container identifier, container process ID (PID), container ID (CID), OH distributed device ID used for associated distributed camera scheduling, and application signature of the container for permission verification;

[0065] The functional requirements list includes target scenarios, resolution range, frame rate requirements, image format requirements, advanced functional requirements, and real-time performance levels. Target scenarios include preview, photo capture, recording, and multi-stream concurrency. Image format requirements include NV21, RGBA, or H.264. Advanced functional requirements include AF, AE, AWB, HDR, or distributed multi-camera collaboration. Real-time performance levels include low latency of no more than 100ms and normal latency of no more than 200ms.

[0066] Resource quota requests include the number of CPU cores, memory size, number of buffers, and interrupt channel requirements. For example, the number of CPU cores is 2 cores, the memory size is 16MB, the number of buffers is 4 for preview streams and 6 for recording streams, and the interrupt channel requirement is an independent interrupt channel.

[0067] The permission declaration specifies the type of device access permission requested, such as read, write, or control, and the validity period of the permission, such as during application runtime.

[0068] In the core structure v4l2_device of the device management module of the V4L2 driver, new OH container adaptation extension fields have been added, including: a container association list for storing information on currently authorized OH containers, including CID, PID, permissions, and resource quotas; a resource quota record table for recording the resource allocation of physical cameras, including CPU cores, memory, buffers, and interrupt channels; a permission mapping table for storing the correspondence between OH application signatures and V4L2 device access permissions; and a device capability cache for caching the hardware capabilities of physical cameras, including supported resolutions, frame rates, control items, and format conversion capabilities, to avoid repeatedly reading hardware registers.

[0069] Meanwhile, a new OH resource authorization submodule has been added to the V4L2 driver. This module is registered as the receiver for netlink communication and is specifically designed to handle resource request messages from OH containers.

[0070] This embodiment establishes the following two-way authorization collaboration process:

[0071] When sending resource requests, the V4L2 resource request submodule of OH uses the Linux netlink communication mechanism. This mechanism ensures communication security for the underlying channel of the reused OH distributed communication framework. It sends a standardized resource request message to the V4L2-driven OH resource authorization submodule. The message carries the container identifier, functional requirements list, resource quota application and permission declaration, and is authenticated by OH application signature.

[0072] Driver resource verification: After receiving the request, the V4L2 OH resource authorization submodule performs verification in the following order: Identity verification: Query the permission mapping table to verify whether the OH application signature is valid. If invalid, authorization failure is returned directly; Hardware capability verification: Read the device capability cache to determine whether the physical camera supports the functional requirements of the OH application, such as 4K resolution and HDR function. If some functions are not supported, it is marked as needing to be downgraded; Resource quota verification: Check whether the host's current idle resources, such as idle CPU cores, remaining memory, number of available buffers, and unused interrupt channels, meet the application requirements. If resources are insufficient, a resource shortage prompt and alternative resource configurations are returned.

[0073] Dynamic permission configuration: After successful verification, the V4L2 driver automatically performs the following authorization operations without manual intervention, including creating an OH-specific logical device node and calling the device_create() function to generate a logical node in the format / dev / oh_video_[CID]. This logical node is strongly bound to the container's CID, and only the container process with the corresponding CID is allowed to access this node to prevent unauthorized access by other containers.

[0074] Permission mapping is performed by adding the process user ID (UID) of the OH container to the host video user group through the setgroups() system call, and then granting the container process camera capture permission CAP_VIDEO_CAPTURE, real-time scheduling permission CAP_SYS_NICE, and memory locking permission CAP_IPC_LOCK through the capset() system call, ensuring that the container process has complete camera access capabilities.

[0075] The SELinux or AppArmor policy is dynamically loaded to generate a security policy file for the OH container, explicitly allowing the container process to access the / dev / oh_video_[CID] node, execute V4L2 related ioctl commands, and access the shared memory area. The policy is loaded through selinux_load_policy() or apparmor_parser, and the policy's effectiveness status is reported through netlink after loading.

[0076] Interrupt channel allocation: A separate interrupt channel is allocated for the camera data transmission of the OH container. This is achieved by binding the interrupt handling function through the request_irq() function, thus avoiding response delays caused by sharing the interrupt channel with other devices on the host.

[0077] In the resource request response, the V4L2 driver encapsulates the authorization result into a `struct oh_v4l2_auth_resp` structure, which includes the logical device node path, resource quota allocation result, a list of supported advanced features, and a description of format conversion capabilities. This is returned to the OH's V4L2 resource request submodule via netlink. The submodule then synchronizes the information to the OH's Camera Framework, completing the container resource request and authorization. The resource quota allocation result includes the actual allocated CPU core ID, memory size, number of buffers, and interrupt channel number. The list of supported advanced features indicates whether they are fully supported, partially supported, or require downgrading. The format conversion capability description lists the supported hardware conversion channels.

[0078] S2. Multi-container camera resource contention arbitration: A new OH container resource arbitration submodule has been added to the device management module of the V4L2 driver. This module solves the problem of multiple OH containers simultaneously requesting the same physical camera resource contention based on priority preemption and fair scheduling mechanism.

[0079] S2.1 The core components include container request queue, resource occupancy status table, arbitration strategy configuration, priority preemption mode and fair scheduling mode.

[0080] The system includes a container request queue, which stores all unauthorized OH container resource requests in order of arrival time and records request priorities. Priority is given to low-latency scenarios based on the real-time performance level of the OH application, while higher priorities are given to normal scenarios. A resource occupancy status table updates the physical camera resource occupancy status in real time, including the currently occupying container, occupancy duration, and resource utilization efficiency. The arbitration strategy configuration supports two arbitration modes, configurable via driver parameters: priority preemption mode and fair scheduling mode. In priority preemption mode, high-priority container requests can preempt camera resources from low-priority containers, pausing the low-priority containers and saving their current state. In fair scheduling mode, all containers take turns using camera resources in the order of requests, with each container's single usage duration configurable, defaulting to 5 minutes.

[0081] S2.2 Add a resource contention handling unit to the CameraManager of the Camera Framework in OH to support receiving resource arbitration results driven by V4L2 and handling resource preemption or waiting logic.

[0082] Preemption notification handling: If the camera resource of the current container is preempted by a higher priority container, the current camera state, including stream configuration parameters, control parameters, and buffer state, is automatically saved, and a callback of resource preemption is returned to the application to prompt the user to wait; Waiting queue management: If the camera resource is occupied and the arbitration mode is fair scheduling, it is added to the waiting queue, and the resource release status is queried from the V4L2 driver periodically. After the resource is released, a re-authorization request is automatically initiated.

[0083] S2.3 The resource arbitration process includes competition detection. When the V4L2-driven OH resource authorization submodule receives resource requests from multiple OH containers simultaneously, the resource arbitration submodule is triggered to work.

[0084] Arbitration decision-making: The resource arbitration submodule reads the priority and resource requirements of each request, combines them with the resource occupancy status table, and executes the decision according to the configured arbitration mode: Priority preemption mode: compares the priorities of all requests, selects the request with the highest priority to grant resources, and if the current camera is already occupied by a low-priority container, sends a preemption notification to the low-priority container, requesting it to release resources; Fair scheduling mode: adds all requests to the container request queue according to their arrival time, and when the usage time of the currently occupied container expires, notifies it to release resources, and wakes up the next request in the queue in turn.

[0085] The arbitration result feedback returns a successful authorization response to the container that won the request, and a resource contention failure response to the other requesting containers, explaining the reason for the failure, such as being preempted by a higher priority or waiting in the queue, and the alternative solution, such as choosing another physical camera;

[0086] During resource handover, after receiving the preemption notification, the low-priority container performs resource release preprocessing such as stopping streaming, saving state, and sending a resource release ready message to the V4L2 driver. After the V4L2 driver confirms the resource release, it reallocates the resource to the high-priority container to ensure uninterrupted resource handover.

[0087] S3 and OH distributed device discovery work in conjunction with V4L2 device enumeration. A new V4L2 device adaptation unit has been added to the OH distributed device management module DeviceManager. This unit deeply integrates OH distributed device discovery protocols such as mDNS or CoAP with V4L2 device enumeration logic. Its core functions include:

[0088] Device scanning is triggered by listening for calls to the CameraManager.getCameraDevices() interface in the OH application, which in turn triggers a scan of V4L2 logical device nodes, with higher priority than traditional OH distributed device scans. Data structure conversion allows direct conversion of device information returned by the V4L2 driver into the standard OH CameraDeviceInfo structure without requiring an additional parsing layer. Distributed device registration registers the adapted camera devices in the OH distributed device list, marking the device type as a V4L2 collaborative device and associating it with logical device node paths, resource quotas, and other information.

[0089] Modify the enumeration subsystem of the V4L2 driver. Based on the V4L2 extended command specification (command code range from 0x8000 to 0xFFFF), add the following OH-specific enumeration commands to the ioctl command set of the V4L2 driver to adapt to the distributed device information format of OH:

[0090] VIDIOC_OH_ENUM_DEV_INFO, with command code 0x8001, is used to return camera information conforming to the OH distributed device information format. The command response structure struct oh_v4l2_dev_info is 64 bytes in size and 4 bytes in alignment. It contains basic device information with an offset of 0 to 15 bytes, specifically a 16-byte string for the device name, 2 bytes for the vendor ID, 2 bytes for the product ID, and a 32-byte string for the logical device node path.

[0091] Resolution and frame rate information, offset from 16 to 31 bytes, specifically 1 byte for the number of supported resolutions and the maximum frame rate corresponding to each resolution. The maximum frame rate corresponding to each resolution is in array form, with each resolution occupying 2 bytes, and a maximum of 7 resolutions are supported.

[0092] The format support information, with an offset of 32 to 47 bytes, includes the supported V4L2 native format codes, the corresponding OH format codes, and the hardware-accelerated conversion support identifier. The supported V4L2 native format codes are in array form, with each format code being 4 bytes, and a maximum of 4 formats are supported. The corresponding OH format codes are also in array form, corresponding one-to-one with the native format codes. The hardware-accelerated conversion support identifier is a 4-byte bitmap, with each bit corresponding to one format conversion.

[0093] Distributed attributes, offset from 48 to 55 bytes, specifically 8 bytes of OH distributed device ID, 1 byte of whether cross-container collaboration is supported, and 1 byte of maximum concurrent stream count. The cross-container collaboration support is indicated by 0 for no support and 1 for support.

[0094] The hardware capability identifier, offset from 56 to 63 bytes, consists of a 4-byte advanced feature bitmap, a 1-byte DMA transfer support status, and a 1-byte ISP hardware support status. In the advanced feature bitmap, each bit corresponds to a function, such as bit0 for AF, bit1 for AE, bit2 for AWB, bit3 for HDR, etc.

[0095] VIDIOC_OH_ENUM_STREAM_CAP, command code 0x8002, is used to return the camera's multi-stream concurrency capability. The response structure struct oh_v4l2_stream_cap contains: 1 byte of the number of supported stream combinations, the stream type combination of each stream combination, 4 bytes of the maximum total bandwidth of each stream combination, and the priority limit of each stream. Among them, the stream type combination of each stream combination, such as preview and photo capture, preview and recording, occupies 2 bytes for each combination; the priority limit of each stream, such as 1 byte per stream;

[0096] VIDIOC_OH_ENUM_CTRL_CAP, command code 0x8003, is used to return details of supported control functions. The response structure struct oh_v4l2_ctrl_cap contains: the number of control items (1 byte), the V4L2 control item ID for each control item, the valid value range of parameters, the control response time, and whether real-time adjustment is supported. The V4L2 control item ID for each control item is 4 bytes, the valid value range of parameters (including minimum, maximum, and step size) is 2 bytes per item, the control response time is 2 bytes per item in milliseconds, and whether real-time adjustment is supported is 1 byte per item.

[0097] During the initialization of v4l2_device in the V4L2 driver, the hardware registers of the physical camera, such as the sensor model register and ISP capability register, are automatically read. Combined with the driver configuration parameters, a device capability cache containing all the information required by the above enumeration commands is generated, avoiding repeated reading of hardware each time enumeration occurs and improving enumeration efficiency.

[0098] This embodiment establishes a collaborative device discovery process, specifically including:

[0099] When the scan is triggered, the OH application calls the CameraManager.getCameraDevices() interface. The CameraFramework first checks whether the obtained logical device node path is valid. If it is valid, it triggers the V4L2 device scan; otherwise, it performs the traditional distributed device scan.

[0100] For device information acquisition, the V4L2 device adaptation unit of OH sends the ioctl command VIDIOC_OH_ENUM_DEV_INFO to the V4L2 driver through the logical device node path; after receiving the command, the V4L2 driver reads the information from the device capability cache, fills the struct oh_v4l2_dev_info structure, and returns the response to the OH framework through ioctl.

[0101] For multi-stream and control capability acquisition, the OH V4L2 device adapter unit continues to send VIDIOC_OH_ENUM_STREAM_CAP and VIDIOC_OH_ENUM_CTRL_CAP commands to obtain the camera's multi-stream concurrency capability and control function capability, which are used for subsequent parameter negotiation and control function mapping.

[0102] In distributed device registration, the OH V4L2 device adapter unit automatically converts the struct oh_v4l2_dev_info structure into the OH CameraDeviceInfo format, mapping the fields one by one. For example, the device name of oh_v4l2_dev_info is mapped to the deviceName field of CameraDeviceInfo, and the device is registered in the OH distributed device list and marked as directly accessible.

[0103] For multi-device collaborative scanning, if the OH application requests multi-camera collaboration, such as synchronous acquisition by master and slave cameras, the OH DeviceManager sends a multi-device enumeration request to the V4L2 driver through the V4L2 device adapter unit. The V4L2 driver returns information on all bound physical cameras, such as the master camera / dev / oh_video_1001 and the slave camera / dev / oh_video_1002. The OH DeviceManager automatically establishes distributed collaborative relationships between cameras, such as master / slave identifiers and synchronization clock IDs, to complete multi-device discovery and enumeration.

[0104] S4 and OH multi-stream parameter negotiation and V4L2 format configuration collaboration: A new V4L2 stream adaptation submodule has been added to the StreamManager of OH Camera HAL 3.0. This module deeply integrates OH's stream scheduling logic with V4L2's format configuration requirements.

[0105] Convert the streaming configuration requests built by the OH application using StreamConfig.Builder into the following format, specifically including:

[0106] Basic stream information includes stream ID, stream type, and stream priority defined by the OH framework. The stream ID is a unique identifier assigned by the OH framework. The stream type is 1 byte, such as preview, photo, and recording. The stream priority defined by the OH framework has 3 levels, such as 3 levels for preview, 2 levels for photo, and 1 level for recording.

[0107] Format and resolution parameters include OH request format identifier, target resolution, and frame rate. The OH request format identifier is 4 bytes, and the frame rate is 2 bytes in fps.

[0108] Buffer requirements include the number of 1-byte buffers, the minimum size of 4-byte buffers, and a preference for 1-byte memory types, such as ION memory.

[0109] Stream combination compatibility pre-verification: Based on the acquired multi-stream concurrency capability, it verifies whether the application requests the stream combination is supported. If not, it automatically generates alternative solutions such as reducing the recording resolution to 1080P and pops up a parameter adjustment prompt to the application.

[0110] Stream parameter caching caches the final stream configuration parameters after negotiation, which are used for subsequent buffer initialization and parameter synchronization during data transmission.

[0111] The core structure `v4l2_format` of the V4L2 driver adds extended fields for OH stream attributes, including stream priority, OH format identifier, buffer binding identifier, and format conversion channel ID. The stream priority is 1 byte, mapping to the OH stream priority and used for buffer queue scheduling; the OH format identifier is 4 bytes, recording the original format request of the OH application and used for format conversion verification; the buffer binding identifier is 4 bytes, associated with the OH's ION memory handle, used for two-way buffer binding; and the format conversion channel ID is 4 bytes, recording the identifier of the hardware-accelerated conversion channel, used for format conversion during data transmission. Simultaneously, a new stream parameter negotiation submodule is added to the V4L2 driver. This module supports parsing OH stream configuration requests and dynamically adjusting driver parameters, with its core logic coordinating with the OH stream scheduling logic.

[0112] This embodiment constructs a bidirectional parameter negotiation process, specifically as follows:

[0113] When a stream configuration request is sent, the OH's StreamManager encapsulates the standardized stream configuration request into a structoh_v4l2_stream_cfg structure containing all the configuration parameters of the streams and the stream combination relationships. This structure is then sent to the V4L2 driver via the ioctl command VIDIOC_OH_SET_STREAM_CFG (command code 0x8004). The command code for VIDIOC_OH_SET_STREAM_CFG is 0x8004.

[0114] For driver parameter adaptation and optimization, after receiving a request, the V4L2 stream parameter negotiation submodule queries the format support list in the device capability cache and compares the format identifier of the OH request with the native V4L2 format code. If the formats are consistent, such as the OH request for H.264 natively supported by V4L2, the format identifier is directly recorded without conversion. If the formats are inconsistent, such as the OH request for NV21 natively supported by V4L2 for YUYV, the hardware acceleration conversion support identifier is checked. If supported, the format conversion rule and the corresponding hardware conversion channel ID are recorded. If not supported, the most compatible format closest to the OH request format is selected from the format support list, and a format adjustment suggestion is returned. Based on the hardware capabilities of the physical camera, the frame rate parameter of the OH request is verified. If the requested frame rate is not greater than the maximum hardware frame rate, it is directly adopted; otherwise, the frame rate is adjusted to the maximum hardware frame rate, and the adjusted frame rate information is returned. At the same time, the resolution is verified to be in the support list. If not, a list of supported resolutions is returned for the OH framework to choose from.

[0115] Map the priority of the OH stream to the scheduling priority of the V4L2 buffer queue. For example, map OH priority 3 to V4L2 scheduling priority 90, and OH priority 2 to V4L2 scheduling priority 80, etc., to ensure that the buffer queue of the preview stream is scheduled first and avoid multi-stream data conflicts.

[0116] Calculate the optimal number of buffers based on the stream's resolution, frame rate, and memory bandwidth; the optimal number of buffers is ceil(frame rate). The sum of 20ms / 1000ms and 2, where 2 represents the two reserved buffers. For example, if the preview stream is 30fps, then ceil(30... 20 / 1000)=6, the final number of buffers is 6+2=8 blocks, ensuring no data overflow and optimal memory usage.

[0117] Based on the maximum total bandwidth limit of the stream combination, bandwidth quotas are allocated to each stream, such as 40% for the preview stream and 60% for the recording stream, to avoid frame rate drops caused by insufficient bandwidth when multiple streams are running concurrently; the V4L2 driver encapsulates the final stream configuration result into a struct oh_v4l2_stream_cfg_resp structure, which contains the actual resolution, frame rate, format, number of buffers, format conversion method, stream priority mapping result, and bandwidth quota for each stream, and returns it to OH's StreamManager via ioctl response;

[0118] Based on the feedback, OH's StreamManager initializes the Stream Pipeline's stream channels, buffer management queue, and data distribution logic, and synchronizes the negotiated parameters to OH's ION memory manager and CameraControl module, preparing for subsequent memory initialization and control function interaction.

[0119] S5, cross-framework shared memory pool construction and two-way buffer binding.

[0120] S5.1. A new V4L2 shared memory adapter unit is added to OH's ION memory manager. This unit deeply integrates OH's ION memory management logic with the V4L2 shared memory mechanism.

[0121] The V4L2 shared memory adapter unit uses the Linux mmap() system call to directly map the physically contiguous memory region created by the V4L2 driver to the user space of the OH container, generating an OH standard ION memory handle BufferHandle. The handle contains information such as memory address, size, format, and permissions.

[0122] The core attributes of the V4L2 buffer, including the buffer queue ID, stream ID, format conversion channel ID, and timestamp offset, are stored in the extended field `priv` of `BufferHandle`, enabling the OH framework to directly manage the V4L2 buffer. Based on OH's distributed permission model, an Access Control List (ACL) is added to the ION memory handle, allowing only the OH application process that initiated the camera request to access the memory area, and prohibiting other applications within the container from unauthorized access. The shared memory area is locked through the `mlock()` system call to prevent it from being swapped to disk by the Linux kernel, ensuring real-time data transmission. At the same time, the exit event of the OH application is monitored, and memory mapping is automatically unmapped and handle is reclaimed.

[0123] S5.2. A new OH shared memory pool is added to the memory allocation module of the V4L2 driver. This memory pool is built based on DMA and is deeply adapted to the ION memory management requirements of OH. Specifically:

[0124] The total size of the memory pool is (the size of a single buffer for all streams) (Number of buffers) 1.2 (20% redundancy reserved), where the size of a single buffer is the resolution width. High resolution The number of pixels in bytes, such as the single buffer size for 4K YUYV format, is 3840. 2160 2 = 15.36MB;

[0125] The shared memory pool is divided into independent buffer blocks according to the stream ID. Each buffer block contains all the buffers of the corresponding stream and is labeled with the stream ID, priority, and format conversion identifier. For example, the preview stream buffer block contains 8 buffers. The header of each buffer stores metadata including data length, timestamp, acquisition status, and format identifier.

[0126] The dma_alloc_coherent() function requests physical contiguous memory from the DMA controller, generates kernel virtual addresses and physical addresses, and then maps the physical memory to the user space of the V4L2 driver through the v4l2_mmap() interface of V4L2, thus realizing memory sharing between kernel mode and user mode.

[0127] The newly added buffer binding submodule binds buffer blocks in the shared memory pool to the V4L2 buffer queue v4l2_buffer_queue. Each stream corresponds to an independent buffer queue, which stores the physical address, metadata address, and status of the buffer, where the status is idle, ready, or processing.

[0128] After the S5.3 and V4L2 drivers create the shared memory pool, they use the pgd_offset(), pud_offset(), pmd_offset(), and pte_offset_kernel() functions to traverse the Linux kernel page table and find the page table entry (PTE) corresponding to the shared memory region.

[0129] Modify the page table entry attribute and set the _PAGE_SHARED flag, which allows the kernel-mode V4L2 driver and the user-mode OH container to share the page table entry, avoiding page table copying; the V4L2 driver uses the ioctl command to synchronize the physical address, page table entry address, and memory size of the shared memory to the OH's ION memory manager; the OH's ION memory manager calls the mmap() function to map the physical address of the shared memory to the OH container's user space. During the mapping process, the page table entry of the V4L2 driver is reused to achieve one page table and dual-space access;

[0130] After mapping is complete, the OH's ION memory manager generates a BufferHandle, where the virAddr field is set to the mapping address of the container's user space, the phyAddr field is set to the physical address of the shared memory, the size field is set to the buffer size, and the priv field stores the V4L2 buffer queue ID and the format conversion channel ID. The msync() function is used to synchronize the memory views of the kernel space and the user space to ensure that the read and write operations of memory by both sides are consistent and to avoid data inconsistency problems.

[0131] S5.4 In this embodiment, a two-way binding between shared memory and buffers is established. After the parameter negotiation is completed, the V4L2 driver requests physical contiguous memory through dma_alloc_coherent() according to the negotiation result containing the number and size of buffers, creates an OH shared memory pool, and generates a struct oh_v4l2_shm_info structure containing the base address of the memory pool, the total size, the distribution of each buffer block, and the page table entry address.

[0132] The OH's ION memory manager sends a memory mapping request to the V4L2 driver via the ioctl command VIDIOC_OH_GET_SHM_INFO with command code 0x8005. The request contains the process address space identifier of the OH container, namely the mm_struct pointer.

[0133] After receiving the request, the V4L2 driver executes the page table sharing process, mapping the shared memory pool to the user space of the OH container; after the OH's ION memory manager completes the mapping, it generates an ION memory handle BufferHandle and synchronizes the handle to the OH's StreamManager;

[0134] The buffer binding submodule traverses each buffer block of the shared memory pool, writes the physical address and metadata address of each buffer into the corresponding V4L2 buffer queue v4l2_buffer_queue, and stores the ION memory handle address of the OH in the userptr field of the v4l2_buffer structure, thereby realizing the association between the V4L2 driver and the OH buffer.

[0135] StreamManager binds the ION memory handle to the StreamBufferQueue of each stream. The buffer queue of each stream directly points to the corresponding buffer block in the shared memory pool. When the OH application requests a buffer through the Surface interface, it directly obtains the ION memory handle from this queue without allocating a local buffer, thus achieving one-time allocation and bidirectional sharing.

[0136] The V4L2 driver and the OH framework each write initial information to the metadata area of ​​the buffer. Specifically, the V4L2 driver writes the format identifier, stream ID, and priority, while the OH framework writes the buffer handle ID, access permissions, and data consumption callback function pointer.

[0137] S6, multi-stream data zero-copy transmission and format conversion collaboration.

[0138] S6.1 By adding a V4L2 data receiving unit to the StreamConsumer module of OH, the streaming data processing logic of OH and the DMA transfer mechanism of V4L2 are deeply integrated.

[0139] Image data in shared memory can be read through the virAddr field of the ION memory handle, without copying data from the V4L2 driver user space, thus achieving zero-copy transfer from the V4L2 driver to the OH application;

[0140] If the buffer data is in the native V4L2 format and needs to be converted to the format required by the OH application, the format conversion is performed in shared memory through OH's hardware acceleration framework, such as OH's GPU acceleration module and VA-API adaptation module. During the conversion process, no data migration is required, only the data storage format is modified.

[0141] Read the format conversion channel ID from the priv field of the ION memory handle, call the corresponding hardware acceleration interface such as vaPutImage(), pass in the shared memory address, format conversion rules, and resolution parameters, and trigger hardware conversion; after conversion, check whether the data format meets the requirements of the OH application. If it does, mark the buffer as consumable; otherwise, trigger software downgrade conversion.

[0142] Based on the stream ID and priority in the buffer metadata, the data is distributed to the corresponding OH application stream processing channel, such as sending the preview stream data to the UI rendering module SurfaceFlinger, sending the photo stream data to the image encoding module ImageCodec, and sending the recording stream data to the video encoder MediaCodec.

[0143] After the OH application finishes consuming the data, it marks the buffer as free through the BufferQueue::queueBuffer() interface and notifies the V4L2 data receiving unit. The receiving unit then uses the ioctl command to send the ID of the free buffer back to the V4L2 driver, triggering buffer reclamation and reuse.

[0144] S6.2. Add an OH stream priority scheduling unit to the buffer queue management module driven by V4L2 to adapt to the OH stream priority scheduling logic.

[0145] Preemptive priority scheduling is adopted, which prioritizes the buffer queues of high-priority streams based on their priority. For example, a DMA transfer request can preempt the transmission channel of the recording stream buffer queue with a priority of 70, which is the preview stream buffer queue with a priority of 90, to ensure a stable preview stream frame rate.

[0146] Modify the DMA transfer method of the V4L2 driver to support writing data collected by physical cameras into the target buffer block of the shared memory pool. During the transfer process, the scatter-gather function of the DMA controller is used to achieve continuous transfer of multiple buffers, thereby improving the transfer efficiency.

[0147] Configure the DMA controller to trigger an independent interrupt after data writing is complete. The interrupt handler function quickly updates the buffer metadata, sets the data ready flag and write timestamp, and notifies the OH's V4L2 data receiving unit.

[0148] By monitoring the free buffer IDs returned by the OH framework, the corresponding buffer is moved from the processing queue back to the free queue, waiting for the next DMA transfer, thus realizing the circular reuse of the buffer and avoiding the overhead of repeated allocation.

[0149] S6.3 When the OH application calls the CameraCaptureSession.startCapture() interface, the OH StreamManager sends a stream start request to the V4L2 driver via the ioctl command VIDIOC_OH_START_STREAM with command code 0x8006. The request contains the IDs of all streams and the buffer queue IDs.

[0150] After receiving the request, the V4L2 driver initializes the transmission parameters of the DMA controller, takes the address of the camera sensor data register as the source address, the physical address of the shared memory pool buffer block as the destination address, the transmission length as the size of a single buffer, and starts buffer queue scheduling.

[0151] The physical camera acquires image data at a negotiated frame rate, and the data is written to the sensor's data register in real time; the V4L2-driven DMA controller writes the data in the register directly to the target buffer block of the shared memory pool according to the buffer queue order, without CPU intervention during the transmission process.

[0152] After the DMA transfer is complete, an independent interrupt is triggered. The interrupt handler function of the V4L2 driver writes the data length, acquisition timestamp, data checksum, and acquisition status to the metadata area of ​​the buffer block. In the acquisition status, 0 indicates idle and 1 indicates ready. A data ready message is sent to the V4L2 data receiving unit of the OH via netlink. The message contains the stream ID, buffer ID, and metadata address. After receiving the notification, the V4L2 data receiving unit of the OH reads the buffer metadata, verifies the CRC32 checksum, and ensures the integrity of data transmission. It checks the format conversion flag in the ION memory handle. If conversion is required, it calls the hardware acceleration module to perform the format conversion in shared memory. The processed data is distributed to the corresponding OH application stream processing channel according to the stream ID. For example, preview stream data is sent to the UI rendering module via Surface::queueBuffer() to display the preview screen in real time.

[0153] After the OH application finishes consuming the data, it calls BufferQueue::dequeueBuffer() to mark the buffer as free. The OH's V4L2 data receiving unit uses the ioctl command VIDIOC_OH_RELEASE_BUFFER (command code 0x8007) to send the free buffer ID back to the V4L2 driver. The V4L2 driver's buffer queue scheduling unit moves the buffer back to the free queue, waiting for the next DMA transfer, thus achieving circular reuse.

[0154] The V4L2 OH stream priority scheduling unit monitors the buffer queue status of each stream in real time. When a high-priority stream has a data transmission requirement, it prioritizes the allocation of DMA transmission channels and CPU resources. Low-priority streams will only transmit when the high-priority streams have no transmission tasks, ensuring stable frame rates when multiple streams are running concurrently.

[0155] S7 and OH advanced control functions are dynamically mapped to V4L2 control items.

[0156] S7.1, Based on the OH camera framework control function specification and the V4L2 control item standard, namely V4L2 ControlAPI, a dynamically extensible mapping dictionary is established. The dictionary is stored in XML format in the / etc / v4l2 / oh_ctrl_map.xml configuration file of the V4L2 driver, and supports runtime dynamic loading. The core mapping rules include:

[0157] The basic control item mapping shows that the OH autofocus (AF) function corresponds to the V4L2_CID_FOCUS_ABSOLUTE control item in V4L2, with a control ID of 0x009a090a. The mapping entry includes the OH parameter range [0, 100], where 0 represents closest focus and 100 represents infinity. The V4L2 parameter range [0, 1023] represents the actual range supported by the camera hardware. The parameter mapping formula is: V4L2 parameter value = OH parameter value. The result of 10.23 is then rounded down; supported operation types include SET, GET, and REALTIME adjustment.

[0158] The OH's Auto Exposure (AE) function corresponds to the V4L2_CID_EXPOSURE_AUTO control item in V4L2, with a control ID of 0x009a0901. The mapping entries include the correspondence between AE modes such as Auto, Manual, Shutter Priority, and Aperture Priority. The OH's White Balance (AWB) function corresponds to the V4L2_CID_AUTO_WHITE_BALANCE control item in V4L2, with a control ID of 0x009a090c. The mapping entries include the correspondence between AWB modes such as Auto, Cloudy, Sunny, and Fluorescent.

[0159] Extended control item mapping includes:

[0160] The OH HDR function corresponds to the V4L2 custom control item V4L2_CID_OH_HDR_MODE, with a control ID of 0x80000001. The mapping entries include OH HDR modes such as Off, Auto, Standard, and Enhanced. The V4L2 HDR mode control values ​​are 0, 1, 2, and 3. The dependency condition is that the physical camera ISP supports HDR synthesis. The degradation strategy is to automatically turn off the HDR function and return to normal mode if the ISP does not support HDR.

[0161] The digital zoom function of OH corresponds to the custom control item V4L2_CID_OH_DIGITAL_ZOOM of V4L2, with a control ID of 0x80000002. The mapping entry contains parameter mappings for zoom magnification ranges from 1.0x to 10.0x.

[0162] The distributed control item mapping ranges from OH multi-camera collaboration to V4L2 distributed control items. The master-slave camera synchronous focusing function of OH corresponds to the custom control item V4L2_CID_OH_DISTRIBUTED_FOCUS in V4L2, with a control ID of 0x80000003. The mapping entries include a list of collaborative camera IDs, such as the master camera ID and slave camera ID. The synchronous control command format is the focus parameter and the synchronous clock stamp. Each mapping entry contains a support status field, including fully supported, partially supported, and not supported, which is dynamically updated by the V4L2 driver during initialization based on hardware capabilities.

[0163] S7.2 Add a V4L2 control adapter unit to OH to integrate the control logic of OH with the control item operation of V4L2.

[0164] The control requests from the OH application are converted into a struct oh_v4l2_ctrl_req structure, which contains the control item type, V4L2 control item ID, parameter value, request type, and timeout. The control item type is 1 byte and includes basic control items, extended control items, and distributed control items. The V4L2 control item ID is 4 bytes. The parameter value is 4 bytes or 8 bytes, depending on the control item type. The request type is 1 byte and includes setting 0x01, querying 0x02, and synchronizing 0x03. The timeout is 2 bytes and the unit is ms.

[0165] At startup, the mapping dictionary of the V4L2 driver is loaded and the mapping rules are cached to avoid repeated parsing for each request; if the mapping dictionary is updated, the cache is refreshed; the control response of the V4L2 driver is received, the V4L2 parameter value is converted into a format that the OH application can recognize, such as converting the V4L2 focus parameter 512 into OH 50, and the application is notified through a callback.

[0166] If the V4L2 driver return function is not supported, the degradation strategy defined in the mapping dictionary will be executed, such as turning off HDR, using the default focus mode, and returning a degradation prompt to the application.

[0167] S7.3. Add an OH control instruction parsing unit to the V4L2 driven control subsystem to adapt to the OH control logic.

[0168] During driver initialization, the mapping dictionary file is read, the mapping rules are parsed and cached in kernel-mode memory, and the dictionary can be dynamically updated via the debugfs interface. The struct oh_v4l2_ctrl_req request of OH is parsed, the corresponding operation is performed according to the V4L2 control item ID, the native v4l2_ctrl_set() or v4l2_ctrl_get() function of V4L2 is called, the camera hardware registers are operated, the newly added control implementation function such as oh_v4l2_ctrl_set_hdr_mode() is called to configure the corresponding registers of ISP or sensor, and the control commands are synchronized to the V4L2 driver of the cooperating camera to ensure multi-camera control synchronization.

[0169] The control status of the surveillance camera hardware, such as focus completion and exposure adjustment, is fed back to the OH framework through the oh_v4l2_ctrl_resp structure. It includes the execution result, current parameter value, status flag, and response time. The execution result is success or failure, the current parameter value is in V4L2 format, the status flag is incomplete, completed, or needs to be downgraded, and the response time is 2 bytes in milliseconds.

[0170] If a control operation fails, such as a hardware register write error, a specific error code such as -ENODEV or -EINVAL will be returned, along with detailed error information to facilitate troubleshooting.

[0171] S7.4 The OH application calls the CameraControl interface. The CameraControl's V4L2 control adapter unit queries the mapping dictionary to determine the corresponding V4L2 control item ID and parameter value, generates a standardized control request, and sends it to the V4L2 driver through the ioctl command VIDIOC_OH_SET_CTRL with command code 0x8008.

[0172] After receiving the request, the V4L2 OH control instruction parsing unit queries the mapping dictionary based on the V4L2 control item ID to confirm the control item type, parameter range, and dependencies. It checks whether the physical camera supports the control item (e.g., HDR depends on ISP support); if not, it returns a downgraded response. It calls the corresponding control function and configures the camera hardware registers, such as setting the ISP's HDR mode register to 1. If it's a real-time adjustment of the control item, it continuously monitors the hardware status until the control operation is complete. The V4L2 driver encapsulates the control result and current status into a `struct oh_v4l2_ctrl_resp` structure and returns it to the OH framework via an ioctl response. The OH V4L2 control adaptation unit converts the V4L2 parameter values ​​to OH format (e.g., converting HDR mode 1 to HDR_MODE_AUTO), notifies the application via `CameraControlCallback`, and updates the local control status to ensure consistency between the application and hardware states.

[0173] For distributed control requests, such as master-slave camera synchronous focusing, the OH control command parsing unit of the V4L2 driver sends the control command to the V4L2 driver of the cooperating camera. The cooperating driver performs the same control operation and feeds back the execution result. The master driver summarizes the results of all cooperating drivers and feeds them back to the OH framework in a unified manner to realize multi-camera control synchronization.

[0174] S8, Status Monitoring and Dynamic Optimization.

[0175] S8.1 Add a camera adaptation monitoring unit to the HiviewDFX monitoring module of OH, integrating OH's performance monitoring logic and V4L2 driver monitoring data.

[0176] The following key metrics are collected at 100ms intervals and stored in a circular buffer: performance metrics, resource metrics, and status metrics. Performance metrics include the actual frame rate of each stream, data transmission latency, control command response time, and format conversion time. The actual frame rate is the number of frames collected within 100ms. 10. Data transmission latency is the time difference from the completion of DMA transfer to the OH application receiving the data; control command response time is the time difference from the sending of the request to the feedback of the result; resource indicators include shared memory utilization, CPU utilization, buffer free rate, and DMA transfer bandwidth. Among them, shared memory utilization is the ratio of used size to total size, buffer free rate is the ratio of the number of free buffers to the total number of buffers, and DMA transfer bandwidth is the ratio of the total amount of data transferred in 100ms to 100ms; status indicators include format conversion power, control function execution success rate, number of buffer overflows, number of interrupt responses, and number of data verification failures.

[0177] It supports configuring indicator thresholds through OH's system configuration interface, such as setting the preview frame rate threshold to 25fps, triggering an alarm when it is lower than this value; setting the transmission latency threshold to 100ms, triggering an alarm when it is higher than this value; and setting the control response time threshold to 200ms, triggering an alarm when it is higher than this value.

[0178] When the metric exceeds the threshold, an optimization request is triggered, and an alarm notification is sent to the system administrator through the HiviewDFX alarm mechanism; the collected metric data is synchronized to the monitoring node of the V4L2 driver at a 1-second interval for joint optimization on the driver side.

[0179] A new debug node named oh_camera, / sys / kernel / debug / v4l2 / oh_camera, is added to the debugfs directory of the V4L2 driver. This node supports outputting the following log information for monitoring unit collection and troubleshooting: detailed process of device enumeration, parameter negotiation, data transmission, and control execution, including timestamps, operation types, results, and error codes; sensor frame rate, ISP load, DMA transfer error count, interrupt response latency, and hardware register status of the physical camera (interrupt response latency is the time difference from interrupt triggering to handler function execution); shared memory allocation details, CPU core binding status, interrupt channel usage status, and buffer allocation and reclamation records.

[0180] A new V4L2 adaptation optimization unit was added to the Camera Framework of OH. After receiving optimization requests from the monitoring unit, and combining the debugfs logs of the V4L2 driver, the performance bottleneck was located using the following rules:

[0181] If the preview frame rate is less than 25fps and the buffer idle rate is less than 30%, the bottleneck is insufficient buffer quantity; if the transmission latency is greater than 100ms and the CPU utilization is greater than 80%, the bottleneck is low CPU scheduling priority; if the control response time is greater than 200ms and the control execution success rate is less than 90%, the bottleneck is redundant control parameter negotiation process; if the format conversion time is greater than 10ms and the hardware acceleration conversion efficiency is less than 90%, the bottleneck is excessive load on the hardware conversion channel.

[0182] After identifying the type of performance bottleneck, select the corresponding built-in optimization strategy to perform optimization: Buffer optimization strategy: Send a request to the V4L2 driver to adjust the number of buffers, adding two buffers for the corresponding stream, such as increasing the number of preview stream buffers from 8 to 10, to improve data transmission continuity; CPU scheduling optimization strategy: Request the V4L2 driver to adjust CPU affinity, binding camera-related processes to idle CPU cores of the host and increasing process scheduling priority; Interrupt optimization strategy: Request the V4L2 driver to increase interrupt response priority and reduce interrupt response latency; Format conversion optimization strategy: Disable unnecessary format conversions, such as using the native format directly when the OH application supports the V4L2 native format, or switching to a backup hardware conversion channel; Control flow optimization strategy: Cache commonly used control parameters to avoid repeated hardware queries, simplify the control parameter negotiation process, and reduce the number of ioctl command interactions; Bandwidth optimization strategy: If DMA transfer bandwidth is insufficient, reduce the frame rate of low-priority streams and allocate more bandwidth to high-priority streams.

[0183] Optimize execution and verify results:

[0184] The optimization request is sent to the V4L2 driver via the ioctl command. The V4L2 driver executes the corresponding optimization operations, such as adjusting the number of buffers and CPU affinity. The monitoring unit continuously collects metrics. If the metrics recover to within the threshold after optimization, the optimization configuration is maintained. If they do not recover, other optimization strategies are selected from the strategy library and executed again until the metrics meet the target or all strategies have been tried. The effective optimization configuration is stored in the V4L2 driver's configuration file / etc / v4l2 / oh_optimize.conf, and is automatically loaded on the next startup to avoid duplicate optimizations.

[0185] Example optimization process:

[0186] The monitoring unit detected a preview frame rate of 22fps, which is lower than the threshold of 25fps, and a buffer idle rate of 20%, which is lower than the threshold of 30%, thus meeting the triggering conditions.

[0187] The bottleneck was identified as insufficient buffer size causing data transmission interruptions and a drop in frame rate.

[0188] Send a request to the V4L2 driver to adjust the number of buffers, increasing the number of preview stream buffers from 8 to 10;

[0189] After optimization, the number of preview frames captured within 100ms increased from 2.2 frames to 2.8 frames, the frame rate was restored to 28fps, and the buffer idle rate was 35%.

[0190] Write the preview stream buffer size of 10 to the optimization configuration file, and apply it automatically on the next startup.

[0191] S9. Anomaly Handling and Fault Recovery.

[0192] Based on the abnormal scenarios of the OH camera framework and V4L2 driver, the following abnormal types and corresponding fault levels are defined: Device abnormalities are classified as severe (physical camera disconnection, V4L2 driver crash, logical device node failure); resource abnormalities are classified as normal (shared memory overflow, buffer allocation failure, CPU / memory resource exhaustion, DMA transfer error); data abnormalities are classified as normal (data verification failure, format conversion failure, buffer data corruption); control abnormalities are classified as minor (control command execution failure, control parameters exceeding hardware limits, control response timeout); and container abnormalities are classified as severe (OH container crash, container permissions tampered with, SELinux policy failure).

[0193] A new V4L2 exception handling unit has been added to OH's Camera Framework to monitor, capture, and record exception information in a timely manner, such as timestamps, exception types, error codes, and field parameters. This includes monitoring the return values ​​of ioctl commands, memory operations, and flow control interfaces; triggering exceptions if error codes are returned; monitoring the status flags of buffer metadata and the exception feedback flags of the V4L2 driver; triggering exceptions if data corruption or device offline occurs; setting timeouts for control requests and data reception operations; triggering exceptions if timeouts are not completed; and executing corresponding processing logic based on the fault level. In the case of a severe exception, all streaming transmissions are immediately stopped, the current camera status (such as stream configuration and control parameters) is saved, and a response is sent to the application. When a device malfunctions and a callback is received, the user is prompted to turn off the camera. Simultaneously, a fault notification is sent to the V4L2 driver via netlink, triggering driver-side fault recovery. If the malfunction is a common one, automatic recovery is attempted, such as reinitializing the buffer or switching the format conversion channel. If recovery is successful, operation continues, and a temporary malfunction recovery callback is returned to the application. If recovery fails, degraded operation is implemented, such as reducing the frame rate or disabling advanced features. If the malfunction is minor, the malfunction information is recorded, operation continues, and periodic repair attempts are made, such as resending control commands, without affecting application usage. The malfunction information, including field parameters and driver logs, is reported to the system log server via the OH log reporting interface for troubleshooting.

[0194] A new OH fault recovery unit is added to the V4L2 driver. Upon receiving a fault notification from the OH framework, it stops DMA transfer, buffer queue scheduling, and interrupt response to prevent the fault from escalating. The core structure of the driver is reset to the initial state, releasing shared memory, buffers, interrupt channels, and other resources occupied by the exception. Recovery operations are performed according to the exception type. If it is a device exception, it attempts to reinitialize the physical camera and recreate the logical device node. If initialization fails, it returns a recovery failure. If it is a resource exception, it reallocates the shared memory pool and buffers and rebinds the CPU core and interrupt channel. If it is a DMA transfer error, it reconfigures the DMA controller parameters and restarts DMA transfer.

[0195] After the recovery operation is completed, a recovery result message is sent to the OH framework via netlink, which includes a success or failure indicator and resource configuration information after the restart.

[0196] During the preview process of the OH application, if the monitoring unit detects a data verification failure and triggers a data anomaly (a common fault), the V4L2 anomaly handling unit will capture the anomaly, record anomaly information such as timestamp, buffer ID, and checksum, attempt to switch to a backup format conversion channel, and reread the buffer data. The OH framework will send a data anomaly notification to the V4L2 driver via ioctl commands. Upon receiving this notification, the V4L2 OH fault recovery unit will mark the buffer as corrupted, remove it from the buffer queue, and allocate a new buffer to replace it. The OH framework will reread the new buffer data, and after successful verification, resume normal preview and return a temporary anomaly recovery callback to the application. If a serious fault such as a sudden disconnection of the physical camera occurs, and the V4L2 driver's OH fault recovery unit fails to reinitialize the device, it will return a recovery failure message to the OH framework. The OH framework will immediately stop streaming, save the current state, and return a camera disconnection callback to the application, prompting the user to check the hardware.

[0197] S10, Resource Release and Container Environment Cleanup.

[0198] When the OH application calls the CameraDevice.close() interface or exits, the Camera Framework performs resource release operations in the following order: It sends a stream stop request to the V4L2 driver via the ioctl command VIDIOC_OH_STOP_STREAM (command code 0x8009), containing the IDs of all streams and requesting the cessation of DMA transfers and buffer queue scheduling; it notifies the ION memory manager to unmap shared memory, release ION memory handles, and mark buffers as awaiting reclamation; it sends control commands to the V4L2 driver to reset the camera's control parameters to their default values; and it sends a resource release message to the V4L2 driver via netlink, containing the container CID, logical device node path, and a list of allocated resources, such as CPU core ID, interrupt channel number, and shared memory address.

[0199] Upon receiving a stream stop request, the V4L2 driver immediately halts DMA transfers and buffer queue scheduling, waiting for all currently processed buffers to complete their operations. Upon receiving a resource release message, it performs cleanup operations in the following order to ensure no resource leaks: It calls the `dma_free_coherent()` function to release the physically contiguous memory allocated by DMA and reclaims shared memory pool resources; it destroys all stream buffer queues and releases queue-related data structures; it calls the `device_destroy()` function to delete the ` / dev / oh_video_[CID]` logical node to prevent invalid node remnants; it removes the OH container's UID from the `video` user group using the `setgroups()` function and reclaims permissions such as `CAP_VIDEO_CAPTURE` using the `capset()` function; it uninstalls the corresponding SELinux / AppArmor security policy for the container to avoid security policy redundancy; it releases the allocated independent interrupt channel using the `free_irq()` function and unbinds the CPU core; it updates the resource quota record table, marking the released resources as free for use by other containers; and it resets the OH container adaptation extension field, device capability cache, and resource quota record table of the `v4l2_device` structure to their default values ​​to prepare for the next camera access.

[0200] After the V4L2 driver completes the cleanup operation, it sends a cleanup completion message to the OH framework via netlink. The OH framework checks whether local resources such as ION memory handles, buffer queues, and stream pipes have been released. If all is confirmed, it completes the camera shutdown process. If resources are not released, a forced reclamation mechanism is triggered to ensure a clean container environment.

[0201] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A containerized HarmonyOS camera high-efficiency access method based on cross-frame shared page tables, characterized in that, Specifically, the following steps are included: Using Linux as the host and OpenHarmony as the container, the V4L2 driver within the host generates camera capabilities. When the container starts, the host grants the container process access permissions to the V4L2 logical device node. When the container application initiates a request to open the camera, the container generates a resource request message and sends it to the host, triggering the V4L2 driver to create a container logical device node bound to the container ID, granting the container process camera-related permissions and access permissions to the container logical device node and shared memory, allocating an interrupt channel for the container camera data, and generating a resource request response to return to the container. The container application initiates a camera list query request. The container V4L2 device adaptation unit obtains the container logical device node, triggers the V4L2 driver to query the camera capability to form a device data packet, converts it into container device information, and the container registers the camera and marks it as a V4L2 collaborative device accordingly. The container application sends a stream configuration request, which is encapsulated by the container into a container stream configuration request and sent to the V4L2 driver. This triggers the V4L2 driver to match the format, determine the frame rate and resolution list, convert the container application priority into the host buffer scheduling priority, calculate the number of buffers, and send a stream configuration response to trigger the container to initialize the stream channel, buffer management queue, and data distribution logic. The V4L2 driver allocates shared memory and divides it into buffers corresponding to the data streams, maintaining a buffer state queue. The V4L2 driver associates the shared memory with its own buffer queue and binds it to the container handle. The container then attaches the shared memory handle to its own stream queue. The container application sends a request to start image acquisition. The container generates and sends a data stream start message to trigger the V4L2 driver to configure the DMA hardware. The image data acquired by the camera is written to shared memory via the DMA hardware. Then, a data ready notification is sent to trigger the container to distribute the image data to the corresponding processing module according to the stream identifier and priority. After processing, the corresponding buffer is marked as idle. The V4L2 driver adds it to the idle queue to complete the access.

2. The containerized HarmonyOS camera high-efficiency access method according to claim 1, characterized in that, The container sends a container stream configuration request containing basic stream information, format and resolution, buffer requirements, and stream combination relationships, triggering the V4L2 driver to match the format. Specifically, the V4L2 driver queries the format support list of the camera's capabilities, compares the requested format identifier with the V4L2 native format code, and records the format identifier if the formats match. Otherwise, if the host supports hardware-accelerated conversion, it records the format conversion rules and the corresponding hardware conversion channel ID. If the host does not support hardware-accelerated conversion, it selects the compatible format that is closest to the requested format.

3. The containerized HarmonyOS camera high-efficiency access method according to claim 1, characterized in that, The container application sends a camera control request. The container determines the V4L2 control item ID and parameter value, encapsulates the container application's function command, parameter value, operation type, and maximum waiting time into a container control request, and sends it to the V4L2 driver. After receiving the request, the V4L2 driver converts the parameter value according to the mapping conversion relationship, completes the configuration writing of the camera sensor and ISP hardware registers, and then encapsulates the operation execution result, actual hardware parameters, running status, and command time into response data and sends it back to the container. After receiving the response data, the container reverse-engineers the original hardware parameters into standard values ​​that the application can recognize, synchronously sends the operation result notification to the container application, and updates the locally stored camera operating status.

4. The containerized HarmonyOS camera high-efficiency access method according to claim 3, characterized in that, If it is a multi-camera collaborative control command, the V4L2 driver will synchronously distribute the container control request to other camera drivers to execute unified configuration, and then encapsulate the execution results of all cameras into response data and send it back to the container.

5. The containerized HarmonyOS camera high-efficiency access method according to claim 1, characterized in that, When a container application sends a request to turn off the camera or when the application process exits, the container generates a container data stream stop request based on all running data stream identifiers, sends the request to the V4L2 driver, synchronously notifies the container to delete the address mapping relationship of shared memory, reclaims the container memory management handle and marks the corresponding buffer as to be reclaimed, sends control instructions to the V4L2 driver to restore the camera hardware parameters to the default values, and sends a resource release request to the V4L2 driver. After receiving a request to stop the container's data stream, the V4L2 driver terminates hardware data transmission and buffer queue scheduling, waits for all ongoing data processing tasks to finish, and releases resources in sequence upon receiving a resource release request. Then, it sends a cleanup notification to the container. The container verifies whether all local resources have been released. After verification and release, it shuts down the camera. If there are any unreleased resources, it activates a forced reclamation mechanism.

6. The containerized HarmonyOS camera high-efficiency access method according to claim 1, characterized in that, When the V4L2 driver receives multiple container camera resource requests at the same time, it reads the priority and resource requirements of each container request, stores all requests in the container request queue and sorts them according to arrival time. Then, read the resource occupancy status table to obtain the real-time occupancy information of the camera, and determine the resource allocation method and form an arbitration result according to the arbitration configuration mode preset by the driver; According to the arbitration result, the container that received the authorization feedback was granted the camera resources, while other containers received a notification of failure to compete.

7. The containerized HarmonyOS camera high-efficiency access method according to claim 6, characterized in that, The process of determining the resource allocation method and forming the arbitration result based on the driver's preset arbitration configuration mode is as follows: In preemptive mode, all requests in the container request queue are traversed, and resources are allocated to high-priority containers first. If the resource occupancy status table shows that the camera is currently occupied by a low-priority container, a preemption notification is sent to the low-priority container to make it relinquish the camera. The container that receives the preemption notification stops image acquisition and saves its running status, reports to the host that the resource release is complete, and after the host confirms that the resources are cleared, the camera resources are allocated to a high-priority container. In fair scheduling mode, the arbitration module continuously reads the occupied time recorded in the resource occupancy status table. After the current occupied container's usage time expires, the resource is released, and then subsequent containers in the container request queue are scheduled in sequence.

8. The containerized HarmonyOS camera high-efficiency access method according to claim 1, characterized in that, When a container application needs multiple cameras to collect data synchronously, the container sends a batch device query request to the V4L2 driver. After receiving the request, the V4L2 driver returns the device file paths corresponding to all currently bound cameras. The container sets the master-slave identity and synchronization clock identity of each camera based on the returned information, and establishes a distributed collaborative relationship between multiple cameras.

9. The containerized HarmonyOS camera high-efficiency access method according to claim 1, characterized in that, When the container collects operational metrics data from the camera, it acquires frame rate, data transmission latency, response time to control commands, format conversion overhead, memory usage, processor load, buffer usage percentage, data transmission bandwidth, and success rate of various functions at fixed intervals. It uses a circular storage method to save the collected data, sets the normal threshold range for each metric according to the system configuration, and sends performance optimization requests and system alarm information when the operating values ​​exceed the limit range. At the same time, it periodically sends the monitoring data to the V4L2 driver monitoring node. The container adaptation and tuning unit determines the cause of the performance bottleneck based on the anomaly information and driver logs, and sends the corresponding optimization and adjustment instructions to the V4L2 driver. The driver receives and executes the instructions to complete the optimization. The container monitors changes in indicators, and records the current optimization configuration when the running data is within a reasonable range. If the indicators do not return to normal, the optimization strategies are activated sequentially until the problem is fixed.

10. The containerized HarmonyOS camera high-efficiency access method according to claim 1, characterized in that, When the container detects abnormal operation information from the camera, the abnormality handling unit inside the container obtains the call return status, buffer operation identifier and instruction execution timeout event, records the time of abnormality occurrence, fault type, error identifier and on-site operating parameters, and performs tiered processing according to three fault levels: severe, normal and minor. When a serious fault occurs, all data stream transmission is terminated, the current camera operating configuration is saved and the device fault information is pushed to the container application, and a fault alarm message is sent to the V4L2 driver. When a common fault occurs, priority is given to resetting the buffer and switching the image format path. If the repair is successful, normal business operation is restored and the application is notified that the fault has been resolved. If the repair fails, the operating indicators are reduced or the image function is stopped. When a minor fault occurs, only the fault record is retained. The business process continues to operate normally and the command is retried periodically. At the same time, the complete fault scene data is reported to the system log service. After receiving the container fault notification, the V4L2 driver closes the data transmission, queue scheduling and hardware interrupt response behavior, resets various internal operating data structures of the driver, reclaims the system resources occupied by the fault, and completes the repair operations of hardware device reinitialization, memory block reallocation and transmission path reconfiguration according to the fault type. After processing, the repair results and updated resource configuration data are returned to the container.

Images