A cross-environment camera sharing method based on hardware access slicing

Abstract

The application discloses a cross-environment camera sharing method based on hardware access slices, and comprises the following steps: initializing a permission verification module when a host is started, creating a first shared memory pool, starting a conflict arbitrator to establish a concurrent request scheduling mechanism, detecting hardware information by a hardware capability slice module, splitting a management logic slice, and completing capability information adaptation and delivery by a host adapter; deploying a permission proxy module after a container is started, forwarding permission requests to the host verification module, completing shared memory mapping by a container adapter, interfacing with the conflict arbitrator and adapting the hardware capability format of the container environment; completing camera permission verification after an application is started, querying unified hardware capability to build business data flow configuration, and initiating a resource access request; allocating a logical slice by the hardware capability slice module, determining a hardware benchmark configuration to start the camera, binding the slice and the shared memory, and delivering the slice to enter the collection stage; and executing control instructions after arbitration by the arbitrator, so that the multi-application collaborative sharing of the camera is realized.

Description

Technical Field

[0001] This invention belongs to the field of computer software development technology, and specifically relates to a cross-environment camera sharing method based on hardware access slices. Background Technology

[0002] In scenarios where a host system runs third-party system applications using container technology, the physical camera, as a core hardware resource, typically only supports exclusive access by a single client in its driver layer. Furthermore, there are significant differences between the camera frameworks of the host system and the container system. For example, the host system is based on a distributed camera framework, while the container system is based on a pipelined camera model. This results in the two types of applications being unable to use the camera efficiently at the same time.

[0003] In existing technologies, time-sharing multiplexing solutions, such as polling to switch clients, are often used. The latency generated by this method is usually greater than 100ms. Alternatively, solutions that modify the driver layer, such as adding support for multiple clients, are used. However, this approach is limited by the constraint that the driver source code cannot be modified. Therefore, the former can lead to performance issues such as preview stuttering and a sharp drop in recording frame rate, while the latter is difficult to solve the problem of cross-system protocol incompatibility.

[0004] Furthermore, existing solutions suffer from the following problems: First, they lack a unified permission verification mechanism, which can easily allow container applications to bypass host permission controls; second, they lack logic for exposing consistent device capabilities, leading to inconsistencies in resolution or frame rate information obtained by applications on both systems; and third, they lack a refined conflict arbitration process, handling only resolution conflicts and ignoring scenarios such as bandwidth and mutually exclusive resources. These issues can result in security vulnerabilities, functional abnormalities, or inefficient arbitration for application access. Summary of the Invention

[0005] In view of this, the present invention provides a cross-environment camera sharing method based on hardware access slicing. By using hardware capability logical slicing and cross-system protocol conversion, it realizes concurrent sharing of physical camera resources and efficient arbitration of conflicts. It is suitable for dual-system applications that simultaneously perform camera preview, photo taking, and video recording.

[0006] This invention provides a cross-environment camera sharing method based on hardware access slices, which specifically includes the following steps:

[0007] When the host starts, the permission verification module receives permission requests from cross-environment applications, allocates the first shared memory pool to provide a zero-copy data channel between the application and the camera; the conflict arbitrator resolves resource access conflicts, and the capability slicing module collects camera information and encapsulates it into a device capability structure; the host adapter converts the device capability structure into its standard structure to provide hardware capabilities for the host application. When the container starts, the permission proxy module forwards permission requests to the permission verification module, and the container adapter maps the first shared memory pool, converting the device capability structure into its standard structure to provide hardware capabilities for the container application.

[0008] After the application starts, it directly or through the permission proxy module sends a camera permission request to the permission verification module. Upon successful verification, the permission status is recorded and the authorization result is returned to the application. The application obtains hardware capabilities through the corresponding adapter, constructs a session configuration, and initiates a camera resource access request containing the session configuration. This request is converted into a cross-environment standard resource request by the corresponding adapter and sent to the conflict arbitrator. The conflict arbitrator calculates the request priority and adds the request to the scheduling queue accordingly. The hardware capability slicing module allocates logical slices to the application based on the request and labels the slice configuration. The allocation result is sent to the conflict arbitrator. After determining the hardware baseline configuration, the camera is started, and the logical slice is bound to the first shared memory pool. The conflict arbitrator issues the logical slice to the application through the corresponding adapter, and the application starts the business data flow according to the logical slice parameters.

[0009] The camera captures images, and the hardware capability slicing module processes the raw output images according to the slicing configuration, writes them to the first shared memory pool, and notifies the application to read them. The application's control commands are sent from the first shared memory pool to the conflict arbitrator for arbitration. After the arbitration is successful, the hardware capability slicing module executes hardware control to complete the concurrent access and collaborative control of multiple applications.

[0010] Furthermore, the first shared memory pool includes a control instruction block, a raw data block, and a metadata block. The control instruction block uses a circular buffer structure to store hardware control instructions, the raw data block stores the raw images output by the camera, and the metadata block stores auxiliary information for each frame of the image.

[0011] Furthermore, the conflict arbitrator uses a blocking queue to cache and schedule concurrent requests from multiple runtime environments. The queue capacity supports simultaneous requests from multiple applications, and a busy prompt is returned when the queue is full.

[0012] Furthermore, the application constructs the session configuration by acquiring hardware capabilities through the corresponding adapter as follows: distinguishing business usage scenarios, and building data streams corresponding to preview, shooting, and video recording respectively; setting the running parameters of the data streams according to the resolution, frame rate, and image data format supported by the hardware; binding image rendering, image storage, and video encoding business output components to complete the association configuration between the data streams and the output carriers; merging multiple data streams with different purposes to form the camera usage session configuration, including preview data stream configuration and shooting data stream configuration, specifically including screen size, frame rate, data format, output binding components, and business usage type.

[0013] Furthermore, the method of converting the request into a cross-environment standard resource request via the corresponding adapter is as follows:

[0014] The host adapter and container adapter intercept camera resource access requests initiated by applications within their respective environments, parse the built-in session configuration and business template information, distinguish the actual usage scenarios, extract the business flow type, screen resolution, running frame rate and data format, read the application identification information and determine the application's category and permission level; complete standardized encapsulation according to the globally unified message format, generate cross-environment standard resource request messages, and uniformly label the request source, application ID, application type, independent request number, and the screen specifications, running frame rate, data format, output binding identifier and permission level corresponding to each business flow, distinguishing the operating environment only by the source field.

[0015] Furthermore, the hardware capability slicing module allocates logical slices and labels the slice configuration according to the application request in the following way:

[0016] Step 4.1: Traverse the idle logical slice library, match logical slices with the same resolution as the request, frame rate not less than the request frame rate and the smallest difference between the two, and data format consistent with the request. If the match is successful, mark the logical slice as pending allocation and execute step 4.4. If the match fails, execute step 4.2.

[0017] Step 4.2: Filter idle logical slices with a resolution greater than the requested resolution, a frame rate not less than the requested frame rate, and a consistent data format. Prioritize selecting the logical slice with the smallest resolution difference. If a match is found, mark the slice and configure it for resolution downsampling, then proceed to step 4.4; if a match fails, proceed to step 4.3.

[0018] Step 4.3: Match idle logical slices with the same resolution, frame rate greater than the requested frame rate, and consistent data format. If the match is successful, mark the slice configuration as frame rate limiting or frame skipping and proceed to step 4.4. If the match fails, return a resource allocation failure message.

[0019] Step 4.4: The hardware capability slicing module assigns a unique logical slice to each data stream and updates the slice occupancy status.

[0020] Furthermore, the binding logic slice and the first shared memory pool are configured as follows:

[0021] The first shared memory pool includes control instruction blocks, raw data blocks, and metadata blocks. Memory blocks of corresponding capacity are allocated according to the logical slice resolution; logical slices with resolutions greater than a threshold are allocated memory blocks of a capacity greater than the threshold, and logical slices with resolutions lower than the threshold are allocated memory blocks of a capacity less than the threshold. Region isolation is achieved through memory address offsets. Each logical slice is independently bound to a dedicated metadata block to store frame auxiliary information, which is associated with the original image through a logical slice identifier. The memory block address information corresponding to the logical slice is recorded.

[0022] The host adapter maps the memory blocks corresponding to the logical slices in the first shared memory pool to the memory space of the host application; it points the buffer of the application's screen rendering component to this memory space; the hardware output data is processed by the hardware capability slicing module and written to the corresponding memory block of the first shared memory pool; the application reads the data in real time through the mapped address.

[0023] The container adapter maps the first shared memory pool to the container runtime environment and determines the target memory block based on the logical slice address offset; it allocates an image receiving buffer for the container application and establishes an address mapping relationship between it and the target memory block; it injects the bound image receiving buffer into the application's image receiving component so that hardware output data can be directly delivered to the application; the hardware output data is processed by the hardware capability slicing module and written to the shared target memory block, which the container application can directly obtain through the mapping relationship.

[0024] The hardware capability slicing module writes the frame auxiliary information corresponding to each frame of image into the metadata block bound to the logical slice; the adapter associates the original image with the metadata through timestamps, converts it into the standard format of its respective runtime environment, and then synchronously calls it back to the application.

[0025] Furthermore, the hardware capability slicing module processes the original image it outputs according to the slicing configuration, writes it to the first shared memory pool, and notifies the application to read it. Specifically, the method is as follows:

[0026] The hardware capability slicing module receives raw images output by the camera driver in a loop through an independent thread. It performs preprocessing on each frame of data, including data verification, timestamp generation, and metadata extraction. Data verification checks whether the frame data length matches the baseline resolution and discards abnormal data. A globally unique timestamp is generated for each frame of data for data synchronization and association. Metadata extraction extracts auxiliary information from the frame.

[0027] The hardware capability slicing module performs corresponding processing on logical slices according to their slicing configurations, enabling the distribution of single-channel hardware data to multiple logical slices. When the slice parameters are completely consistent with the hardware baseline configuration, the original image is written to the shared memory block bound to the logical slice, the data is marked as valid, and the timestamp is updated. When the slice resolution is lower than the hardware baseline configuration, the original image is scaled by the hardware image processor, converted to the slice target resolution, written to the corresponding shared memory block, and marked as valid. When the slice frame rate is lower than the frame rate of the hardware baseline configuration, a frame counting skipping strategy is used to control the output frequency, and the reserved frames are written to the shared memory block and marked as valid.

[0028] Frame auxiliary information is written to the metadata block bound to the logical slice to ensure consistency with the timestamp of the original image; when the data validity flag is set, the adapter triggers a data ready callback to the application, and the application reads the screen from the shared memory block and performs rendering.

[0029] Furthermore, the application's control commands are sent from the first shared memory pool to the conflict arbitrator for arbitration, specifically in the following manner:

[0030] The first control instruction is established, which includes instruction type, parameters, source, application ID, associated slice, timestamp and verification information. Supported instructions include flash control, autofocus, zoom adjustment and exposure compensation. When the application initiates control, the adapter intercepts and encapsulates them into the first control instruction. After calculating the verification value, the first control instruction is written into the control instruction block of the first shared memory pool according to the circular buffer mechanism.

[0031] The hardware capability slicing module periodically scans the control instruction blocks of the first shared memory pool, verifies the integrity of the instructions, and sends valid instructions to the conflict arbitrator. The conflict arbitrator determines whether there is resource contention in the instructions. If there is a conflict, it will adjudicate according to preset rules. If there is no conflict, it will allow the instructions to proceed directly.

[0032] Furthermore, the conflict arbitrator determines whether there is resource contention in the instructions, specifically in the following way:

[0033] Define four types of conflicts: parameter conflict, mutual exclusive resource conflict, bandwidth conflict, and operation mutual exclusive conflict.

[0034] The conflict arbitrator periodically scans the system's operating status, statistically analyzes the resolution, frame rate, and data format of multiple applications, and determines whether the hardware can support them simultaneously; it maintains a mutual exclusion resource occupancy table, recording resource ownership and locking status, and triggers a conflict when a new request conflicts with the current occupancy status; it calculates the total load of the memory bus and hardware encoder in real time, and triggers a conflict when the required total bandwidth exceeds a safety threshold; it monitors the application operation sequence, and triggers an operation conflict if it identifies a business combination that cannot be executed concurrently.

[0035] After a conflict is triggered, the conflict arbitrator makes a unified judgment on the conflict when multiple applications use the service concurrently, based on preset rules, and executes the following steps in sequence: Weights are assigned according to the type of application operation, with the weight decreasing from recording to taking pictures to previewing, prioritizing the preservation of the original parameters of the operation with the highest weight; weights are assigned according to the system type to which the application belongs, with the host application having a higher weight than the container application, prioritizing the parameter requirements of the host application; weights decrease from system-level applications to third-party tool applications to entertainment applications, prioritizing the preservation of the original parameters of the application with the highest weight, while the application with the lowest weight adjusts its own parameters for adaptation; when the total bandwidth exceeds a threshold, the frame rate of the corresponding applications is reduced in ascending order of weight; for non-shareable hardware resources, only a single application is allowed to use them exclusively, and the usage order is coordinated through a queue mechanism.

[0036] After the conflict resolution is completed, a resolution result is generated, which includes the associated request identifier, the affected slice identifier, the comparison of parameters before and after adjustment, and the reason for the conflict. The resolution result is then pushed to the application through the corresponding adaptation module.

[0037] Beneficial effects:

[0038] This invention initializes the permission verification module, divides and partitions the first shared memory pool after the host system starts, and starts a conflict arbitrator to build a concurrent request scheduling mechanism. The hardware capability slicing module detects hardware information and splits management logic slices. At the same time, the host adapter is started to complete the capability information adaptation and distribution. After the container starts, the permission proxy module is deployed to uniformly forward permission requests. The container adapter completes shared memory mapping, connects to the conflict arbitrator, and adapts to its own environment's hardware capability format. After the application starts, it sequentially completes camera permission verification, queries unified hardware capabilities, and builds an adapted business data stream configuration. Then, it initiates a resource access request carrying session configuration information, which is then processed by the host. After the container adapters are standardized, they are sent to the conflict arbitrator. The arbitrator calculates the priority and stores it in the scheduling queue. Then, the hardware capability slicing module allocates logical slices according to a three-level strategy. The slice parameters are summarized to determine the hardware baseline configuration and the camera is started. The shared memory block binding relationship and the dual-environment zero-copy data path are established. After the slice is distributed, the application officially enters the image acquisition stage. During the acquisition process, the system completes image data preprocessing and distribution. Hardware control commands are uniformly arbitrated by the conflict arbitrator and executed to realize the collaborative use of multiple applications. Finally, when the application stops using it, the system uniformly reclaims the logical slices, releases the memory mapping, and shuts down the camera to release resources if there is no business usage. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a cross-environment camera sharing method based on hardware access slices provided by the present invention. Detailed Implementation

[0040] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0041] This invention provides a cross-environment camera sharing method based on hardware access slicing. The core idea is as follows: After the host system starts, it initializes the permission verification module, divides the first shared memory pool into partitions, starts a conflict arbitrator to build a concurrent request scheduling mechanism, and the hardware capability slicing module detects hardware information and splits management logic slices. Simultaneously, the host adapter is started to complete the capability information adaptation and distribution. After the container starts, a permission proxy module is deployed to uniformly forward permission requests. The container adapter completes shared memory mapping, connects to the conflict arbitrator, and adapts to its own environment's hardware capability format. After the application starts, it sequentially completes camera permission verification, queries unified hardware capabilities, builds adapted business data stream configurations, and then initiates a sharing mechanism. Resource access requests for configuration information are sent to the conflict arbitrator after being standardized by the host and container adapter. The arbitrator calculates the priority and stores it in the scheduling queue. Then, the hardware capability slicing module allocates logical slices according to a three-level strategy. The slice parameters are summarized to determine the hardware baseline configuration and the camera is started. The shared memory block binding relationship and the dual-environment zero-copy data path are established. After the slice is distributed, the application officially enters the image acquisition stage. During the acquisition process, the system completes image data preprocessing and distribution. Hardware control commands are uniformly arbitrated by the conflict arbitrator and executed to achieve multi-application collaborative use. Finally, when the application stops using it, the system uniformly reclaims the logical slices, releases the memory mapping, and shuts down the camera to release resources if there is no business usage.

[0042] This invention provides a cross-environment camera sharing method based on hardware access slices, the processing flow of which is as follows: Figure 1 As shown, the specific steps include:

[0043] Step 1: When the host system starts, the permission verification module is started and initialized to receive permission requests from the host and container applications. The camera operation permission is divided into three levels of control. The permission mapping rules between the host and the container are established through the system configuration file to realize the automatic conversion and unified control of permissions in the dual-running environment.

[0044] A contiguous physical memory region is allocated as the first shared memory pool to provide a zero-copy data interaction channel for applications in multiple operating environments and camera hardware. The first shared memory pool includes a control instruction block, a raw data block, and a metadata block. The control instruction block uses a ring buffer structure to transmit hardware control instructions such as flash, focus, and zoom. The raw data block is used to store the raw image frame data output by the camera. The metadata block is used to store frame auxiliary information such as exposure, ISO, and focus distance corresponding to each frame image.

[0045] The conflict arbitrator is activated as a unified decision-making center for resource conflicts. Based on preset rules and logical slice status, the conflict arbitrator quickly adjudicates resource access conflicts. It uses a blocking queue to cache and schedule concurrent requests from multiple runtime environments. The queue capacity supports simultaneous requests from multiple applications, and a busy prompt is returned when the queue is full.

[0046] The host initiates a hardware capability query to the capability slicing module. The capability slicing module collects hardware information through the driver layer and encapsulates it into a unified device capability structure across environments. The capability slicing module is deployed between the hardware abstraction layer and the system camera framework layer, serving as a logical intermediary layer between physical cameras and applications in multiple operating environments. It accesses the hardware through a standard driver interface and splits the physical camera into logical slices based on three dimensions: resolution, frame rate, and data format, forming a logical slice library. At the same time, a first hash table is used to manage the occupancy status of the logical slices. The occupancy status includes whether it is idle, the bound application ID, the data stream type, and the processing flag. The processing flag includes whether downsampling is required and whether frame rate adjustment is required.

[0047] The host adapter is activated, converting the device capability structure into a standard information structure that can be recognized by the host camera frame, and exposing it to the application through the host standard capability query interface.

[0048] The three-tiered control system includes basic preview permissions, extended photo permissions, and advanced video recording permissions. Basic preview permissions only allow previewing the camera image and reading real-time frame data, and do not support photo taking, video recording, or control operations such as flash and zoom. Extended photo permissions include all the capabilities of basic preview permissions, and additionally support high-resolution single-frame capture, image encoding, and local storage, but do not support video recording. Advanced video recording permissions include all the capabilities of photo permissions, and additionally support advanced operations such as video recording, audio-video synchronization, flash control, and zoom adjustment. Audio recording permissions must be obtained simultaneously when using these permissions.

[0049] Step 2: When the container starts, the permission proxy module is started and initialized to take over the native permission management within the container. It forwards all camera permission requests from applications within the container to the host permission verification module for centralized judgment. The container adapter is started to complete the mapping of the first shared memory pool and establish a communication connection with the host's conflict arbitrator. The container adapter converts the device capability structure into a standard information structure that can be recognized by the container's camera frame, builds the correspondence between resolution and frame rate, and exposes it to applications through the container's standard capability query interface.

[0050] Step 3: After the application starts, both the host application and the container application initiate camera permission requests. The host application sends a permission request to the permission verification module, while the container application's permission request is forwarded to the host's permission verification module via the permission proxy module for verification. The permission verification module performs identity and signature validity verification and permission declaration integrity verification on the application in sequence. After the verification is passed and confirmed by the user, the permission verification module records the application's permission status and returns the authorization result to the application, completing the permission application process.

[0051] Applications query camera hardware capabilities through standard interfaces. The host adapter and container adapter return the converted device capability structure that is recognizable by their respective runtime environments to the host application and container application, respectively, so that the application can obtain completely consistent resolution, frame rate, data format, flash and zoom capabilities. Based on this, the application builds a session configuration that conforms to the hardware support range.

[0052] The application constructs a session configuration that conforms to the hardware support range by: distinguishing business usage scenarios and building corresponding data streams for preview, shooting, and video recording; setting the running parameters of each data stream according to the resolution, frame rate, and image data format supported by the hardware; binding business output components such as screen rendering, image storage, and video encoding to complete the association configuration between the data stream and the output carrier; and merging multiple data streams with different purposes to form a complete and callable camera usage session configuration.

[0053] Step 4: The application initiates a camera resource access request containing session configuration. The host adapter and container adapter convert the heterogeneous request into a cross-environment standard resource request and send it to the conflict arbitrator. The conflict arbitrator calculates the request priority based on the operation type, application type, and permission level, and adds the request to the scheduling queue for processing. The hardware capability slicing module completes the logical slice allocation based on the application request parameters through a three-level strategy of precise matching, resolution downgrade matching, and frame rate limit matching. It marks the image processing method required for the slice and updates the occupancy status, and then sends the allocation result to the conflict arbitrator.

[0054] After all logical slices are allocated, the hardware capability slicing module determines the hardware baseline configuration based on the parameters of all allocated logical slices, starts the physical camera and establishes a data output path; it completes the binding of logical slices with the original data blocks and metadata blocks in the first shared memory pool, and the host application and container application establish zero-copy data paths through memory mapping respectively; the conflict arbitrator issues logical slices to the application through the corresponding adapter, and the application starts its own business data flow according to the logical slice parameters and enters the image acquisition and operation stage.

[0055] Furthermore, the way an application initiates a camera resource access request containing session configuration is as follows: the application establishes the data stream configuration required by the business according to its own camera application framework, including preview data stream configuration and shooting data stream configuration. The session configuration information used by the camera is formed by all preset parameters such as screen size, running frame rate, data format, output binding components, and business usage type. The application then initiates a camera resource access request based on the session configuration information.

[0056] The preview data stream is configured with the appropriate display size, frame rate, and common image data format for the scene. It determines the output carrier of the preview screen and binds it to the rendering area. It adapts the rendering area specifications according to the data stream parameters, establishes a data receiving association channel between the data stream and the rendering component, starts data listening to wait for frame data to be transmitted, completes the image data format adaptation and conversion, and continuously pushes the screen to achieve real-time preview. The shooting data stream is configured with the high-definition imaging size and image encoding format, and the image storage component for local image saving is configured.

[0057] Furthermore, the host adapter and container adapter uniformly convert heterogeneous requests into cross-environment standard resource requests in the following way:

[0058] The host adapter and container adapter intercept camera resource access requests initiated by applications within their respective environments, parse the built-in session configuration and business template information, distinguish the actual usage scenarios, and extract core operating parameters such as business flow type, screen resolution, frame rate, and data format. At the same time, they read the application identification information and determine the application's category and permission level. They complete standardized encapsulation according to the globally unified message format to generate a general resource request message with consistent structure. The message is uniformly labeled with the request source, application ID, application type, independent request number, as well as the screen specifications, frame rate, data format, output binding identifier, and permission level corresponding to each business flow. The operating environment is distinguished only by the source field.

[0059] Furthermore, the conflict arbitrator calculates request priority based on operation type, application type, and permission level as follows: The conflict arbitrator obtains standard resource requests and calculates request priority according to preset rules. The initial priority value in the preset rules is the sum of the product of the operation type base score and the application type weight, and the permission level bonus. The operation type base score is set according to the importance of the business, with recording operations having the highest score, followed by taking photos, and previewing the lowest. If the request contains multiple data streams, the type with the highest score is used for calculation. The application type weight is divided into weight levels according to the application's operating environment and attributes, with the host system application having the highest weight and the container third-party application having the lowest weight. The permission level bonus is an additional score added based on the permission level already obtained by the application, with higher permissions corresponding to higher bonuses.

[0060] The hardware capability slicing module allocates logical slices based on application request parameters using a three-level strategy: precise matching, resolution downgrade matching, and frame rate limit matching. Specifically:

[0061] Step 4.1: Traverse the idle logical slice library and match logical slices that have the same resolution as the requested slice, a frame rate greater than or equal to the requested value with the smallest difference, and completely consistent data format. If a match is successful, mark the logical slice as pending allocation and proceed to step 4.4. If a match fails, proceed to step 4.2 to perform resolution downgrading matching.

[0062] Step 4.2: Filter idle logical slices with resolution higher than the requested specification, frame rate that meets the requirements, and data format that are consistent. Prioritize selection based on the principle of minimum resolution difference. If a match is successful, mark the logical slice as needing resolution downsampling and proceed to step 4.4. If a match fails, proceed to step 4.3 to perform frame rate limit matching.

[0063] Step 4.3: Match idle slices with the same resolution, frame rate higher than the requested value, and data format. If the match is successful, mark the logical slice as needing frame rate limiting or frame skipping processing before proceeding to step 4.4. If the match fails, return a resource allocation failure message.

[0064] Step 4.4: The hardware capability slicing module assigns a unique logical slice to each data stream, updates the slice occupancy status, generates matching results, and feeds them back to the conflict arbitrator.

[0065] Furthermore, the resource allocation failure information includes the reason for the error and the available parameters supported by the current hardware. The conflict arbitrator forwards the error information to the application, indicating that the request cannot be fulfilled and suggesting adjustments to parameters such as resolution and frame rate.

[0066] The hardware capability slicing module determines the hardware baseline configuration based on the parameters of all allocated logical slices as follows: It collects the resolutions of all allocated logical slices, calculates the total number of pixels for each resolution, and selects the resolution with the largest number of pixels as the candidate baseline resolution. It then verifies whether the hardware supports the specification; if not, it selects the next largest resolution in turn until a hardware-compatible configuration is found, which becomes the final baseline resolution. Next, it collects the requested frame rates of all allocated slices and selects the largest frame rate as the candidate baseline frame rate. Finally, it verifies whether the hardware supports the frame rate at the baseline resolution; if not, it downgrades to the maximum frame rate supported by the hardware and simultaneously notifies all slices to adapt. Finally, it uniformly adopts the image data format with the highest native hardware support and strongest compatibility to meet the processing needs of previewing, taking photos, and recording videos across all scenarios.

[0067] Specifically, the logical slice is bound to the original data block and metadata block in the first shared memory pool. The host application and the container application establish zero-copy data paths through memory mapping, as follows:

[0068] Memory blocks of corresponding capacity are allocated according to the logical slice resolution. High-resolution slices are allocated memory blocks with a capacity greater than the threshold, while low-resolution slices are allocated memory blocks with a capacity less than the threshold. Region isolation is achieved through memory address offset. A dedicated metadata block is independently bound to each logical slice to store frame auxiliary information such as exposure, ISO, and focus, and is associated with the original image data through the logical slice identifier. The memory block address information corresponding to each logical slice is recorded for quick location of data storage at runtime.

[0069] The host adapter maps the memory blocks corresponding to the logical slices in the first shared memory pool to the memory space of the host application; it points the buffer of the application's screen rendering component to this memory space; the hardware output data is processed by the hardware capability slicing module and written to the corresponding memory block of the first shared memory pool; the application reads the data in real time through the mapped address.

[0070] The container adapter maps the first shared memory pool to the container runtime environment and determines the target memory block based on the logical slice address offset; it allocates an image receiving buffer for the container application and establishes an address mapping relationship between it and the target memory block; it injects the bound image receiving buffer into the application's image receiving component so that hardware output data can be directly delivered to the application; the hardware output data is processed by the hardware capability slicing module and written to the shared target memory block, which the container application can directly obtain through the mapping relationship.

[0071] The hardware capability slicing module writes frame auxiliary information such as exposure, ISO, focus distance, and flash status corresponding to each frame image into the metadata block bound to the slice. The host and container adapter associate the original image data with the metadata through timestamps, convert it into their respective runtime environment standard formats, and then synchronously call back to the application to ensure that the application obtains complete frame information.

[0072] Step 5: During the camera's image acquisition process, the hardware capability slicing module downsamples or adjusts the frame rate of the output raw image data according to the slicing configuration, writes it into the corresponding shared memory block in the first shared memory pool, and notifies the application to read it. The control commands of the dual-environment application are transmitted to the conflict arbitrator for unified arbitration via the first shared memory pool. After the arbitration is successful, the hardware capability slicing module executes hardware control to realize concurrent access and collaborative control of multiple applications.

[0073] Specifically, the hardware-capable slicing module downsamples or adjusts the frame rate of the output raw image data according to the slicing configuration, writes it to the corresponding shared memory block in the first shared memory pool, and notifies the application to read it. The specific method is as follows:

[0074] The hardware capability slicing module receives raw image frames from the driver output in a loop through an independent thread. It performs preprocessing on each frame, including data verification, timestamp generation, and metadata extraction. Data verification checks whether the frame data length matches the reference resolution and discards abnormal data to ensure image integrity. Timestamp generation generates a globally unique microsecond-level timestamp for each frame for data synchronization and association. Metadata extraction extracts and temporarily stores frame auxiliary information such as exposure, ISO, and focus distance from the driver information.

[0075] The hardware capability slicing module executes corresponding processing logic for each slice based on the processing flag of the logical slice, realizing the distribution of single-channel hardware data to multiple logical slices. When the slice parameters are completely consistent with the hardware baseline configuration, the original data is written to the shared memory block bound to the slice, the data is marked as valid, and the timestamp is updated. When the slice resolution is lower than the baseline configuration, the original data is scaled by the hardware image processor to convert the high-resolution image into the slice target resolution. After processing, the data is written to the corresponding shared memory block and marked as valid. When the slice frame rate is lower than the baseline frame rate, a frame counting and frame skipping strategy is used to control the output frequency. Valid frames are retained and redundant frames are discarded according to preset rules. Only the retained frames are written to the shared memory block and marked as valid to ensure that the output frame rate is consistent with the request.

[0076] The frame auxiliary information, such as exposure, ISO, focus, and flash, corresponding to the frame is written into the metadata block bound to the slice to ensure complete consistency with the image data timestamp. The host and container adapter monitor the status of the shared memory block in real time. When the data validity flag is set, a data ready callback is triggered to the application, and the application reads the image from the shared memory block and renders it.

[0077] Furthermore, the host application obtains new frames through data-ready callbacks, reads image data from the shared memory block to complete business processing, specifically including: after the application determines that a new frame has arrived, it locks the screen rendering area, converts the image data in the shared memory block into a renderable format and displays it directly, completing an efficient and low-power preview; the application initiates a photo request, reads high-definition image data from the shared memory block of the corresponding slice, generates an image file through hardware encoding, and saves it to local storage; when the application creates a video encoder, it reads image data from the shared memory block for hardware encoding, and encapsulates the encoded video stream into a standard format file for saving.

[0078] Furthermore, the container application obtains frame data through the image receiving component callback and uses shared memory mapping to achieve zero-copy reading. Specifically, the application obtains image data and converts it into texture data, and renders and displays it through the graphics interface to ensure smooth and stable preview. The video encoder directly reads image data from the shared memory block for encoding, and encapsulates the video and audio streams into a standard video file to achieve stable recording.

[0079] In this process, control commands for dual-environment applications are transmitted to the conflict arbitrator via the first shared memory pool for unified arbitration. Specifically, the method is as follows:

[0080] Establish a universal first control command across environments, which includes command type, parameters, source, application ID, associated slice, timestamp and verification information. Supported commands include flash control, autofocus, zoom adjustment and exposure compensation. When the host application and container application initiate control operations, the host adapter and container adapter respectively intercept and encapsulate them into the first control command. After calculating the verification value, they are written into the control command block of the first shared memory pool according to the circular buffer mechanism.

[0081] The hardware capability slicing module periodically scans the control instruction blocks in the first shared memory pool, verifies the integrity of the instructions, and sends valid instructions to the conflict arbitrator. The conflict arbitrator determines whether there is a resource contention between the instructions. If there is a conflict, it will decide according to the preset rules. If there is no conflict, it will be allowed directly.

[0082] After arbitration is successful, the hardware capability slicing module converts the first control command into a control signal that the hardware driver can recognize, and completes operations such as flash, focus, zoom, and exposure. After the command is executed, the result information is generated and sent back to the application through the adapter.

[0083] Furthermore, the conflict arbitrator determines whether there is a resource contention in the instructions as follows:

[0084] Define four types of conflicts: parameter conflict, mutual exclusive resource conflict, bandwidth conflict, and operation mutual exclusive conflict.

[0085] The conflict arbitrator scans the system's operating status at fixed intervals, statistically analyzes the resolution, frame rate, and format requirements of multiple applications, and determines whether they exceed the hardware's simultaneous support capabilities; it maintains a mutual exclusion resource occupancy table, recording resource ownership and locking status, and triggers a conflict when a new request conflicts with the current occupancy status; it calculates the total load of the memory bus and hardware encoder in real time, and triggers a conflict when the total bandwidth requirement exceeds a safety threshold; it monitors the application operation sequence, and triggers an operation conflict if it identifies a business combination that cannot be executed concurrently.

[0086] The conflict arbitrator, based on preset rules, uniformly judges conflicts when multiple applications are used concurrently, and executes them in sequence: Prioritize applications according to their operation type, with the weight decreasing from recording to taking photos to previewing, prioritizing the original parameters of the operation with the highest weight; prioritize applications according to their system type, with the host application having a higher weight than the container application, prioritizing the parameter requirements of the host application; prioritize system-level applications to third-party tool applications to general entertainment applications, with the original parameters of the application with the highest weight being retained, and the application with the lowest weight adjusting its own parameters for adaptation; when the total bandwidth exceeds the limit, reduce the frame rate of the corresponding applications in ascending order of weight; for non-shareable hardware resources such as flash and laser autofocus, only a single application is allowed to use them exclusively, and the usage order is coordinated through a queue mechanism.

[0087] After completing the conflict resolution, a structured resolution result is generated, which includes the associated request identifier, the affected slice identifier, the comparison of parameters before and after adjustment, and the reason for the conflict. The resolution result is then pushed to the host and container applications respectively through the corresponding adaptation modules.

[0088] After receiving the ruling, if the resolution is adjusted, the image rendering and receiving components are rebuilt to match the new resolution specifications; if the frame rate is adjusted, the video encoding parameters are updated synchronously to ensure that the encoding is compatible with the new frame rate.

[0089] The hardware capability slicing module updates the processing flags of the corresponding slices based on the new parameters in the adjudication results and recalculates the hardware baseline configuration; at the same time, it synchronously updates the hardware driver parameters to ensure that the hardware output matches the new parameters.

[0090] Among them, parameter conflicts include resolution conflicts, frame rate conflicts, and data format conflicts, mainly caused by the inability of hardware capabilities to simultaneously meet the parameter requirements of multiple applications; mutual exclusion resource conflicts refer to hardware resources that can only be occupied by a single user at the same time, including flash, laser focus, HDR mode, etc., which cause conflicts when the same resource is requested by multiple applications at the same time; bandwidth conflicts include memory bus bandwidth conflicts and hardware encoding bandwidth conflicts, which are triggered when the total bandwidth requirement of multiple data streams exceeds the hardware's carrying capacity limit; operation mutual exclusion conflicts refer to combinations of business operations that cannot be executed simultaneously, such as fast continuous shooting and video recording being mutually exclusive, manual zoom and autofocus being mutually exclusive, etc.

[0091] Step 6: When the application exits or the camera is actively turned off, the system releases the allocated logical slices, unmaps the shared memory blocks, and updates the slice status to free. When no slices are occupied, the physical camera is turned off and hardware energy efficiency degradation is triggered, completing the entire process of resource reclamation.

[0092] Example:

[0093] This embodiment employs a cross-environment camera sharing method based on hardware access slicing provided by the present invention. Using OpenHarmony as the host system and Android as the container system, it achieves concurrent sharing of physical camera resources and efficient conflict arbitration through hardware capability logical slicing and cross-system protocol conversion. The specific process includes:

[0094] S1. Unified cross-system camera permission verification and device capability discovery: Before an application can access a camera, it must pass a unified permission verification and obtain consistent device capability information.

[0095] S1.1 Deployment and rule definition of the Auth-Module for permission verification.

[0096] S1.1.1 Module Integration Architecture: Auth-Module is integrated as an independent service into OpenHarmony's permission management service PermissionManagerService. It connects to two parts through the Binder interface: one is the permission request channel of the OpenHarmony application, which handles the Context.requestPermissionsFromUser() interface; the other is the permission proxy module Android-Permission-Proxy in the Android container. This proxy module replaces the native PermissionManager implementation of the Android system to ensure that all permission requests of the Android application are routed to Auth-Module.

[0097] S1.1.2, Three-level permission levels and operation mapping:

[0098] CAMERA_BASIC permission level 1: Only preview operations are allowed. The specific permission scope includes: real-time screen rendering such as SurfaceView display and frame data reading. Calling interfaces such as takePicture() and startRecording() is prohibited.

[0099] CAMERA_CAPTURE Permission Level 2: Includes all CAMERA_BASIC permissions, additionally allows photo taking, supports single-frame high-resolution capture such as 4K, JPEG encoding and storage, and disables video recording related interfaces;

[0100] CAMERA_RECORD Permission Level 3: Includes all CAMERA_CAPTURE permissions, additionally allowing recording to support H.264 or H.265 encoding, flash control such as turning on or off, zoom adjustment and other advanced operations, and requires RECORD_AUDIO permission to ensure audio and video synchronization.

[0101] S1.1.3 Dual-system permission mapping table design: The mapping relationship is defined through the configuration file / etc / camera_perm_mapping.conf. The core mapping rules are shown in the table below.

[0102] Table 1 Mapping Rules Table android.permission.CAMERA CAMERA_BASIC Preview CAMERA + WRITE_EXTERNAL_STORAGE CAMERA_CAPTURE Preview + Take a photo (save locally) CAMERA + RECORD_AUDIO CAMERA_RECORD Preview + Take a photo + Record a video (including audio)

[0103] S1.2 Application permission application and dynamic verification process.

[0104] S1.2.1, The entire process of applying for OpenHarmony application permissions:

[0105] Applications declare target permissions in config.json, such as ohos.permission.CAMERA_RECORD;

[0106] The request is initiated by calling Context.requestPermissionsFromUser(new String[]{"ohos.permission.CAMERA_RECORD"}, requestCode);

[0107] After the Auth-Module intercepts the request, it performs three verification steps: verifying whether the application signature is valid by comparing it with the system's trusted certificate list; verifying whether the permission list declares the target permission to avoid unauthorized requests; and popping up a unified style permission pop-up window, such as the one consistent with the OpenHarmony system pop-up window style, displaying "Allow [application name] to use the camera to record video?".

[0108] After user authorization, the Auth-Module updates the application's permission status, stores it in / data / system / permission.db, and returns PERMISSION_GRANTED through the onRequestPermissionsResult() callback.

[0109] S1.2.2, Android application permission request proxy process:

[0110] An Android application calls Activity.requestPermissions(new String[]{Manifest.permission.CAMERA}, requestCode);

[0111] Android-Permission-Proxy intercepts requests, converts Android permissions into OpenHarmony permission format, such as android.permission.CAMERA → ohos.permission.CAMERA_BASIC, and sends them to Auth-Module via Binder;

[0112] The Auth-Module performs the same verification logic as the OpenHarmony application, and the permission pop-up content is adapted to the Android application scenario, such as "Do you allow [Android application name] to use the camera for preview?";

[0113] The verification result is converted into Android's onRequestPermissionsResult() callback through the proxy module, such as PackageManager.PERMISSION_GRANTED, to ensure seamless adaptation for Android applications.

[0114] S1.2.3 Dynamic Permission Verification Mechanism: Unlike the existing one-time authorization mode that is permanently valid, the Auth-Module triggers real-time verification before each core operation of the application:

[0115] Preview operation: Before calling Surface.readFrame(), verify whether you have CAMERA_BASIC or higher permissions;

[0116] Photo taking operation: Before calling takePicture(), check whether you have CAMERA_CAPTURE or higher permissions and whether the storage permissions are valid;

[0117] Recording operation: Before calling startRecording(), check whether you have CAMERA_RECORD and RECORD_AUDIO permissions. If not, throw a SecurityException with an error message such as: Recording permission is required to complete recording.

[0118] S1.3 Unified discovery and standardized exposure of equipment capabilities.

[0119] S1.3.1 Details of raw hardware capability acquisition: The OH-Adapter calls the HCS-Slicer's getHardwareCapabilities(cameraId) interface, and the HCS-Slicer obtains the raw capabilities through the following steps:

[0120] Enable the camera driver device node, such as / dev / video0;

[0121] Call ioctl(fd, VIDIOC_ENUM_FRAMESIZES, &frameSize) to enumerate all supported resolutions and store them as a list of width*height;

[0122] Call ioctl(fd, VIDIOC_ENUM_FRAMEINTERVALS, &frameInterval) to enumerate the frame rate range corresponding to each resolution, such as 1080P supporting 15 to 60fps;

[0123] Call ioctl(fd, VIDIOC_QUERYCTRL, &ctrl) to query control capabilities, such as V4L2_CID_FLASH_LED_MODE to determine if the flash is supported, and V4L2_CID_ZOOM_ABSOLUTE to determine if zoom is supported;

[0124] Shut down the driver node and organize the collected raw capabilities into structured data.

[0125] S1.3.2, Cross-system unified capability format DeviceCapability definition: Implemented using C structures to ensure fixed memory layout and cross-language parsing. The structure definition is as follows:

[0126] typedef struct {

[0127] char cameraId

[16] ; / / Unique identifier for the camera, such as 0 for rear camera and 1 for front camera.

[0128] char type

[32] ; / / Camera type, enumeration values: "BACK" / "FRONT" / "ULTRA_WIDE" / "TELEPHOTO"

[0129] struct Resolution {

[0130] int width; / / Resolution width, such as 1920

[0131] int height; / / Resolution height, such as 1080

[0132] } resolutions

[16] ; / / List of supported resolutions, up to 16

[0133] int resolutionCount; / / The actual number of resolutions, such as 5

[0134] struct FpsRange {

[0135] int minFps; / / Minimum frame rate, e.g., 15

[0136] int maxFps; / / Maximum frame rate, such as 60

[0137] } fpsRange; / / Global frame rate range, the intersection of all supported resolutions

[0138] char formats[8]

[32] ; / / Supported data formats, such as YUV420_888, RAW10

[0139] int formatCount; / / The actual number of formats, e.g., 2

[0140] bool flashSupported; / / Whether to support flash, true or false

[0141] struct ZoomRange {

[0142] float minZoom; / / Minimum zoom level, such as 1.0x

[0143] float maxZoom; / / Maximum zoom level, such as 10.0x

[0144] float step; / / Zoom step size, such as 0.1x

[0145] zoomRange; / / Zoom range

[0146] DeviceCapability;

[0147] S1.3.3, Dual System Capability Adaptation and Exposure Details:

[0148] OpenHarmony side adaptation: The OH-Adapter converts DeviceCapability into a CameraDeviceInfo object from the Camera Kit. The core field mappings are as follows:

[0149] cameraId→getCameraId();

[0150] resolutions→getSupportedResolutions() (converts to a list of Size);

[0151] flashSupported→isFlashSupported();

[0152] The CameraManager.getCameraList() method returns the list of resolutions to the application, which can then call cameraDeviceInfo.getSupportedResolutions() to obtain a list of resolutions that match the hardware.

[0153] Android-side adaptation: A-Adapter converts DeviceCapability into an Android CameraCharacteristics object, and populates the core key-value pairs using CameraCharacteristics.Builder.

[0154] LENS_FACING: Based on the type mapping, BACK is mapped to LENS_FACING_BACK, and FRONT is mapped to LENS_FACING_FRONT;

[0155] SCALER_STREAM_CONFIGURATION_MAP: Constructs a StreamConfigurationMap, which combines resolutions and fpsRange into a resolution-to-frame-rate mapping table, such as 1080P corresponding to 15 to 60fps;

[0156] FLASH_INFO_AVAILABLE: Directly mapped flashSupported;

[0157] The CameraManager.getCameraCharacteristics(cameraId) is returned to the Android application to ensure that the capability information obtained by the application is completely consistent with that of the OpenHarmony application, such as both showing support for 4K / 30fps.

[0158] S2. Cross-system camera resource management architecture initialization: After the permission verification is passed, the system initializes the core components and builds the infrastructure for cross-system resource sharing.

[0159] S2.1, Definition and Deployment of Core Components.

[0160] S2.1.1, Physical Camera Capability Slicer HCS-Slicer:

[0161] Definition and Positioning: Deployed between the HAL layer / vendor / lib64 / hw / camera.provider.ohos.so and the framework layer / system / lib64 / libcamerakit.so in OpenHarmony, it serves as a "logical middleware" between the physical camera and the dual-system application. It directly calls the standard V4L2 interfaces of the camera driver, such as open, ioctl, or read, and can achieve logical decomposition of hardware capabilities without modifying the driver source code.

[0162] Core functions and implementation methods:

[0163] Slice library generation algorithm: Based on the original hardware capabilities, all possible slice combinations are enumerated according to three dimensions: resolution, frame rate, and data format, to generate a slice library. The enumeration rules are as follows:

[0164] Resolution: Select a commonly used value from the list of resolutions supported by the hardware, such as excluding the uncommon 2560*1440 and keeping 720P, 1080P, and 4K;

[0165] Frame rate: Each resolution has 3 fixed frame rate levels: low is 15fps, medium is 30fps, and high is 60fps. If the hardware does not support it, it will be downgraded. For example, if 4K only supports 30fps, then only 30fps will be retained.

[0166] Data format: YUV420_888 is selected by default, which is compatible with more than 99% of application requirements;

[0167] Slice naming rules: slice_resolution identifier_frame rate_format, such as slice_1080p_30_yuv420, slice_4k_30_yuv420;

[0168] Slice state management: The slice state is managed through a hash table sliceMap, where the key is the slice ID and the value is a SliceState structure. SliceState contains: whether it is idle (isFree), the bound application ID (boundAppId), the associated stream type (streamType), and whether downsampling or frame rate adjustment (processFlag) is required.

[0169] Hardware Capabilities Query Interface: Provides the getHardwareCapabilities(cameraId) and matchSlices(CrossSystemRequest) interfaces, which are used for capability acquisition and slice matching, respectively.

[0170] S2.1.2, Cross-system shared memory pool CS-MemPool.

[0171] Definition and Location: A contiguous physical memory region initialized by OpenHarmony's memory management service / system / bin / memorymanager at system startup, with a fixed size of 100MB, which can be adjusted via / sys / module / cs_mem_pool / parameters / size. It serves as a zero-copy channel for data interaction between dual-system applications and camera data, avoiding the repeated copying losses of traditional Socket transmission. Existing solutions require at least 3 copies, while this embodiment only requires 1 mapping.

[0172] The structure and parameter design of the three-level memory block are shown in the table below.

[0173] Table 2. Parameters of Level 3 Memory Block Structure Control instruction block 4 16KB The maximum length of a single instruction is ≤1KB, and four circular buffers can meet the requirements of concurrent instructions within 100ms. Transmit flash, focus, and other control commands raw data blocks 6 12MB 4K YUV420 data volume = 3840 × 2160 × 1.5 = 11,943,936 bytes ≈ 12MB, 6 blocks can cache 200ms of data. Store the raw YUV frame data output by the camera. Metadata blocks 8 4KB Single-frame metadata (exposure time, ISO, etc.) ≤ 1KB, 8 blocks can meet the 30fps frame rate requirement. Store metadata corresponding to the original frame, linked by timestamp.

[0174] Memory synchronization and mutual exclusion mechanisms:

[0175] Memory mapping methods: OpenHarmony applications use shared memory Ashmem for mapping, while Android applications use POSIX shared memory / dev / shm / oh_cam_shm for mapping. Both access the same physical memory region; [corresponding to the shared memory block above---]

[0176] Mutex lock design: 8 bytes of pthread_mutex_t mutex lock are reserved at the beginning of each memory block to ensure that there is no data competition when dual-system applications read and write concurrently. For example, when the OpenHarmony application writes a command, the Android application must wait for the lock to be released.

[0177] Circular buffer management: The control instruction block adopts a circular buffer design, which manages data reading and writing through read pointers and write pointers. When the write pointer catches up with the read pointer, it triggers an overwrite, discarding the oldest instruction and avoiding memory overflow.

[0178] S2.1.3, Conflict Arbiter CA-Arbiter:

[0179] Definition and Positioning: The independent service process / system / bin / camera_arbiter deployed on the OpenHarmony side is registered as a system service at startup and can be obtained through getService("camera_arbiter"). It serves as the decision center for resource conflicts between dual-system cameras, and achieves millisecond-level arbitration based on preset rigid rules, with an average processing time of no more than 5ms.

[0180] Core modules and implementation:

[0181] Request receiving queue: The blocking queue BlockingQueue is used to cache the CrossSystemRequest of the dual-system application. The queue capacity is 16, which can meet the simultaneous requests of 8 applications. When the queue is full, the following message is returned: Unable to process temporarily, please try again later.

[0182] Rules engine: Loads the preset arbitration rule table / etc / camera_arbiter_rules.json. The rules are stored in JSON format, support dynamic updates, and do not require a service restart.

[0183] State synchronization module: Communicates with HCS-Slicer via Socket, port number 10086, using AF_INET field and SOCK_STREAM type to obtain slice occupancy status; communicates with OH-Adapter / A-Adapter via Binder to push arbitration results.

[0184] S2.1.4 Dual-system protocol adaptation modules OH-Adapter and A-Adapter.

[0185] Definition and Positioning: OH-Adapter acts as the proxy implementation of Camera Kit, replacing the CameraDevice instance in libcamerakit.so. A-Adapter acts as the proxy of Camera2 Service within the Android container, replacing CameraDeviceClient in / system / lib64 / libcamera_client.so. Both of them jointly undertake the role of protocol translation, ensuring that heterogeneous requests from the two systems can be handled uniformly.

[0186] Core functions:

[0187] Request conversion: Convert OpenHarmony's SessionConfiguration and Android's CaptureRequest to a unified CrossSystemRequest. During the conversion, parameter differences need to be handled, such as OH's STREAM_TYPE_VIDEO corresponding to Android's TEMPLATE_RECORD.

[0188] Data conversion: Converts the YUV data output by HCS-Slicer into a format that the application can recognize, such as OH's SurfaceBuffer or Android's Image;

[0189] Callback synchronization: Convert arbitration results and data readiness notifications into native callbacks of the dual system frameworks, such as OH's onFrameAvailable and Android's onImageAvailable.

[0190] S2.2 Component Collaborative Initialization Process.

[0191] S2.2.1, HCS-Slicer Initialization Full Link:

[0192] When the system starts, the Camera HAL loads camera.provider.ohos.so, triggering HCS-Slicer initialization;

[0193] Call the initSlicer() function:

[0194] Scan all camera device nodes / dev / video* to obtain a list of camera IDs;

[0195] For each camera, call getHardwareCapabilities(cameraId) to collect its raw capabilities;

[0196] Generate a slice library according to the slice generation algorithm, initialize the sliceMap, and mark all slices as free;

[0197] Register slice status callbacks with CA-Arbiter to notify CA-Arbiter when a slice is occupied or released.

[0198] S2.2.2, CS-MemPool Initialization Process:

[0199] When the memory management service starts, it calls initCsMemPool():

[0200] Request 100MB of contiguous physical memory and map it to the kernel space using mmap, with an address range of 0xFFFF00000000-0xFFFF00064000;

[0201] The memory is divided into three levels: a 4*16KB control instruction block, a 6*12MB raw data block, and an 8*4KB metadata block.

[0202] Initialize the mutex lock for each memory block;

[0203] Create the file / dev / shm / oh_cam_shm, map physical memory to this file, and set the permissions to 0666 to allow the Android container to access it.

[0204] S2.2.3, CA-Arbiter Initialization Process:

[0205] When the service starts, initArbiter() is called:

[0206] Load the arbitration rule table camera_arbiter_rules.json, parse the rules and store them in memory, such as operation type priority and system type weight;

[0207] Create a Socket server on port 10086 and wait for HCS-Slicer to connect;

[0208] Create a Binder server and register it as a camera_arbiter service;

[0209] Initialize the request receiving queue and thread pool, with 4 worker threads processing requests.

[0210] S2.2.4 Adapter Initialization Process:

[0211] OH-Adapter initialization: When Camera Kit is loaded, CameraDevice is instantiated and replaced with the OH-Adapter implementation by calling initOHAdapter():

[0212] Bind the CA-Arbiter Binder service;

[0213] Register the CrossSystemRequest sending interface;

[0214] The initialization data conversion module YUV is replaced with SurfaceBuffer.

[0215] A-Adapter initialization: When the Android container starts, the CameraDeviceClient is instantiated and replaced with the A-Adapter implementation, calling initAAdapter():

[0216] Map the shared memory of / dev / shm / oh_cam_shm;

[0217] The Binder service is bound to CA-Arbiter via a Binder proxy within the container;

[0218] Initialize the request conversion module, converting CaptureRequest to CrossSystemRequest.

[0219] S3. Application camera request parsing and resource slice matching: After the application obtains the device's capabilities and passes the permission verification, it initiates a specific access request. The system parses the request and matches the appropriate resource slice.

[0220] S3.1, OpenHarmony application request parsing.

[0221] S3.1.1 Application Request Construction Details: The OpenHarmony application constructs a SessionConfiguration using Camera Kit, with the core parameters configured as follows:

[0222] Preview stream configuration: Created by calling StreamConfiguration.Builder(STREAM_TYPE_PREVIEW), setting the resolution Size (1920, 1080), frame rate 30fps, data format FORMAT_YUV420_888, and associating it with the Surface of the SurfaceView, which is obtained through surfaceView.getSurface() for real-time rendering;

[0223] Image stream configuration: Created by calling StreamConfiguration.Builder(STREAM_TYPE_PHOTO), setting the resolution Size(3840, 2160), data format FORMAT_JPEG, and associating it with the ImageReader's Surface (ImageReader.newInstance(3840, 2160, ImageFormat.JPEG, 2)) for storing captured images;

[0224] Session Configuration: Add the preview stream and photo stream to SessionConfiguration, and set the session type to SESSION_TYPE_VIDEO_PHOTO to support video recording and photo taking.

[0225] S3.1.2, OH-Adapter Interception and Conversion Process:

[0226] The OH-Adapter intercepts requests through the CameraDevice.createCaptureSession() interface and calls parseSessionConfig(sessionConfig) to parse the parameters.

[0227] Extracting the stream type: Iterate through sessionConfig.getStreams() and use stream.getType() to distinguish between the preview stream STREAM_TYPE_PREVIEW and the photo stream STREAM_TYPE_PHOTO;

[0228] Extracting resolution: Use stream.getSize().getWidth() and stream.getSize().getHeight() to get the resolution, and convert it to width*height, such as 1920*1080;

[0229] Extract the frame rate: Get it using stream.getFrameRateRange().getMax(), and take the maximum frame rate to ensure smoothness;

[0230] Extract application information: Get the appId, such as com.oh.camera, through Context.getBundleName(), and determine the application type, such as system application or third-party application, through PackageManager.getApplicationInfo();

[0231] Generate a JSON-formatted CrossSystemRequest, serialize it using the cJSON library, and include the following core fields, along with value range constraints:

[0232] {

[0233] "source": "OH", / / Fixed value: "OH" represents OpenHarmony

[0234] "appId": "com.oh.camera", / / Application package name, no longer than 256 characters

[0235] "appType": "THIRD_PARTY", / / Enumeration values: SYSTEM for system applications, THIRD_PARTY for third-party applications

[0236] "requestId": "req_12345678", / / Unique identifier for the request, format: req_ and timestamp, timestamp in microseconds.

[0237] "streams": [

[0238] {

[0239] "type": "preview", / / Enumeration values: preview, photo, video

[0240] "resolution": "1920x1080", / / Resolution format: width*height, must be in the hardware support list

[0241] "fps": 30, / / Frame rate, must not exceed the hardware's maximum frame rate

[0242] "format": "YUV420_888", / / Enumeration values: YUV420_888, JPEG, H.264

[0243] "surfaceId": "surface_001", / / The application's Surface identifier format is surface_ followed by the serial number.

[0244] "permLevel": "CAMERA_RECORD" / / Apply the current permission level for arbitration priority calculation

[0245] },

[0246] {

[0247] "type": "photo",

[0248] resolution: 3840x2160

[0249] "format": "JPEG",

[0250] "surfaceId": "surface_002",

[0251] "permLevel": "CAMERA_RECORD

[0252] } ]

[0254] }

[0255] Call sendRequestToArbiter(requestJson) to send the serialized JSON data to the CA-Arbiter request receiving queue via Binder.

[0256] S3.2, Android application request parsing.

[0257] S3.2.1 Application Request Construction Details: Android applications construct CaptureRequest and output surfaces through Camera2. The core configuration is as follows:

[0258] Preview stream configuration: Create an ImageReader (ImageReader.newInstance(1280, 720,ImageFormat.YUV_420_888, 2)) and obtain its Surface; construct a CaptureRequest using the TEMPLATE_PREVIEW template, set the resolution using captureRequestBuilder.set(CaptureRequest.SCALER_CROP_REGION, new Rect(0, 0, 1280, 720)), and set the frame rate to 24fps using set(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, new Range<>(24, 24));

[0259] Video stream configuration: Create a MediaCodec (MediaCodec.createEncoderByType("video / avc")), configure it to 720P / 24fps / H.264, and obtain its input Surface; construct a CaptureRequest using the TEMPLATE_RECORD template and associate it with the Surface.

[0260] S3.2.2, A-Adapter Interception and Conversion Process:

[0261] A-Adapter intercepts requests through the CameraCaptureSession.setRepeatingRequest() interface and calls parseCaptureRequest(captureRequest, surfaces) to parse the parameters:

[0262] Extract the stream type: Based on the CaptureRequest template, TEMPLATE_PREVIEW corresponds to the preview stream, TEMPLATE_RECORD corresponds to the video stream, and TEMPLATE_STILL_CAPTURE corresponds to the photo stream;

[0263] Extracting resolution: Obtain the width and height using CaptureRequest.SCALER_CROP_REGION, such as rect.width()=1280, rect.height()=720, and convert it to 1280*720;

[0264] Frame rate extraction: Obtain the upper limit of CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, such as range.getUpper()=24;

[0265] Extracting application information: Get the appId, such as com.android.camera, through context.getPackageName(); determine whether it is a system application through PackageManager.getApplicationInfo(), identified by ApplicationInfo.FLAG_SYSTEM.

[0266] Generate a CrossSystemRequest, whose format is consistent with the OpenHarmony side, except that the source field is set to Android. An example is shown below:

[0267] {

[0268] "source": "Android",

[0269] "appId": "com.android.camera",

[0270] "appType": "SYSTEM",

[0271] "requestId": "req_12345679",

[0272] "streams": [

[0273] {

[0274] "type": "preview",

[0275] resolution: 1280x720

[0276] "fps": 24,

[0277] format: "YUV420_888",

[0278] "surfaceId": "image_reader_001",

[0279] "permLevel": "CAMERA_BASIC

[0280] } ]

[0282] }

[0283] The request is sent to CA-Arbiter via the IPC channel within the container, such as the Socket path / dev / socket / android_cam_ipc based on AF_UNIX. Socket communication uses binary transmission to avoid the loss of JSON text parsing. The data format is that the request length is 4 bytes and the JSON data is N bytes.

[0284] S3.3 Resource slice matching and allocation.

[0285] S3.3.1 Request Priority Preprocessing: The CA-Arbiter worker thread retrieves a CrossSystemRequest from the request receiving queue and calls calculateRequestPriority(request) to calculate the initial priority, which is used for subsequent conflict arbitration reference and does not affect the matching result. The priority calculation formula is as follows:

[0286] Initial priority = Base score for operation type * Weight for application type + Bonus score for permission level

[0287] The base score for operation type is sorted by importance: 100 points for video recording, 80 points for photo taking, and 50 points for previewing. If the request contains multiple streams, such as previewing and taking photos, the base score of the highest stream type is taken.

[0288] Application type weights: OpenHarmony system applications are 1.2, Android system applications are 1.0, OpenHarmony third-party applications are 0.9, and Android third-party applications are 0.8.

[0289] Permission level bonus points: CAMERA_RECORD +10 points, CAMERA_CAPTURE +5 points, CAMERA_BASIC 0 points;

[0290] Example: A recording request from an OpenHarmony system application, including preview, with an initial priority of 100 × 1.2 + 10 = 130 points.

[0291] S3.3.2 Execution of the three-level slice matching algorithm: CA-Arbiter sends a matchSlices(requestJson) request to HCS-Slicer, and HCS-Slicer performs the following three-level matching for each stream:

[0292] Level 1 matching, or exact matching, involves iterating through the idle slice library to find slices with identical resolution, frame rate not less than the requested frame rate with the smallest difference, and identical format. The matching logic is as follows:

[0293] Resolution matching: The width and height of the slice resolution are exactly the same as the requested resolution. For example, if the request is 1920*1080, only slices with a resolution of 1920*1080 will be matched.

[0294] Frame rate matching: The slice frame rate is not less than the requested frame rate, and the difference between the slice frame rate and the requested frame rate is minimal. For example, if the requested frame rate is 24fps, a 30fps slice is matched first, rather than a 60fps slice, to reduce frame rate adjustment losses.

[0295] Format matching: The slice data format must be exactly the same as the request format. For example, if the request is for YUV420_888, only slices with the format YUV420_888 will be matched.

[0296] If a slice that meets the conditions is found, the slice is marked as pending binding, and the process proceeds to the next step of stream matching; if no slice is found, the process proceeds to the second-level matching.

[0297] Secondary matching, also known as downgraded compatibility or resolution downgrade: When exact matching fails, such as when a 1080P slice is already occupied, a free slice with a resolution higher than the requested resolution, a frame rate not lower than the requested frame rate, and the same format is searched. The core logic is as follows:

[0298] Resolution filtering: The product of the width and height of the slice resolution is greater than the product of the width and height of the requested resolution. For example, if you request 1080P (2073600 pixels), you can filter slices that are 2K (3686400 pixels) or 4K (8294400 pixels).

[0299] Priority sorting: Sort by the difference between the product of the slice resolution width and height and the product of the requested resolution width and height, from smallest to largest. For example, prioritize 2K slices over 4K slices because the downsampling scaling ratio from 2K to 1080P is 0.84, which is closer to 1, resulting in less image loss.

[0300] Processing type: If a match is found, mark the slice's processFlag as needing downsampling and record the downsampling parameters, such as a scaling ratio of 0.84 from 2K to 1080P and a target width and height of 1920*1080; if no match is found, proceed to level three matching.

[0301] Level 3 matching, i.e., frame rate compatibility and frame rate limitation: When level 2 matching fails, such as when all high-resolution slices are occupied, a free slice with the exact same resolution, frame rate greater than the requested frame rate, and consistent format is searched. The core logic is as follows:

[0302] Resolution and format matching: Same as exact matching.

[0303] Frame rate filtering: If the slice frame rate is greater than the requested frame rate, such as requesting 24fps, filter slices with 30fps and 60fps.

[0304] Mark processing type: After a successful match, mark the slice's processFlag as requiring frame rate limit and record the frame skipping strategy, such as discarding the 5th frame every 5 frames from 30fps to 24fps; discarding the 3rd and 5th frames every 5 frames from 60fps to 24fps; if no match is found, return a match failure.

[0305] S3.3.3 Matching Result Feedback and Application Adaptation:

[0306] Successful match processing: HCS-Slicer assigns a unique slice ID to each stream, such as slice_1080p_30_yuv420 for the preview stream and slice_4k_30_yuv420 for the photo stream. It updates the sliceMap with isFree=false, boundAppId=application ID, and streamType=stream type, generating a JSON-formatted MatchResult containing a list of requestId, sliceId, and processFlag, which is then sent to CA-Arbiter via a socket. CA-Arbiter returns the MatchResult to the application through an adapter.

[0307] OpenHarmony applications: Upon returning via the CameraCaptureSession.StateCallback.onConfigured() callback, the application calls session.startRepeatingRequest() to start the stream;

[0308] Android applications: The application calls session.setRepeatingRequest() to start the stream after returning via the CameraCaptureSession.StateCallback.onConfigured() callback.

[0309] Match failure handling: HCS-Slicer returns a MatchError containing requestId, errorCode, and the maximum supported parameters supportedParams; CA-Arbiter returns an error through the adapter.

[0310] OpenHarmony application: Callback to onConfigureFailed(), error message: Unable to meet the requested resolution of 4K, please try 1080P;

[0311] Android application: Callback to onConfigureFailed(), error message: Camera error: Requestedresolution not supported.

[0312] S4. Physical camera startup and data path binding: After resource slicing is completed, the system starts the physical camera and establishes a direct data path between the dual system application and the slice.

[0313] S4.1 Physical camera parameter configuration and startup.

[0314] S4.1.1 Baseline Configuration Calculation Mechanism: HCS-Slicer calls the calculateBaseConfig(sliceIdList) function to determine the baseline configuration according to the principle of maximizing the satisfaction of all slice requirements and compatibility with hardware capabilities.

[0315] Baseline resolution calculation:

[0316] Collect the resolutions of all assigned slices, such as 3840*2160, 1920*1080, and 1280*720;

[0317] Calculate the total number of pixels for each resolution, i.e., width * height, and select the resolution with the largest total number of pixels as the candidate baseline resolution, such as 3840*2160, which has the largest total number of pixels of 8,294,400.

[0318] Verify hardware compatibility: Call the checkHardwareSupport(candidateRes, candidateFps) interface to confirm whether the hardware supports the combination of this resolution and the subsequent calculated baseline frame rate. For example, if the candidate resolution is 4K, and the hardware only supports 4K frame rate of 24fps, then the subsequent baseline frame rate needs to be adjusted to 24fps.

[0319] Final baseline resolution: If the candidate resolutions are compatible, they are determined as the baseline resolution; if they are not compatible, the resolution with the second largest total number of pixels, such as 2K, is selected, and the verification is repeated until a compatible resolution is found.

[0320] Baseline frame rate calculation:

[0321] Collect the requested frame rates for all allocated slices, such as 30fps, 24fps, and 15fps;

[0322] Select the maximum requested frame rate as the candidate baseline frame rate, such as 30fps;

[0323] Verify hardware compatibility: Confirm whether the hardware supports the candidate frame rate at the base resolution. For example, if the base resolution is 4K and the hardware supports 4K frame rate of 30fps, then the candidate frame rate of 30fps is valid.

[0324] Final baseline frame rate: If the candidate frame rate is compatible, it is determined as the baseline frame rate; if it is incompatible, such as the hardware only supporting 4K frame rate 24fps, the baseline frame rate is adjusted to the maximum frame rate supported by the hardware 24fps, and all slices are notified to adjust according to the new frame rate in the future.

[0325] The baseline format is determined as follows: YUV420_888 is selected uniformly, with a native hardware support rate of no less than 99%, and is compatible with the subsequent processing needs of preview, photo taking, and video recording.

[0326] Example calculation results: The slice list contains 4K frame rate 30fps, 1080P frame rate 24fps, and 720P frame rate 15fps. If the hardware supports 4K frame rate 30fps, then the baseline configuration is 4K, 30fps, YUV420_888.

[0327] S4.1.2, Full Process of Driver Parameter Configuration:

[0328] Open the driver node: Call open(" / dev / video0", O_RDWR) to open the camera driver and obtain the file descriptor fd;

[0329] Set the base format: Construct a v4l2_format structure, set type=V4L2_BUF_TYPE_VIDEO_CAPTURE, fmt.pix.width=3840, fmt.pix.height=2160, fmt.pix.pixelformat=V4L2_PIX_FMT_YUV420, and call ioctl(fd, VIDIOC_S_FMT, &fmt) to set the format;

[0330] Set the baseline frame rate: Construct a v4l2_streamparm structure, set type=V4L2_BUF_TYPE_VIDEO_CAPTURE, parm.capture.timeperframe.numerator=1, parm.capture.timeperframe.denominator=30, which means 1 / 30th of a second per frame, i.e., 30fps, and call ioctl(fd, VIDIOC_S_PARM, &parm) to set the frame rate;

[0331] Allocate buffers: Construct a v4l2_requestbuffers structure, set count=4, allocate 4 buffers, type=V4L2_BUF_TYPE_VIDEO_CAPTURE, memory=V4L2_MEMORY_MMAP, and call ioctl(fd,VIDIOC_REQBUFS, &reqbufs) to allocate buffers;

[0332] Mapping Buffers: For each buffer, construct a v4l2_buffer structure, call ioctl(fd, VIDIOC_QUERYBUF, &buf) to obtain buffer information, and then call mmap(NULL, buf.length, PROT_READ|PROT_WRITE, MAP_SHARED, fd, buf.m.offset) to map the buffer to user space for receiving raw data output by the driver.

[0333] S4.1.3, Camera Startup and Data Output:

[0334] Start the stream: Construct a v4l2_buffer structure, set type=V4L2_BUF_TYPE_VIDEO_CAPTURE, memory=V4L2_MEMORY_MMAP, index=0~3, traverse the 4 buffers, call ioctl(fd, VIDIOC_QBUF, &buf) to put the buffer into the driver's output queue; call ioctl(fd, VIDIOC_STREAMON, &type) to start the stream, and the driver begins to push raw frame data to the mapped buffers, with an interval of approximately 33ms between each frame;

[0335] Data reception: HCS-Slicer creates a data reception thread, which repeatedly calls ioctl(fd, VIDIOC_DQBUF, &buf) to retrieve a buffer filled with data from the driver queue, reads the raw YUV data, stores it in the memory pointed to by buf.m.userptr, and after processing, calls ioctl(fd, VIDIOC_QBUF, &buf) to put the buffer back into the driver queue, thus realizing continuous data reception.

[0336] S4.2, Binding of application data paths between dual systems.

[0337] S4.2.1, Slice and Shared Memory Block Binding: HCS-Slicer calls the bindSliceToMemPool(sliceIdList) function to allocate raw data blocks and metadata blocks from CS-MemPool to each slice. The binding rules are as follows:

[0338] Raw data block allocation: Allocation is based on slice resolution size. 4K slices are allocated 12MB blocks, 1080P slices are allocated 6MB blocks, that is, 1920*1080*1.5=3110400 bytes, which is about 3MB, requires 6MB of reserved space. 720P slices are allocated 3MB blocks. They are distinguished by memory offset. For example, the address range of 12MB block is 0xFFFF00000000 to 0xFFFF0000C000, and the address range of 6MB block is 0xFFFF0000C000 to 0xFFFF00012000.

[0339] Metadata block allocation: Each slice is bound to one 4KB metadata block, such as slice 1 being bound to metadata block 0, slice 2 being bound to metadata block 1, and the metadata block is associated with the original data block through the slice ID;

[0340] Binding relationship storage: Update the original data block address memBlockAddr and metadata block address metaBlockAddr fields in sliceMap to ensure that memory blocks can be quickly located during subsequent data distribution.

[0341] S4.2.2, OpenHarmony application of zero-copy data path:

[0342] Shared memory mapping: OH-Adapter calls OH_Ashmem_Create("camera_shm",memBlockSize) to create an Ashmem object, where memBlockSize is the size of the original data block corresponding to the slice, such as 12MB. OH_Ashmem_SetProtect(ashmemFd, PROT_READ|PROT_WRITE) is called to set read and write permissions. mmap(NULL, memBlockSize, PROT_READ|PROT_WRITE, MAP_SHARED, ashmemFd, 0) is called to map the Ashmem to user space. The mapped address is consistent with the original data block address of the slice in CS-MemPool, and is set through OH_Ashmem_SetBaseAddr.

[0343] Surface and shared memory binding: The application obtains the Surface object through SurfaceView.getSurface(), and OH-Adapter calls OH_Surface_AttachBuffer(surface, ashmemFd, memBlockSize, memBlockAddr) to bind the Ashmem to the Surface, so that the Surface's buffer directly points to the original data block of CS-MemPool to avoid data copying;

[0344] Data access path: from driver, HCS-Slicer, CS-MemPool raw data block is 12MB, Ashmem mapping, application Surface and then SurfaceView rendering, there is no data copying throughout the process, only memory address mapping, and the data transmission latency is no more than 15ms at 4K resolution.

[0345] S4.2.3, Zero-copy data path for Android applications:

[0346] Shared memory mapping: The A-Adapter calls shm_open(" / dev / shm / oh_cam_shm", O_RDWR,0666) to open the POSIX shared memory file, and calls mmap(NULL, memPoolTotalSize, PROT_READ|PROT_WRITE, MAP_SHARED, shmFd, 0) to map the entire shared memory pool to the user space of the Android container; by using the address of the original data block bound to the slice, such as 0xFFFF00000000, the offset relative to the base address of the shared memory pool is calculated to locate the original data block of the slice;

[0347] The GraphicBuffer is bound to shared memory: The A-Adapter calls GraphicBufferAllocator::get().allocate(width, height, format, 1, usage, &gb, &stride) to allocate a GraphicBuffer. The width and height are the slice resolution, the format is YUV420_888, and the usage is GRALLOC_USAGE_SW_READ_OFTEN | GRALLOC_USAGE_HW_TEXTURE. The gb->map() function is called to get the memory address of the GraphicBuffer. The original data block address of the slice in CS-MemPool is mapped to this address. The address mapping relationship is copied using memcpy, not data copying.

[0348] ImageReader and GraphicBuffer binding: After the application creates ImageReader, A-Adapter calls ImageReader::setBufferCount(2) to set the number of buffers, and calls ImageReader::setBuffers(gbList) to pass the list of GraphicBuffers bound to shared memory to ImageReader, so that the output buffer of ImageReader directly points to the original data block of CS-MemPool;

[0349] Data access path: driver, HCS-Slicer, CS-MemPool raw data block, POSIX shared memory mapping, GraphicBuffer, ImageReader and then to the application onImageAvailable() callback, with only 1 memory mapping and no data copying throughout the process, and the data transmission latency is no more than 20ms at 4K resolution.

[0350] S4.2.4, Metadata Path Binding:

[0351] The HCS-Slicer receives metadata such as exposure time and ISO pushed by the driver, and calls the fillMetaData(metaBlockAddr, exposureTime, iso, focusDistance) function to write the metadata into the metadata block bound to the slice. The metadata block size is 4KB. The metadata block format is shown in the table below. In the table, the time unit is microseconds, the ISO value is 100 or 200, the focus distance unit is meters, and the flash mode is 0 for off and 1 for on.

[0352] Table 3. Metadata Block Format Table 0-7 timestamp uint64 Timestamp 8-11 exposure uint32 Exposure time 12-15 iso uint32 ISO value 16-19 focus float Focusing distance 20-23 flashMode uint8 Flash mode

[0353] OH-Adapter or A-Adapter associates the metadata and original data by matching the timestamp of the metadata block with the timestamp of the original data block. This data is then converted into OpenHarmony's CameraFrameInfo and Android's CaptureResult, respectively, and returned to the application via a callback.

[0354] OpenHarmony application: CameraCaptureSession.FrameCallback.onFrameInfoAvailable();

[0355] Android application: CameraCaptureSession.CaptureCallback.onCaptureCompleted().

[0356] S5. Concurrent data distribution and real-time control command processing: After the physical camera is started, the system distributes data to each slice and processes control commands for dual-system applications.

[0357] S5.1, Raw Data Slicing and Real-time Distribution.

[0358] S5.1.1 Raw Data Reception and Preprocessing: The HCS-Slicer's data reception thread cyclically reads the raw frame data pushed by the driver, such as YUV420_888 format. The data processing flow for each frame is as follows:

[0359] Data integrity verification: Check whether the data length matches the baseline resolution. For example, the length of 4K data should be 3840*2160*1.5=11943936 bytes. If the length does not match, the frame is discarded to avoid corrupted data.

[0360] Timestamp generation: Call clock_gettime(CLOCK_MONOTONIC, &ts) to get the monotonic clock timestamp. The timestamp is in the microsecond range and serves as the unique time identifier for this frame.

[0361] Metadata extraction: Extract metadata from the additional information in the drive buffer, including exposure time, ISO, and focus distance, and temporarily store it in a temporary variable before writing it into the metadata block.

[0362] S5.1.2 Real-time processing of sliced ​​data based on processFlag:

[0363] Processing type 1: processFlag=0 means no special processing is required: the slice resolution, frame rate and baseline configuration are completely consistent. For example, if a 4K slice matches the 4K baseline configuration, the raw data is directly written to the raw data block bound to that slice.

[0364] Locating the memory block: Obtain the original data block address memBlockAddr of the slice through sliceMap;

[0365] Data writing: Call memcpy(memBlockAddr, rawData, dataLen) to write the raw data to the memory block. The 4K data copy takes about 1ms, which is optimized by CPU pipeline.

[0366] Mark valid: Set the isValid flag in the memory block header to true, with a size of 1 byte, where 0 indicates invalid and 1 indicates valid. Update the timestamp field to match the frame timestamp.

[0367] Processing Type 2: processFlag=1 indicates downsampling is required: the slice resolution is lower than the baseline resolution, such as a 1080P slice matched with a 4K baseline configuration, and downsampling is accelerated through hardware ISP.

[0368] Initialize ISP parameters: Call OH_Hardware_ISP_Init() to initialize the hardware ISP, set the input parameters including inWidth=3840, inHeight=2160, inFormat=YUV420_888, and the output parameters including outWidth=1920, outHeight=1080, outFormat=YUV420_888. The scaling algorithm is bilinear interpolation, which provides high image smoothness and fast processing speed.

[0369] Hardware-accelerated downsampling: Call OH_Hardware_ISP_Resize(rawData, inWidth, inHeight,outData, outWidth, outHeight). The ISP processes the three components Y, U, and V in parallel through the hardware pipeline, with a processing time of no more than 1ms / frame, which is much faster than the 5ms / frame of CPU software downsampling.

[0370] Data writing and marking: Write the downsampled outData to the original data block of the slice, and set isValid=true.

[0371] Processing type 3: processFlag=2 requires frame rate limitation: the slice frame rate is lower than the base frame rate, such as a 24fps slice matching a 30fps base configuration, and the output frame rate is controlled by frame skipping.

[0372] Frame counting and frame skipping judgment: Maintain an independent frame counter frameCount for each slice, initialized to 0; after receiving a frame of reference data, frameCount is incremented by 1, and the frame skipping strategy is used to determine whether to retain the frame. For example, from 30fps to 24fps, the frame skipping strategy is to discard the 5th frame every 5 frames. If frameCount%5 != 0, the frame is retained; if it equals 0, the frame is discarded.

[0373] Frame retention processing: If it is necessary to retain the original data, write the original data into the original data block of the slice, set isValid=true, and reset frameCount to 0;

[0374] Frame dropping: If a frame needs to be dropped, set isValid=false for the original data block of the slice. When the application reads the frame, it will skip the frame. The isValid flag is used to determine the frame rate and avoid abnormal frame rate.

[0375] S5.1.3 Metadata Writing and Data Readiness Notification:

[0376] Metadata writing: For each processed slice, HCS-Slicer calls the writeMetaData(sliceId,timestamp,exposureTime,iso,focusDistance) function to write the metadata to the metadata block bound to the slice, ensuring that the metadata timestamp is consistent with the original data block timestamp;

[0377] Data Ready Notification: The OH-Adapter or A-Adapter monitors the isValid flag of the original data block of the slice in real time through a memory block listening thread. When the flag changes from false to true, an application callback is triggered.

[0378] OpenHarmony application: Calling OH_Surface_SignalFrameAvailable(surface) triggers the Surface.onFrameAvailable() callback, and the application reads and renders the data;

[0379] Android application: Calling ImageReader::signalImageAvailable() triggers the ImageReader.OnImageAvailableListener.onImageAvailable() callback, allowing the application to obtain and process the Image.

[0380] S5.2, Dual-system application data consumption.

[0381] S5.2.1, OpenHarmony application data consumption process:

[0382] Preview stream consumption:

[0383] The application detects the new frame through the Surface.onFrameAvailable() callback, calls Surface.lockCanvas(&rect) to lock the canvas, and obtains the address of the YUV data in shared memory;

[0384] Call OH_Graphic_Bitmap_CreateFromYUV(yuvData, width, height, format) to convert YUV data into a Bitmap object;

[0385] Call Canvas.drawBitmap(bitmap, matrix, paint) to render the Bitmap onto the SurfaceView, and call Surface.unlockCanvasAndPost(canvas) to release the canvas;

[0386] Performance metrics: At 4K resolution, the preview frame rate is stable at 28 to 30 fps, and the CPU utilization is no more than 15%;

[0387] Photo stream consumption:

[0388] The application calls CameraCaptureSession.capture(captureRequest, captureCallback, handler) to initiate a photo capture request;

[0389] The OH-Adapter intercepts the request and reads the current frame's YUV data from the raw data block of the 4K slice, specifically the 4K data pointed to by memBlockAddr;

[0390] Call OH_Image_Encode_JPEG(yuvData, width, height, quality, jpegData, &jpegLen) to perform hardware JPEG encoding, with quality=90 and encoding time not exceeding 50ms;

[0391] The JPEG data is returned to the application via the CaptureCallback.onCaptureCompleted() callback, and the application calls FileOutputStream.write(jpegData) to store it locally, such as / storage / emulated / 0 / DCIM / Camera / photo.jpg;

[0392] Video stream consumption:

[0393] The application creates a MediaCodec (MediaCodec.createEncoderByType("video / avc")), configures it for 4K, 30fps, H.264, and obtains its input Surface;

[0394] The recording stream is associated with the Surface by CameraCaptureSession.setRepeatingRequest(), and MediaCodec directly reads YUV data from shared memory for encoding.

[0395] The encoded H.264 stream is output via the MediaCodec.OnFrameAvailableListener callback. The application calls MediaMuxer.writeSampleData() to encapsulate the stream into an MP4 file with a bitrate of 15Mbps. At 4K resolution, the file size is approximately 6.75GB per hour.

[0396] S5.2.2, Android application data consumption process:

[0397] Preview stream consumption:

[0398] The application obtains a new frame through the ImageReader.OnImageAvailableListener.onImageAvailable(imageReader) callback and calls Image image = imageReader.acquireNextImage() to obtain the Image object;

[0399] Call Image.Plane[] planes = image.getPlanes() to get the data of the Y, U, and V planes, and read the Y component data through ByteBuffer buffer = planes[0].getBuffer();

[0400] The YUV data is converted to an OpenGL texture by calling SurfaceTexture.updateTexImage(), and the texture is bound by calling GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_LUMINANCE, width, height, 0, GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, buffer);

[0401] Call GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, 4) to render the texture to the TextureView, and call image.close() to release the Image object;

[0402] Performance metrics: At 720P resolution, the preview frame rate is stable at 23 to 24 fps, and the CPU utilization is no more than 12%;

[0403] Video stream consumption: The application creates a MediaCodec (MediaCodec.createEncoderByType("video / avc")), configures it to 720P, 24fps, H.264, and 5Mbps bitrate, and obtains its input Surface;

[0404] The recording stream is associated with the Surface by CameraCaptureSession.setRepeatingRequest(), and MediaCodec reads YUV data from the ImageReader's Surface and directly maps it to shared memory;

[0405] The encoded H.264 bitstream is output through the MediaCodec.Callback.onOutputBufferAvailable() callback. The application creates a MediaMuxer (new MediaMuxer(outputPath,MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4)) to encapsulate the bitstream and the audio stream captured by AudioRecord into an MP4 file.

[0406] Performance metrics: The recording frame rate is stable at 23 to 24 fps at 720P resolution, with a file size of approximately 2.25 GB per hour.

[0407] S5.3, Cross-system control command synchronization and execution.

[0408] S5.3.1, Unified Control Command Format Definition: The ControlCommand structure is designed as a cross-system universal command format, supporting control operations for all cameras. The structure definition is as follows. If implemented in C language, ensure memory alignment:

[0409] typedef struct {

[0410] char cmdId

[16] ; / / Instruction ID, enumeration values: FLASH, FOCUS_TRIGGER, ZOOM, EXPOSURE

[0411] char value

[32] ; / / Instruction value, such as FLASH ON, OFF, AUTO

[0412] char source[8]; / / Instruction source: OH, Android

[0413] char appId

[256] ; / / Application ID for sending the command

[0414] char sliceId

[64] ; / / Associated slice ID

[0415] uint64_t timestamp; / / Command sending timestamp, in microseconds

[0416] uint32_t crc32; / / CRC32 checksum, ensuring instruction integrity

[0417] ControlCommand;

[0418] The value ranges of cmdId and value for core commands are shown in the table below.

[0419] Table 4 shows the range of cmdId and value values ​​for commands. "FLASH" Flash control "ON" / "OFF" / "AUTO" "FOCUS_TRIGGER" Focus trigger "START" / "CANCEL" "ZOOM" Zoom adjustment "1.0" / "2.0" / "5.0" / "10.0" (multiples) "EXPOSURE" Exposure compensation -2" / -1" / 0" / +1" / +2" (EV value)

[0420] S5.3.2, Instruction Generation and Transmission Process:

[0421] OpenHarmony application command sending:

[0422] The application calls CameraDevice.setFlashMode(FLASH_MODE_ON);

[0423] The OH-Adapter intercepts requests and constructs a ControlCommand: cmdId="FLASH", value="ON", source="OH", appId=application package name, sliceId=associated slice ID, timestamp=current timestamp;

[0424] Calculate the CRC32 checksum. Perform CRC32 calculations on the cmdId, value, source, appId, sliceId, and timestamp fields, and fill the CRC32 fields accordingly.

[0425] Locate the control command block of CS-MemPool: Find the currently writable command block through the write pointer of the circular buffer, and write the ControlCommand structure with a size of sizeof(ControlCommand) = 384 bytes;

[0426] Update the write pointer: writePtr = (writePtr + 1) % 4, where 4 is the number of instruction blocks, and complete the instruction writing.

[0427] Android application command sending:

[0428] The application calls captureRequestBuilder.set(CaptureRequest.FLASH_MODE,CaptureRequest.FLASH_MODE_TORCH);

[0429] The A-Adapter intercepts the request and converts it into a ControlCommand: cmdId="FLASH", value="ON", source="Android", with other fields the same as the OH side;

[0430] Calculate the CRC32 checksum to ensure the instruction has not been tampered with;

[0431] Access the control command block via POSIX shared memory mapping, write it to the Control Command structure according to the circular buffer rule, and update the write pointer.

[0432] S5.3.3, Arbitration and Execution of Instructions:

[0433] Instruction extraction and verification: The instruction processing thread of HCS-Slicer scans the control instruction block every 1ms and determines whether there is a new instruction by the difference between the read pointer and the write pointer. For new instructions, the CRC32 value is first checked. If it does not match, it is discarded to avoid invalid instructions. Then, the ControlCommand is extracted and sent to CA-Arbiter.

[0434] CA-Arbiter arbitration logic: After receiving the instruction, CA-Arbiter determines whether there is a conflict based on the instruction type. For example, if the FLASH instruction has both ON and OFF, there is a conflict. If there is a conflict, the conflict arbitration rules are called to handle it; if there is no conflict, it is passed directly.

[0435] Instruction execution: CA-Arbiter returns the instruction that arbitration passed, and HCS-Slicer calls the driver interface to execute it.

[0436] Flash command cmdId=FLASH: calls ioctl(fd, VIDIOC_S_CTRL, &ctrl), ctrl.id=V4L2_CID_FLASH_LED_MODE, ctrl.value is mapped according to value, that is, ON is mapped to 1, OFF is mapped to 0, and AUTO is mapped to 2;

[0437] The zoom command cmdId=ZOOM: Parses the value as a floating-point value, such as 2.0, and calls ioctl(fd, VIDIOC_S_CTRL, &ctrl), ctrl.id=V4L2_CID_ZOOM_ABSOLUTE, ctrl.value=zoom factor * 100, which is usually expressed in one percent by hardware.

[0438] Results Feedback: After execution, HCS-Slicer generates a CommandResult, which includes cmdId, status, and errorMsg. The status is either SUCCESS or FAILED, and this result is returned to the application via the adapter.

[0439] OpenHarmony application: CameraDevice.Callback.onCommandCompleted();

[0440] Android application: CameraCaptureSession.CaptureCallback.onCaptureCompleted().

[0441] S6. Conflict arbitration based on rigid rules.

[0442] When multiple application requests cause resource conflicts, CA-Arbiter arbitrates in real time based on preset rigid rules to ensure the core experience.

[0443] S6.1 Conflict type identification and triggering mechanism.

[0444] S6.1.1, Refined Definition of Conflict Types:

[0445] Parameter conflicts: These can be further divided into resolution conflicts, such as two applications requesting 4K but the hardware only supports single 4K output; frame rate conflicts, such as the total frame rate requirement exceeding the hardware limit, 30fps + 30fps > 50fps; and format conflicts, such as applications requesting incompatible formats, such as one requiring YUV420 and the other requiring RAW10, causing the hardware to be unable to output simultaneously.

[0446] Mutual exclusion resource conflicts: such as the flash can only be ON or OFF at a time, the laser autofocus cannot be occupied by other applications when it is started, and the HDR mode is mutually exclusive with the normal mode when it is turned on. Each type of resource is marked by a MutexResource structure, including resourceId, currentOwner, and isLocked.

[0447] Bandwidth conflicts: memory bus bandwidth conflicts are further subdivided by data type, i.e., total data volume / time > bus bandwidth, such as 4K / 30fps + 1080P / 30fps = 4.5GB / s > 4GB / s; encoding bandwidth conflicts, i.e., total bitrate of multi-application encoding > hardware encoder limit, such as 4K / 15Mbps + 720P / 5Mbps = 20Mbps > 18Mbps.

[0448] Operational conflicts: In addition to the conflict between taking photos and recording videos, a new conflict has been added between continuous shooting and recording videos. That is, continuous shooting requires a high frame rate per frame, which affects the stability of recording videos; zoom and autofocus conflict, that is, autofocus must be paused when zooming manually.

[0449] S6.1.2 Conflict Triggering Timing and Detection Indicators:

[0450] Real-time detection cycle: CA-Arbiter scans the system status every 5ms, including slice parameters, resource usage, bandwidth usage, and triggers conflict detection;

[0451] Parameter conflict detection:

[0452] Resolution conflict: Count the number of applications requesting 4K resolution. If the number is greater than 1 and the hardware does not support dual 4K output, a conflict will be triggered.

[0453] Frame rate conflict: Calculate the sum of the requested frame rates of all slices. If it is greater than the hardware's maximum total frame rate, such as 50fps, a conflict is triggered.

[0454] Mutex resource conflict detection: Maintain a mutexResourceMap, key: resourceId, value: MutexResource. A conflict is triggered when the resourceId of a new instruction is already occupied by the currentOwner and has not been released.

[0455] Bandwidth conflict detection:

[0456] Memory bus bandwidth: Obtain the real-time bus bandwidth using dmesg | grep "bus bandwidth" and calculate the current total bandwidth, which is: Σ resolution * frame rate * 1.5 bytes / pixel. If the current total bandwidth is greater than 90% of the bus bandwidth, a 10% redundancy is reserved, and a conflict is triggered.

[0457] Encoding bandwidth: The real-time bitrate is obtained through the hardware encoder driver / sys / class / video_encoder / bitrate. If the total bitrate exceeds the encoder's upper limit, a conflict is triggered.

[0458] Operation mutual exclusion conflict detection: Monitor the application operation sequence, and if the timestamp difference between takePicture() and startRecording() is less than 100ms, a conflict is triggered.

[0459] S6.2 Rigid Arbitration Rules and Enforcement Procedures.

[0460] S6.2.1, Rule 1: Operation type priority rule.

[0461] Priority weight: Video recording has a weight of 1.0, photo taking has a weight of 0.8, and preview has a weight of 0.5. The higher the weight, the more likely the original parameters will be preserved.

[0462] Parameter adjustment algorithm:

[0463] Calculate the priority difference between conflicting operations, such as the difference between a recording weight of 1.0 and a preview weight of 0.5, which is 0.5;

[0464] The principle for adjusting low-priority operation parameters according to the difference ratio is as follows: resolution adjustment ratio = 1 - difference, such as when it is 0.5, the resolution is reduced to 50%; frame rate adjustment ratio = 1 - difference, such as when it is 0.5, the frame rate is reduced to 50%.

[0465] Typical scenario processing:

[0466] Scenario 1: Application A is recording at 1080P and 30fps, while application B is previewing at 4K and 30fps. Application A and application B conflict. The priority difference of B is 1.0 - 0.5 = 0.5. Therefore, adjust the resolution of B to 4K × 50% = 2K and the frame rate to 30fps × 50% = 15fps. The total bandwidth is reduced from 4.5GB / s to 2.25GB / s.

[0467] Scenario 2: Application A is for taking photos in 4K resolution, while application B is for recording videos in 1080P resolution at 30fps. Application A and application B conflict. The priority difference of A is 1.0 - 0.8 = 0.2. Therefore, the resolution of A is adjusted to 4K × 80% = 3.2K to balance photo quality and video recording stability.

[0468] S6.2.2, Rule 2: System type priority rule.

[0469] The priority weights are shown in the table below.

[0470] Table 5 Application Type Priority Weight Table OpenHarmony system applications 1.2 For example, the system camera performs basic functions. Android system applications 1.0 For example, Android containers have built-in cameras. OpenHarmony third-party applications 0.9 Non-system pre-installed OH applications Android third-party applications 0.8 Android applications not pre-installed with the system

[0471] Parameter adjustment strategy:

[0472] Calculate the weight ratio of conflicting applications. For example, the ratio of an OH system application with a weight of 1.2 to an Android third-party application with a weight of 0.8 is 1.5.

[0473] The parameter adjustment ratio for low-weight applications is 1 - 1 / weight ratio. For example, when the weight is 1.5, it is 1 - 1 / 1.5, which is approximately 0.33, so the parameter is reduced by 33%.

[0474] Typical scenario processing:

[0475] The OpenHarmony system camera is used for recording with parameters of 4K, 30fps, and weight 1.2, which conflicts with the Android third-party application's preview parameters of 1080P, 30fps, and weight 0.8. Since the weight ratio is 1.5, the frame rate of the third-party application will be adjusted to 30×(1-0.33)=20fps, reducing the total bandwidth by 10%.

[0476] S6.2.3, Rule 3: Mutually Exclusive Resource Exclusivity Rule.

[0477] Resource exclusivity priority: Combining rule 1 and rule 2, the calculation formula is: Resource priority = operation type weight × system type weight;

[0478] Design of the queue to be executed:

[0479] Queue structure: Priority queue PriorityQueue is used, and each element is a PendingCommand containing cmdId, value, source, appId, and priority;

[0480] Triggering condition: Queue elements are executed only when the current occupant releases the resource, such as when a flashlight changes from ON to OFF;

[0481] Timeout mechanism: Queue elements that have not been executed within 3 seconds are automatically discarded to avoid resources being occupied for a long time;

[0482] Typical scenario processing:

[0483] Application A is an OH system camera, recording, flash ON, with priority = 1.0 × 1.2 = 1.2, which conflicts with application B, an Android third-party application, preview, flash OFF, with priority = 0.5 × 0.8 = 0.4. In this case, the ON instruction of A is executed, and the OFF instruction of B is stored in the execution queue. When A sends the OFF instruction to turn off the flash, the CA-Arbiter retrieves the instruction of B from the queue and executes it.

[0484] S6.2.4, Rule 4: Bandwidth overload protection rule.

[0485] Level 1 adjustment, meaning a year-on-year reduction in frame rate:

[0486] The overload ratio is calculated as (current total bandwidth - bus bandwidth) / bus bandwidth. For example, if 4.5GB / s - 4GB / s = 0.5GB / s, then the overload ratio is 12.5%.

[0487] All slice frame rates are reduced by the overload ratio, such as 30fps being reduced to 30×(1-12.5%)=26.25fps, which is rounded down to 26fps;

[0488] Level 2 adjustment, which involves downgrading lower-priority resolutions, occurs if the system is still overloaded after Level 1 adjustment. Resolutions are then reduced from lowest to highest priority.

[0489] Calculate the bandwidth reduction required = current total bandwidth - bus bandwidth. If it is still 0.3GB / s short, select the lowest priority slice and reduce its resolution by one level, such as from 1080P to 720P, which will reduce the bandwidth by about 75%. Repeat until the total bandwidth is no greater than the bus bandwidth.

[0490] Typical scenario processing:

[0491] The total bandwidth is 4.5GB / s, and the bus limit is 4GB / s. The first-level adjustment is to reduce all frame rates from 30fps to 26fps. The total bandwidth is 4.5×26 / 30=3.9GB / s, which is not greater than the total limit of 4GB / s. Therefore, no second-level adjustment is needed.

[0492] S6.2.5, Rule 5: Operation mutual exclusion coordination rule.

[0493] Time window control: The photo-taking process is limited to a 1-frame time window, approximately 33ms at 30fps, to reduce the impact on video recording;

[0494] Step-by-step execution process:

[0495] Pre-notification: CA-Arbiter sends a TEMP_PAUSE notification to the recording application, which then pauses MediaCodec encoding while preserving the encoding context;

[0496] Parameter switching: HCS-Slicer temporarily switches the hardware resolution to the high resolution requested for taking a picture, such as switching from 1080P to 4K, in no more than 10ms.

[0497] Single frame capture: The driver outputs 1 frame of 4K data for JPEG encoding, with a time of no more than 16ms;

[0498] Recovery parameters: HCS-Slicer switches the hardware resolution back to 1080P, applies recovery encoding to the recording, inserts I-frames to ensure continuous footage, and takes no more than 7ms.

[0499] Performance metrics: The recording frame rate only briefly dropped to 24fps within 33ms, and then recovered to 30fps, with no noticeable stuttering for the user.

[0500] S6.3 Synchronization and application adaptation of conflict resolution results.

[0501] S6.3.1, Arbitration result structure generation: CA-Arbiter generates a ConfigurationChange object, which includes: associated request Id, affected slice Id, original parameters oldParams, adjusted parameters newParams, and conflict reason, such as bandwidth overload.

[0502] S6.3.2, Cross-system result synchronization:

[0503] OpenHarmony applications: The OH-Adapter sends the new resolution / frame rate via the CameraKit.onConfigurationChanged(ConfigurationChange) callback, and the application can obtain the new resolution / frame rate via getNewParams().

[0504] Android application: A-Adapter sends a message via CameraCaptureSession.StateCallback.onCaptureConfigurationChanged(ConfigurationChange) to simulate Android's CameraMetadata change logic;

[0505] S6.3.3, Application of adaptive adjustment:

[0506] Resolution adjustment: The application rebuilds Surface / ImageReader to adapt to the new resolution, such as from 1080P to 720P, and the Surface size is modified accordingly;

[0507] Frame rate adjustment: When the application updates MediaCodec encoding parameters, such as adjusting from 30fps to 24fps, the bitrate is reduced by 20% accordingly.

[0508] S6.3.4 System Configuration Linkage Update: HCS-Slicer updates the slice's processFlag based on newParams. If a slice that originally needed downsampling may not need processing due to resolution adjustment, the hardware baseline configuration is recalculated. If the baseline frame rate can be appropriately increased after the total bandwidth is reduced, the driver interface is called to update the parameters with a delay of no more than 5ms.

[0509] S7. Resource release and slice recycling: When the application exits or the camera is turned off, the system releases resources in a timely manner to avoid waste.

[0510] S7.1, Application Active Release Process.

[0511] S7.1.1 Release Trigger and Request Generation:

[0512] OpenHarmony application: Call CameraDevice.close() to perform a capture operation through OH-Adapter, generate ReleaseRequest, which includes appId, sliceIdList of all slices bound to the application, and releaseType="ACTIVE" to indicate active release;

[0513] Android application: Call CameraDevice.close() to generate a ReleaseRequest via A-Adapter;

[0514] S7.1.2, Slice Unbinding and Status Update:

[0515] After receiving the ReleaseRequest, CA-Arbiter calls releaseSlices(request) to send an unbinding command to HCS-Slicer;

[0516] HCS-Slicer iterates through sliceIdList, marks slices as free in sliceMap (i.e., isFree=true), and clears fields such as boundAppId and streamType.

[0517] Record the release log at / data / log / camera_slice_release.log, which includes appId, sliceId, and releaseTime, for troubleshooting purposes;

[0518] S7.1.3, Memory Mapping Unmapping and Resource Cleanup:

[0519] On the OpenHarmony side: OH-Adapter calls OH_Surface_DetachBuffer(surface) to unbind the Surface from the shared memory block, and calls munmap(addr, size) to release the Ashmem mapping;

[0520] On the Android side: A-Adapter calls ImageReader::close() to release GraphicBuffer and calls munmap(addr, size) to remove POSIX shared memory mapping;

[0521] Clean up any unexecuted instructions for the application in the control block, filtering by appId and marking them as invalid.

[0522] S7.2, Abnormal Release Handling.

[0523] S7.2.1 Real-time Process Crash Detection:

[0524] On the OpenHarmony side: OH-Adapter registers a process observer through AbilityManager.addProcessObserver(). When the application process state changes to PROCESS_STATE_DEAD, the onProcessDied(appId) callback is triggered to automatically generate a ReleaseRequest (releaseType="CRASH").

[0525] On the Android side: A-Adapter checks the application process status periodically (e.g., every 100ms) using ActivityManager.getRunningAppProcesses(). If the process is not in the running list, a ReleaseRequest is generated.

[0526] S7.2.2, Timeout Release Mechanism:

[0527] HCS-Slicer maintains a timestamp lastAccessTime for each slice, representing the last time the application read data.

[0528] The periodic scanning thread calculates currentTime - lastAccessTime. If the result is greater than 3 seconds, this time threshold can be configured in / etc / camera_config.xml as timeoutThreshold to determine that the application is stuck and force the generation of ReleaseRequest (releaseType="TIMEOUT").

[0529] S7.2.2 Resource Leakage Protection:

[0530] CA-Arbiter maintains an appSliceCountMap, where the key is the appId and the value is the number of slices bound to the app. A threshold is set, such as 5 slices per app.

[0531] If the number of slices for an application exceeds a threshold, an alert is triggered, such as dmesg printing "Possible resourceleak in app [appId]", and all slices for the application are forcibly released, retaining only one basic preview slice.

[0532] S7.3, Dynamic adjustment of hardware resources.

[0533] Remaining Slice Parameter Analysis: After each slice is released, HCS-Slicer calls analyzeRemainingSlices() to calculate the maximum resolution, maximum frame rate, and total bandwidth requirement of the remaining slices.

[0534] Dynamic degradation of hardware parameters:

[0535] If the maximum resolution of the remaining slices is less than the current base resolution, such as the original value being 4K and the rest being 1080P, then updateBaseConfig() is called to reduce the base resolution to 1080P, and the hardware ISP power consumption is reduced by 40%, from 800mW to 480mW.

[0536] If the highest frame rate of the remaining slices is less than the current baseline frame rate, such as the original value being 30fps and the remaining slices being 24fps, then the baseline frame rate will be reduced to 24fps, and the sensor power consumption will be reduced by 20%, from 500mW to 400mW.

[0537] Hardware resources are fully released when all slices are released, meaning the percentage of slices with isFree=true in sliceMap is 100%.

[0538] HCS-Slicer calls ioctl(fd, VIDIOC_STREAMOFF, &type) to stop data output and calls close(fd) to close the driver handle;

[0539] Powering off the camera sensor via `echo 0 > / sys / class / camera / power / states` reduces standby power consumption from 150mW to 5mW.

[0540] Dynamic management of shared memory pools:

[0541] Once all slices are released, CS-MemPool marks the original data blocks and metadata blocks as recyclable=true;

[0542] When the system memory usage exceeds 80%, the OpenHarmony memory management service calls recycleCsMemPool() to reclaim memory, releasing 100MB of shared memory and improving system memory utilization.

[0543] Experimental verification shows that this embodiment has the following technical advantages over the prior art:

[0544] End-to-end concurrent sharing capability: By breaking through the exclusivity of physical cameras through hardware capability slicing, it supports OpenHarmony and Android applications to preview, take pictures and record videos simultaneously without modifying the driver source code; For the first time, it covers the entire process from permission verification and device discovery to resource reclamation, solving the security vulnerabilities of traditional solutions such as Android application permission bypass and functional abnormalities such as crashes caused by inconsistent capability information.

[0545] Refined conflict arbitration: Five-level rigid arbitration rules cover all conflict scenarios such as parameters, mutually exclusive resources, bandwidth, and operation mutual exclusion. The conflict handling latency is no more than 5ms, ensuring that the frame rate stability of high-priority applications such as system recording is no less than 95%, and that there are no significant fluctuations in user experience. The pending execution queue mechanism improves the utilization rate of mutually exclusive resources by 30%.

[0546] High-performance data transfer: The zero-copy mechanism of the shared memory pool ensures that the data transfer latency at 4K resolution is no more than 15ms, CPU utilization is reduced by 40%, memory bandwidth consumption is reduced by 60%, and there is no lag in application previews on both systems.

[0547] Strong compatibility and security: The dual-system protocol adaptation module is compatible with Camera Kit and Camera2 framework, allowing 99% of existing applications to support sharing without modification; the unified permission verification mechanism prevents Android applications from bypassing the OpenHarmony permission system, improving security by 30% and reducing the permission abuse rate from 12% to 0.

[0548] Dynamic energy efficiency optimization: The resource recycling and dynamic hardware adjustment mechanism reduces power consumption by 96.7% when the camera is idle, and improves battery life by more than 30%; the average utilization rate of slices increases from 60% to 85%, and the system resource scheduling efficiency is significantly optimized.

[0549] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for cross-environment camera sharing based on hardware access slices, characterized in that, Specifically, the following steps are included: When the host starts, the permission verification module receives permission requests from cross-environment applications and allocates the first shared memory pool to provide a zero-copy data channel between the application and the camera; The system initiates a conflict arbitrator to adjudicate resource access conflicts, and then initiates a capability slicing module to collect camera information and encapsulate it into a device capability structure. Finally, it initiates a host adapter to convert the device capability structure into its standard structure to provide hardware capabilities for the host application. When the container starts, the startup permission proxy module forwards permission requests to the permission verification module, the startup container adapter maps the first shared memory pool, and the device capability structure is converted into its standard structure to provide hardware capabilities for container applications; After the application starts, it directly or through the permission proxy module sends a camera permission request to the permission verification module. After the verification is successful, the permission status is recorded and the authorization result is returned to the application. The application obtains hardware capabilities through the corresponding adapter to build a session configuration and initiates a camera resource access request containing the session configuration. The request is converted into a cross-environment standard resource request by the corresponding adapter and sent to the conflict arbitrator. The conflict arbitrator calculates the request priority and adds the request to the scheduling queue accordingly. The hardware capability slicing module allocates logical slices and marks the slice configuration according to the application request. The allocation result is sent to the conflict arbitrator. After determining the hardware baseline configuration, the camera is started and the logical slice is bound to the first shared memory pool. The conflict arbitrator sends logical slices to the application through the corresponding adapter, and the application starts the business data flow according to the logical slice parameters. The camera captures images, and the hardware capability slicing module processes the raw output images according to the slicing configuration, writes them to the first shared memory pool, and notifies the application to read them. The application's control commands are sent from the first shared memory pool to the conflict arbitrator for arbitration. After the arbitration is successful, the hardware capability slicing module executes hardware control to complete the concurrent access and collaborative control of multiple applications.

2. The cross-environment camera sharing method according to claim 1, characterized in that, The first shared memory pool includes a control instruction block, a raw data block, and a metadata block. The control instruction block uses a circular buffer structure to store hardware control instructions, the raw data block stores the raw images output by the camera, and the metadata block stores auxiliary information for each frame of the image.

3. The cross-environment camera sharing method according to claim 1, characterized in that, The conflict arbitrator uses a blocking queue to cache and schedule concurrent requests from multiple runtime environments. The queue capacity supports simultaneous requests from multiple applications, and a busy message is returned when the queue is full.

4. The cross-environment camera sharing method according to claim 1, characterized in that, The application acquires hardware capabilities and constructs session configurations through corresponding adapters as follows: Data streams for preview, shooting, and video recording are built separately, distinguishing between different business usage scenarios; the running parameters of the data streams are set according to the hardware-supported resolution, frame rate, and image data format; output components for image rendering, image storage, and video encoding are bound to complete the association configuration between the data streams and the output carrier; multiple data streams for different purposes are merged to form a camera usage session configuration, including preview data stream configuration and shooting data stream configuration, specifically including screen size, frame rate, data format, output binding components, and business usage type.

5. The cross-environment camera sharing method according to claim 1, characterized in that, The method of converting the corresponding adapter into a cross-environment standard resource request is as follows: The host adapter and container adapter respectively intercept camera resource access requests initiated by applications in their respective environments, parse the built-in session configuration and business template information, distinguish the actual usage scenarios, extract the business flow type, screen resolution, running frame rate and data format, read the application identification information and determine the application's category and permission level. Standardized encapsulation is performed according to the globally unified message format to generate cross-environment standard resource request messages. The messages are uniformly labeled with the request source, application ID, application type, independent request number, as well as the screen specifications, running frame rate, data format, output binding identifier and permission level corresponding to each business flow. The operating environment is distinguished only by the source field.

6. The cross-environment camera sharing method according to claim 1, characterized in that, The hardware capability slicing module allocates logical slices and marks slice configurations based on application requests in the following way: Step 4.1: Traverse the idle logical slice library, match logical slices with the same resolution as the request, frame rate not less than the request frame rate and the smallest difference between the two, and data format consistent with the request. If the match is successful, mark the logical slice as pending allocation and execute step 4.

4. If the match fails, execute step 4.

2. Step 4.2: Filter idle logical slices with a resolution greater than the requested resolution, a frame rate not less than the requested frame rate, and a consistent data format. Prioritize selecting the logical slice with the smallest resolution difference. If a match is found, mark the slice and configure it for resolution downsampling. Then proceed to step 4.

4. If the match fails, proceed to step 4.3; Step 4.3: Match idle logical slices with the same resolution, frame rate greater than the requested frame rate, and consistent data format. If the match is successful, mark the slice configuration as frame rate limiting or frame skipping and proceed to step 4.

4. If the match fails, return a resource allocation failure message. Step 4.4: The hardware capability slicing module assigns a unique logical slice to each data stream and updates the slice occupancy status.

7. The cross-environment camera sharing method according to claim 1, characterized in that, The binding logic slice and the first shared memory pool are configured as follows: The first shared memory pool includes control instruction blocks, raw data blocks, and metadata blocks. Memory blocks of corresponding capacity are allocated according to the logical slice resolution; logical slices with resolutions greater than a threshold are allocated memory blocks of a capacity greater than the threshold, and logical slices with resolutions lower than the threshold are allocated memory blocks of a capacity less than the threshold. Region isolation is achieved through memory address offsets. Each logical slice is independently bound to a dedicated metadata block to store frame auxiliary information, which is associated with the original image through a logical slice identifier. The memory block address information corresponding to the logical slice is recorded. The host adapter maps the memory blocks corresponding to the logical slices in the first shared memory pool to the memory space of the host application; it points the buffer of the application's screen rendering component to this memory space; the hardware output data is processed by the hardware capability slicing module and written to the corresponding memory block of the first shared memory pool; the application reads the data in real time through the mapped address. The container adapter maps the first shared memory pool to the container runtime environment and determines the target memory block based on the logical slice address offset; it allocates an image receive buffer for the container application and establishes an address mapping relationship between it and the target memory block. The bound image receiving buffer is injected into the application's image receiving component, enabling hardware output data to be directly delivered to the application. After being processed by the hardware capability slicing module, the hardware output data is written to the shared target memory block, which the container application can directly access through the mapping relationship. The hardware capability slicing module writes the frame auxiliary information corresponding to each frame of image into the metadata block bound to the logical slice; The adapter associates the original image with metadata using timestamps, converts it to the standard format of its respective runtime environment, and then synchronously calls back to the application.

8. The cross-environment camera sharing method according to claim 1, characterized in that, The hardware capability slicing module processes the output raw image according to the slicing configuration, writes it to the first shared memory pool, and notifies the application to read it. Specifically, the method is as follows: The hardware capability slicing module receives raw images output by the camera driver in a loop through an independent thread. It performs preprocessing on each frame of data, including data verification, timestamp generation, and metadata extraction. Data verification checks whether the frame data length matches the baseline resolution and discards abnormal data. A globally unique timestamp is generated for each frame of data for data synchronization and association. Metadata extraction extracts auxiliary information from the frame. The hardware capability slicing module performs corresponding processing on logical slices according to their slicing configurations, enabling the distribution of single-channel hardware data to multiple logical slices. When the slice parameters are completely consistent with the hardware baseline configuration, the original image is written to the shared memory block bound to the logical slice, the data is marked as valid, and the timestamp is updated. When the slice resolution is lower than the hardware baseline configuration, the original image is scaled by the hardware image processor, converted to the slice target resolution, written to the corresponding shared memory block, and marked as valid. When the slice frame rate is lower than the frame rate of the hardware baseline configuration, a frame counting skipping strategy is used to control the output frequency, and the reserved frames are written to the shared memory block and marked as valid. Frame auxiliary information is written to the metadata block bound to the logical slice to ensure consistency with the timestamp of the original image; when the data validity flag is set, the adapter triggers a data ready callback to the application, and the application reads the screen from the shared memory block and performs rendering.

9. The cross-environment camera sharing method according to claim 1, characterized in that, The application's control commands are sent from the first shared memory pool to the conflict arbitrator for arbitration, specifically in the following manner: The first control instruction is established, which includes instruction type, parameters, source, application ID, associated slice, timestamp and verification information. Supported instructions include flash control, autofocus, zoom adjustment and exposure compensation. When the application initiates control, the adapter intercepts and encapsulates them into the first control instruction. After calculating the verification value, the first control instruction is written into the control instruction block of the first shared memory pool according to the circular buffer mechanism. The hardware capability slicing module periodically scans the control instruction blocks of the first shared memory pool, verifies the integrity of the instructions, and sends valid instructions to the conflict arbitrator. The conflict arbitrator determines whether there is resource contention in the instructions. If there is a conflict, it will adjudicate according to preset rules. If there is no conflict, it will allow the instructions to proceed directly.

10. The cross-environment camera sharing method according to claim 9, characterized in that, The conflict arbitrator determines whether there is resource contention in the instructions in the following way: Define four types of conflicts: parameter conflict, mutual exclusive resource conflict, bandwidth conflict, and operation mutual exclusive conflict. The conflict arbitrator periodically scans the system's operating status, statistically analyzes the resolution, frame rate, and data format of multiple applications, and determines whether the hardware can support them simultaneously. Maintain a mutual exclusion resource occupancy table to record resource ownership and locking status. A conflict is triggered when a new request conflicts with the current occupancy status. real time Calculate the total load on the memory bus and hardware encoder, and trigger a conflict when the total required bandwidth exceeds the safety threshold; If the system monitors the operation sequence of applications and identifies business combinations that cannot be executed concurrently, it will trigger an operation conflict. After a conflict is triggered, the conflict arbitrator makes a unified judgment on the conflict when multiple applications use the service concurrently, based on preset rules, and executes the following steps in sequence: Weights are assigned according to the type of application operation, with the weight decreasing from recording to taking pictures to previewing, prioritizing the preservation of the original parameters of the operation with the highest weight; weights are assigned according to the system type to which the application belongs, with the host application having a higher weight than the container application, prioritizing the parameter requirements of the host application; weights decrease from system-level applications to third-party tool applications to entertainment applications, prioritizing the preservation of the original parameters of the application with the highest weight, while the application with the lowest weight adjusts its own parameters for adaptation; when the total bandwidth exceeds a threshold, the frame rate of the corresponding applications is reduced in ascending order of weight; for non-shareable hardware resources, only a single application is allowed to use them exclusively, and the usage order is coordinated through a queue mechanism. After the conflict resolution is completed, a resolution result is generated, which includes the associated request identifier, the affected slice identifier, the comparison of parameters before and after adjustment, and the reason for the conflict. The resolution result is then pushed to the application through the corresponding adaptation module.

Images