A method for optimizing cross-system concurrent access to cameras based on multi-level memory channels

Abstract

This invention discloses a cross-system camera concurrent access optimization method based on multi-level video memory channels. When an application initiates a camera call, the system retrieves the hardware camera capabilities to match the corresponding device, creates a parameter negotiation session to determine the camera configuration according to application requirements, builds a two-level image channel based on the shared video memory pool, pre-allocates GPU encoding and rendering resources, generates texture samplers and rendering command chain templates, builds a camera acquisition stream, configures the timing, and binds the DMA transmission path; creates an acquisition thread to acquire the original image according to the timing and stores it in the cache to generate frame metadata, writes it to the first-level channel via DMA and wakes up the distribution and scheduling module, schedules and starts the computation shader to read the frame data and generate shadow mapping metadata, maps the memory to the second-level channel for the application to read, the host reuses the command template to fill in dynamic parameters to generate a rendering buffer, and the container rendering instructions are first converted to the host standard format before generating the buffer, and the integrated rendering command chain is submitted to the compositing pipeline to output the screen.

Description

Technical Field

[0001] This invention belongs to the field of computer software development technology, specifically relating to a method for optimizing cross-system concurrent access to cameras based on multi-level video memory channels. Background Technology

[0002] Currently, even though operating systems like OpenHarmony achieve compatibility with applications on other operating systems, such as Android, through container technology, the frame drop issue when dual-system applications concurrently access the camera has not been fundamentally resolved. The core reasons, besides existing technological limitations, also include process oversights and a lack of optimization in key areas, specifically manifested as follows:

[0003] First, the process lacks completeness, failing to cover details such as the underlying interface calls for multi-camera device enumeration, the context storage format when cameras are paused or resumed, and the coordination of intra-frame prediction algorithms for video encoding. When switching between multiple cameras, interface call delays and context loss during pause and resume can lead to dropped frames or black screens. Second, the video memory channel design is crude, only dividing functional areas without clearly defining the buffer management logic for each channel, such as page table updates for circular buffers and signal synchronization mechanisms for double buffering. This can easily lead to buffer overflows or read blockages during data transfer. Third, the double-buffered shadow mapping mechanism is incomplete, lacking metadata index optimization and page table synchronization steps for address remapping. Race conditions between application reads and GPU writes rely solely on simple flag bits, failing to form a comprehensive mechanism. The system suffers from several shortcomings: First, the synchronization loop for obtaining a response is inadequate; second, the rendering command chain synchronization is coarse-grained, failing to clearly define the instruction composition of the command chain template, the timing start point of the batch submission window, and the user notification mechanism after discarding delayed frames, resulting in low synchronization efficiency and a poor user experience; third, the recording encoding collaboration is lacking, failing to cover the specific execution process of the encoding shader, the frame interval setting for bitrate monitoring, and the priority scheduling details of storage IO, making it prone to frame drops due to encoding algorithm conflicts and insufficient IO bandwidth when two applications record simultaneously; and fourth, the exception handling is weak, with unclear error code definitions, retry strategies, and degradation mechanisms for each step, such as the number of retries for consecutive acquisition failures and alternative solutions for memory allocation failures, making the system prone to crashing rather than gracefully degrading in extreme scenarios. Summary of the Invention

[0004] In view of this, the present invention provides a cross-system camera concurrent access optimization method based on multi-level video memory channels. Through request permission control, hardware parameter negotiation, three-level video memory channel partitioning, DMA hardware acquisition and direct transmission, GPU zero-copy shadow mapping distribution, hardware format adaptation, rendering instruction synchronous scheduling, and vertical synchronous image compositing, the method achieves hardware collaborative concurrency, time synchronization, and stable closed-loop operation of dual-application camera preview and recording services in both the host and container.

[0005] This invention provides a method for optimizing cross-system concurrent camera access based on multi-level video memory channels, specifically including the following steps:

[0006] The application initiates a camera usage request, which is converted into a host request format if it is a container application; it obtains the available camera hardware capabilities and determines the corresponding camera for the application; it creates a parameter negotiation session to determine the camera configuration parameters according to the application's parameter requirements and pushes them to the host and container.

[0007] Based on the configuration parameters, a primary channel for storing the original images output by the camera and a secondary channel for storing format-converted images are established in the shared video memory pool. GPU encoding and rendering hardware resources are pre-allocated, a texture sampler is created for the application, and a rendering command chain template is pre-generated. An acquisition stream is created for the camera, the acquisition timing is configured, and the DMA transfer channel is bound.

[0008] A capture thread is created and bound to a CPU performance core, allocating a temporary capture buffer with the same data format as the primary channel. The capture thread drives the camera to acquire raw images according to the capture sequence and writes them to the temporary capture buffer to form frame metadata. This triggers the DMA engine to write the raw images to the primary channel and wakes up the distribution and scheduling module. The distribution and scheduling module activates the compute shader, reads the frame metadata of the primary channel, creates shadow mapping metadata, and establishes a hash index table of frame sequence numbers and frame metadata addresses. When the image format required by the application is consistent with that of the primary channel, physical memory is mapped to the secondary channel of the host and container applications, allowing the application to read the image. When they are inconsistent, memory address remapping is performed, and the GPU completes the format conversion within the secondary channel, allowing the application to read the image.

[0009] The host application reuses the rendering command chain template, fills in the single-frame dynamic rendering parameters to generate the rendering command buffer for the current frame; the container application's rendering instructions are converted into the host rendering command format to generate the rendering command buffer for the current frame; a rendering command chain is formed and submitted, and the compositing pipeline outputs the image.

[0010] Furthermore, a three-level channel is established within the shared video memory pool according to the configuration parameters. The three-level channel includes a rendering buffer for storing texture previews and an encoding buffer for storing images and video streams to be compressed.

[0011] Generate encoding task identifiers for the host and container recording applications, bind them to independent encoding sub-regions of the three-level channels, split and bind GPU encoding hardware cores according to the application request bitrate ratio, and configure core scheduling affinity; periodically synchronize the encoding configurations of the two applications, and update them in real time after the application adjusts the parameters;

[0012] The encoding module stores the images from the secondary channel into the tertiary channel encoding sub-region, initiates GPU hardware encoding to complete the closed-loop encoding operation, generates a video stream, and stores it in the tertiary channel stream buffer. The actual encoding bitrate is calculated every set frame and saved to the bitrate monitoring buffer. If the bitrate of a single channel exceeds a threshold, parameters are adjusted to compress the data volume. When the total bitrate exceeds the hardware upper limit threshold, the target encoding bitrate is synchronously reduced according to the bitrate ratio of the two application channels. The keyframe generation interval for the two application channels is adjusted, and a shared memory synchronization flag is used to set the keyframe generation sequence.

[0013] Create a storage read / write queue, elevate the scheduling authority for reading and writing video files, and write the encoded video stream to the device directory; when the read / write latency exceeds the threshold, the newly added video stream will be temporarily stored in the emergency encoding area of ​​the third-level channel; if the occupancy rate of the emergency encoding area is lower than the threshold, the stream will continue to be cached; after the emergency encoding area is full, discard the non-critical frame stream with the earliest storage time, register the storage read / write frame loss flag, and alert the application to the risk of busy storage resources.

[0014] Furthermore, the method of outputting the image from the compositing pipeline is as follows: the compositing pipeline obtains the refresh rate of the display controller and binds the compositing frame rate to the refresh rate; the GPU's vertical synchronization mechanism is enabled to synchronize the frame output of the compositing pipeline with the vertical synchronization signal; a rendering latency counter is set during the frame buffer direct write stage to record the total compositing time for each frame; if the compositing time for a single frame is greater than a first threshold, the compositing accuracy is reduced; if the compositing time is not greater than a second threshold, the compositing accuracy is gradually increased; if the device is connected to multiple screens, an independent compositing sub-pipeline is created for each screen, using the vertical synchronization signal of the built-in screen as the synchronization source to synchronize the vertical synchronization signals of other screens; each compositing sub-pipeline generates a compositing frame corresponding to the screen resolution and writes it to the frame buffer of each screen through the multi-screen data distribution channel to complete the synchronous output of the multi-screen image.

[0015] Furthermore, the method for determining the corresponding camera for the application is as follows: the host or container application specifies the type of camera to be used. If the selected camera types are the same, the system occupies the camera hardware. If they are different, the hardware capability identifier is read to determine whether multiple cameras are supported concurrently. If they are supported, independent acquisition links are allocated to the two types of applications respectively. Otherwise, the current interactive application takes priority and the application that initiates the request switches to the camera selected by the current interactive application and issues a device conflict warning. If the camera specified by the application does not exist, a common camera of the same type is matched as a substitute, and a device switching prompt is pushed to inform the application.

[0016] Furthermore, the method by which the parameter negotiation session is created to determine the camera configuration parameters based on the application's parameter requirements is as follows:

[0017] The application's parameter requirements include resolution, frame rate, encoding, and bitrate requirements. The resolution and frame rate are both set to the maximum values ​​among the parameter requirements. When the maximum value is greater than the hardware threshold, the maximum value supported by the hardware is used. The optimal compression standard that is compatible with the hardware is uniformly selected for encoding. The total hardware bandwidth is allocated according to the ratio of the bitrate requested by both parties for recording requests, and a fixed bandwidth is allocated according to the resolution ratio for preview requests. The total bitrate of all requests is less than the hardware peak value.

[0018] Furthermore, the method for allowing the application to read the image is as follows: the application is allowed to read the image only after the front and back buffers in the secondary channel have finished switching. After the application reads the image data in the front buffer in the secondary channel, the shared memory read flag is modified to an idle state. The compute shader monitors the read flag and buffer read / write status in real time. When the current buffer is idle and the back buffer has been mapped, a buffer switch is triggered. The address pointers of the front and back buffers in the secondary channel are atomically swapped, the back buffer is reset to an idle and available state, and a buffer ready notification is pushed synchronously to inform the application to read the latest front buffer image.

[0019] Furthermore, if the application has not finished reading the front buffer while the back buffer has been mapped, the back buffer frame data is temporarily stored in the spare area conflict buffer of the secondary channel. When the conflict buffer is full, the frame with the smallest sequence number is discarded in descending order of frame number, and the buffer conflict frame loss flag is recorded synchronously to trigger dynamic scheduling of the system. After the application releases the front buffer, the frame data in the conflict buffer is bound to the back buffer first.

[0020] Furthermore, the method by which the GPU completes the format conversion is as follows: the GPU polls and monitors conversion tasks, reads the format conversion rules after detecting a new conversion request, and calls the matching pre-compiled shader to complete the format adaptation; it records the loading latency thresholds of various shaders, and shaders with loading latency greater than the threshold are judged as a type of program and stored in a type of GPU cache; after the shader is retrieved, it binds the source buffer address, the target conversion sub-region address, the color conversion matrix, and the color gamut correction parameters; the GPU uses a pixel block parallel method to complete the image format conversion, dividing a single frame image into a set number of pixel units for synchronous batch processing to complete the conversion.

[0021] Furthermore, rendering commands that are not delivered on time within the batch submission window are identified as delayed rendering command chains, and their delay duration and application identifier are recorded. If the delay duration is not greater than the threshold, the arrived delayed command chains are stored in the delayed command chain buffer, waiting to be merged and submitted in the next batch submission window. If the delay duration is greater than the threshold, the rendering command chain of the current frame is discarded, the application is notified, and at the same time, valid data from the previous frame is read from the three-level channel rendering sub-region to generate a compensation frame. The rendering command chain of the compensation frame is included in the current batch submission. If the same application experiences consecutive excessive rendering delays, the window rendering resolution is gradually reduced or high-level rendering effects are turned off. If the delay standard still cannot be restored, the rendering frame rate is reduced. After the application's rendering timing is normal for multiple consecutive frames, the window resolution, rendering effects, and frame rate are gradually restored, returning to the initial rendering configuration.

[0022] Furthermore, when an application switches to the background, the host window management module pushes a background switching event. Upon receiving the event, the application initiates a pause request. The host responds within a set time, marks the corresponding application as paused, and stops distributing new image frames to the application. The host stops pushing secondary and tertiary channel data to the application, stores unprocessed image frames and metadata in the memory pause buffer, and arranges all frames in ascending order of their sequence numbers, retaining temporary storage time markers. The host simultaneously saves the runtime context, storing the rendering instruction template, texture parameters, and window position in the rendering context cache, and storing the encoding bitrate, keyframe position, and quantization parameters in the encoding context cache. At the same time, the host retains the camera hardware configuration and reclaims idle memory resources.

[0023] After the application switches to the foreground, the host pushes a foreground wake-up event. The application initiates a recovery request and reports the desired screen resolution and frame rate. The host prioritizes restoring the camera's hardware parameters, then restores the rendering and encoding runtime environments in sequence. It reallocates the secondary and tertiary channel memory buffers, rebuilds texture resources and rendering instruction templates, and rebinds the recording service to the GPU encoding core. The host resends cached images from the pause buffer in order of frame number, synchronously refreshes the frame timestamps, and notifies the application to read the real-time screen. If the number of frames remaining in the cache is less than the set number, the last frame before the pause is reused to generate a transition screen.

[0024] The host uses a multi-task parallel approach to synchronously restore the rendering context, encoding context, and camera hardware parameters. Multiple tasks are allocated to idle CPU cores for parallel execution, and the task order is set according to the semaphore. After the restoration is completed, a stability monitoring window with a set duration is opened. At set intervals, the buffer read latency, actual rendering frame rate, and encoding bit rate fluctuations are detected. If all indicators meet the requirements, the restoration is considered normal. If the requirements are not met, a second restoration is performed, and the window resolution is reduced synchronously.

[0025] Beneficial effects:

[0026] This invention, upon receiving a request from the host or container application to preview or record video, uniformly converts the container application's request format, sets the number of concurrent requests, and verifies the application's camera permissions and dedicated storage permissions for recording services at each level, intercepting illegal requests. It then iterates through the hardware capabilities of the entire device's cameras and generates a list of shared memory device capabilities. Based on the dual-application camera settings, it determines hardware occupancy rules, allocates independent acquisition links, adapts alternative cameras, and determines service parameters. Based on these service parameters, it divides the system into three levels of independent video memory channels, pre-allocates GPU encoding and rendering computing power, cache resources, and general rendering templates, writes the original frames to the first-level acquisition channel, and wakes up the kernel distribution module after transmission. The distribution module performs zero-copy memory address remapping on both applications; if the format matches, it directly waits for the dual-buffer switching to allow the application to read; otherwise, the GPU performs hardware format conversion before allowing reading. It generates a rendering command chain, defines a batch aggregation window, and submits rendering commands in batches. Finally, the GPU compositing pipeline binds to the screen's vertical synchronization signal to output the image, dynamically and adaptively adjusting the compositing precision to adapt to multi-screen devices and synchronize rendering, achieving stable collaborative operation of dual-application camera preview and recording services. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a cross-system camera concurrent access optimization method based on multi-level video memory channels, provided by the present invention. Detailed Implementation

[0028] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0029] This invention provides a cross-system camera concurrent access optimization method based on multi-level video memory channels. The core idea is as follows: After the host and container applications initiate camera preview or recording requests, the system uniformly converts the container application's request format, sets the number of concurrent requests, and verifies the application's camera permissions and dedicated storage permissions for recording services at each level, intercepting illegal requests; it traverses the hardware capabilities of the entire system's cameras and generates a shared memory device capability list, determines hardware occupancy rules based on the dual-application camera settings, allocates independent acquisition links, adapts alternative cameras, and determines service parameters; based on the service parameters, it divides the system into three levels of independent video memory channels and pre-allocates GPUs. Encoding rendering computing power, caching resources, and general rendering templates, the raw frames are written to the primary acquisition channel. After transmission, the kernel distribution module is awakened. The distribution module performs zero-copy memory address remapping for the two applications. If the format matches, it directly waits for the dual buffer to switch and is available for application reading. If the format does not match, the GPU performs hardware format conversion before opening for reading. A rendering command chain is generated, a batch collection window is defined, and rendering commands are submitted in batches. Finally, the GPU compositing pipeline is bound to the screen vertical synchronization signal to link the output screen, dynamically and adaptively adjust the compositing accuracy, adapt to the time-series synchronous rendering of multi-screen devices, and realize the stable collaborative operation of dual-application camera preview and recording services.

[0030] This invention provides an optimization method for cross-system concurrent camera access based on multi-level video memory channels, the processing flow of which is as follows: Figure 1 As shown, the specific steps include:

[0031] Step 1: The host or container application initiates a camera usage request. The request is uniformly parsed to distinguish between preview and recording. Requests initiated by container applications are converted to the host's standard request format according to the parameter mapping table; requests initiated by host applications do not require conversion. All camera usage requests are added to the request queue. If the current number of concurrent requests exceeds the concurrency threshold, a camera resource occupancy error is returned, and the application is notified to retry later. Camera usage permissions are verified. If the application does not have the permission, an insufficient permission message is returned, and the current application process is terminated. If the application has usage permissions and it is a recording request, the application's media storage write permission is verified. If the application lacks storage permission, the process is terminated. Preview requests do not require media storage write permission verification and pass the current verification.

[0032] The method for converting requests into the host request format based on the parameter mapping table is as follows: The built-in parameter translation unit specifically handles requests from container-side applications, and completes bidirectional conversion of two sets of system parameter fields, image formats, and encoding specifications based on the standardized parameter mapping configuration file; the configuration file is an independent editable text that supports dynamic updates during system operation without restarting the camera service; for example, the container's standard YUV image format and high-definition H.264 encoding specifications can be automatically mapped and converted into corresponding image and encoding identifiers that the host can recognize.

[0033] Furthermore, after completing the permission verification, the present invention verifies hardware compatibility, specifically by: reading the camera hardware capability configuration file and extracting the inherent hardware upper limit of the device, including the maximum supported resolution, peak hardware encoding bitrate, runnable frame rate range, and whether it supports simultaneous operation of multiple cameras, etc.; verifying whether the total bitrate of the two channels recording simultaneously after merging the two applications is less than the hardware peak bitrate, and whether the resolution and running frame rate used by both are less than the hardware single-channel upper limit. When the verification fails, the original parameters of the application that initiated the session first are retained first, and the parameters of the application that started later are reduced, while a parameter change notification is pushed to the application; when the hardware does not support concurrent operation of multiple cameras, the camera selection request of the application that started later is downgraded to be consistent with that of the application that started earlier, and a device specification downgrade warning flag is issued.

[0034] Step 2: Traverse the available cameras to obtain a device list including device number, device category, sensor model, supported resolution and supported encoding format. Obtain the hardware capabilities of each camera, including exposure range, focus mode, etc., and generate a global device capability list and store it in the public area of ​​shared memory.

[0035] Both the host and container applications can specify the type of camera they need to use. If two applications select the same camera type, the camera hardware will be used directly. If there is a conflict between the two applications' selected camera types, the hardware capability identifier will be read to determine whether multiple cameras are supported concurrently. If they are supported, independent acquisition links will be allocated to the two applications respectively. Otherwise, the currently interacting application will take priority, and the application that makes the request later will automatically switch to the camera selected by the currently interacting application and issue a device conflict warning. If the camera specified by the application does not exist, a common camera of the same type will be matched as a substitute, and a device switching prompt will be pushed to inform the application.

[0036] A unique session identifier is generated in the shared memory negotiation area. Both the host and container applications can join the current parameter negotiation session. Applications need to submit their own resolution, frame rate, encoding, and bitrate requirements within a set time. The resolution and frame rate are set to the maximum values ​​in the application's requirements. If the maximum value is greater than the hardware threshold, the maximum value supported by the hardware is used. The optimal compression standard that is compatible with the hardware is uniformly selected for encoding. The total hardware bandwidth is allocated according to the ratio of the bitrate requested by both parties for recording requests, and a fixed bandwidth is allocated according to the resolution ratio for preview requests. The total bitrate of all requests must be less than the hardware peak. If no submission is made within the timeout period, the default preview parameters are uniformly allocated.

[0037] Once the negotiation is completed, the unified parameters will be pushed synchronously to the host and container, and stored in the global parameter configuration area as a unified standard for subsequent collection and encoding.

[0038] The equipment categories include front-facing, rear-facing, ultra-wide-angle, and macro lenses.

[0039] Furthermore, if an application does not agree with the negotiated downgrade parameters, it can initiate a second negotiation within a set time. If the second negotiation still fails to reach an agreement, the negotiated parameters will be forcibly implemented.

[0040] Step 3: Based on the negotiated resolution, frame rate, and bitrate parameters, establish three independent data channels in the shared video memory pool: a primary homogeneous acquisition channel, a secondary format adaptation channel, and a tertiary rendering and encoding channel. The primary homogeneous acquisition channel stores the raw image data output by the camera; the secondary format adaptation channel stores the image data after GPU format conversion; and the tertiary rendering and encoding channel includes two partitions: a rendering buffer and an encoding buffer. The rendering buffer stores texture images for window preview, while the encoding buffer stores the image to be compressed and the encoded video stream. Simultaneously, pre-allocate GPU hardware resources for encoding and rendering: encoding computing power is split and allocated according to the bitrate ratio of the host and container applications; pre-compile and cache the encoding operation program; and allocate buffers to monitor bitrate fluctuations in real time. Create independent texture samplers for the host and container applications respectively and uniformly configure basic sampling parameters; pre-generate general rendering instruction templates and store them in the process cache; and allocate additional buffers to record the rendering time of the most recently set frame as a basis for determining the system's dynamic load scheduling.

[0041] Among them, the primary homogeneous acquisition channel adopts a ring buffer structure, and is equipped with independent read and write pointers that are synchronized using an atomic mechanism. When the buffer is full, the newly acquired image is discarded and a frame loss mark is recorded, triggering dynamic scheduling of the system. Only the acquisition service has write permissions, and the GPU can only read. The original image is stored in this channel through the hardware direct transmission channel.

[0042] The secondary format adaptation channel sets up two sets of 8 frames each with front and back double buffers. It uses read and write status markers to control the writing timing. When there is a buffer switching conflict, temporary data is stored in the 2-frame spare buffer. Only the GPU has write permission, while the upper-layer application and encoding module can only read. The data flows in from the primary channel after format conversion.

[0043] The three-level rendering and encoding channel is divided into two circular rendering buffers and one dedicated encoding buffer. Resources are allocated using frame sequence numbers and marked as idle, rendering, and encoding states. When the buffer space is insufficient, encoded frames that have been fully written to the device disk and persistently saved to the recording file are released first. The converted images are then split into two types of services for reading: rendering and encoding.

[0044] The required memory capacity for the three-level channels is calculated according to unified parameters. The total capacity of the first-level channel is equal to the volume of a single original image frame multiplied by 16 frames, with each YUV format frame occupying 1.5 times the product of its width and height. The total capacity of the second-level channel is the sum of the conversion sizes of single frames from the two applications multiplied by 8 frames. The total capacity of the third-level channel is the sum of the volumes of single frames rendered and encoded by the two applications multiplied by 3 frames. Furthermore, memory allocation prioritizes requesting high-bandwidth contiguous physical memory. If high-bandwidth memory allocation fails, ordinary memory is used instead, and the CPU-GPU write-back cache is enabled. At the same time, the number of buffered frames for the next application is reduced. After memory allocation, it is uniformly mapped to the multi-process virtual address space, and page table rules are configured according to channel permissions. All mapping records are stored in the kernel area to prevent tampering.

[0045] Furthermore, GPU encoding and rendering hardware resources are pre-allocated synchronously, and the encoding computing power is split and allocated according to the ratio of the bitrate requested by the host and container applications. The H.265 computing program is pre-compiled and stored in the GPU high-speed instruction cache, and an additional 10-frame capacity buffer is allocated to monitor bitrate fluctuations in real time.

[0046] Independent texture samplers are created for the host and container applications respectively, basic sampling parameters are configured uniformly, and a general rendering instruction template is pre-generated and stored in the process cache. Only dynamic variable parameters are filled in a single frame.

[0047] Step 4: Create acquisition streams for the cameras determined through negotiation. Each camera corresponds to one acquisition stream. Configure the acquisition timing. All acquired data adopts hardware direct transmission mode and is bound to a dedicated DMA idle transmission channel. When multiple cameras are enabled at the same time, create an independent acquisition stream for each camera, bind an independent DMA transmission channel to each camera, and increase the scheduling priority of the DMA transmission channel.

[0048] A capture thread is created, bound to a high-performance CPU core, configured with the highest real-time scheduling priority, and allocated a temporary capture buffer with a data format consistent with the primary source capture channel. The capture thread performs image capture, and the camera outputs raw image data according to the configured capture sequence, storing it in the temporary capture buffer. The images in the temporary capture buffer undergo integrity and pixel validity checks. After successful checks, the capture thread generates frame metadata including a timestamp, frame sequence number, device ID, and checksum. The capture thread initiates a DMA transfer request containing the source buffer address, the primary source capture channel write address, frame length, and frame metadata address. The GPU DMA engine starts the DMA transfer, writing the image to the current write pointer position of the primary source capture channel. After the transfer is complete, the GPU triggers a DMA completion interrupt, automatically increments the write pointer, notifies the capture thread to process the next frame, and simultaneously wakes up the kernel-mode dispatch and scheduling module.

[0049] Furthermore, the integrity check compares the actual data length with the standard frame length. If the length deviation is greater than the threshold, it is determined to be incomplete. The standard frame length is 1.5 times the product of the width and height of the negotiated resolution. The pixel validity check checks the value range of all pixels. If the proportion of abnormal pixels is greater than the threshold, it is determined to be an invalid image. The Y channel pixel value must be in the range of 0 to 255, and the U or V channel must be in the range of 16 to 240.

[0050] Step 5: The distribution and scheduling module schedules and activates the pre-cached compute shader for data distribution. The compute shader obtains the read pointer of the primary same-source acquisition channel and determines the frame sequence number and corresponding data address to be distributed. The compute shader reads the frame metadata of the primary same-source acquisition channel and creates shadow mapping metadata containing mapping status and timeout flags for both the host and container applications. It records the frame address, status, and frame sequence number, stores them in the metadata index area of ​​shared memory, establishes a hash index table of frame sequence number and frame metadata address, and calculates the checksum of the shadow mapping metadata to prevent tampering. It compares the image format required by the application with that of the primary same-source acquisition channel. If the original format of the image data is consistent with the original format, memory address remapping is performed, mapping physical memory to the respective secondary format adaptation channels of the host and container applications. Then, the shadow mapping metadata is updated to the mapping completed state, and the application is allowed to read image data only after the front and back buffers of the secondary format adaptation channels have finished switching. If the formats are inconsistent, memory address remapping is performed first, mapping physical memory to the respective secondary format adaptation channels of the host and container applications. Then, it is marked as pending format conversion. Within the secondary format adaptation channel, the GPU performs hardware format conversion. After the front and back buffers have finished switching, the application reads the image data that has been format adapted.

[0051] Furthermore, memory address remapping is accomplished by updating the page table using the GPU memory management unit. The physical page table information of the original data from the primary source acquisition channel is copied and associated with the virtual address of the application back buffer corresponding to the secondary format adaptation channel. Only address references are used, and image entity data is not migrated. At the same time, read-only access permissions are configured to restrict applications from tampering with the original image data. If an address conflict occurs during page table update, the current frame is stored in the remapping failure buffer, and the address remapping operation is retried.

[0052] The specific method for allowing the application to read image data after the front and back buffers of the secondary format adaptation channel have finished switching is as follows: After the application reads the image data of the front buffer in the secondary format adaptation channel, it modifies the shared memory read flag to the idle state; the compute shader monitors the read flag and buffer read / write status in real time. When the current buffer is idle and the back buffer has been mapped, it triggers a buffer switch, atomically swaps the address pointers of the front and back buffers in the secondary format adaptation channel, resets the back buffer to the idle and available state, and synchronously pushes a buffer ready notification to inform the application to read the latest front buffer image.

[0053] Furthermore, if the application has not finished reading the front buffer while the back buffer has been mapped, the back buffer frame data is temporarily stored in the spare area conflict buffer of the secondary format adaptation channel. When the conflict buffer is full, the frame with the smallest sequence number is discarded in descending order of frame number, and the buffer conflict frame loss flag is recorded synchronously to trigger dynamic scheduling of the system. After the application releases the front buffer, the frame data in the conflict buffer is bound to the back buffer first.

[0054] Furthermore, the method by which the GPU performs hardware format conversion is as follows:

[0055] The GPU polls and monitors conversion tasks. Upon detecting a new conversion request, it reads the format conversion rules and retrieves the matching pre-compiled shader to complete the format adaptation. It records the loading latency thresholds for various shaders. Shaders with loading latency exceeding the threshold are identified as high-frequency programs and stored in the GPU's high-frequency cache to shorten subsequent loading time. After the shader is retrieved, it binds the source buffer address, target conversion sub-region address, color conversion matrix, and color gamut correction parameters. The GPU uses a pixel-block parallel approach to complete the image format conversion, dividing a single frame image into a set number of pixel units for synchronous batch processing.

[0056] Furthermore, for 4K ultra-high resolution images, the pixel slice size is automatically enlarged to reduce scheduling frequency and GPU scheduling overhead.

[0057] Furthermore, after the single-frame format conversion is completed, an integrity check is performed on the entire frame data. If the check is successful, the conversion is marked as complete, and the application is notified to read the adapted image. If the check fails, the format conversion is automatically retried. If the retry limit is reached and the retry still fails, the current abnormal frame is discarded, and the previous normal frame is retrieved for the application to read.

[0058] Furthermore, the system provides real-time statistics on the entire process time of single-frame format conversion. When the conversion latency exceeds the standard for multiple consecutive frames, it automatically downgrades the conversion accuracy by reducing the image bit depth, disabling color gamut correction, and optimizing pixel slice size to reduce the GPU computing load. After the conversion latency falls back to the standard range, the system gradually restores the image conversion accuracy to ensure image quality. When the application locks the high-precision image quality mode, it prohibits accuracy degradation and ensures stable operation of high-precision format conversion by allocating more GPU computing cores.

[0059] Furthermore, when the application reads the buffered data of the secondary format adaptation channel, it compares the channel's built-in data format with the required image format. If the formats match, the application directly reads the buffered image data; if the formats do not match, it submits a format conversion request to the shared memory conversion request area. The conversion requests are processed according to the priority queue, with high-priority conversion requests being executed first. The queue can hold a maximum of 16 conversion requests. When the queue is full, it is determined that the conversion resources are busy, and the application is prompted to reduce the service frame rate.

[0060] For video recording scenarios, this invention achieves cross-system video encoding collaboration and GPU hardware acceleration, specifically as follows:

[0061] Dedicated encoding task identifiers are generated for both the host and container recording applications, and bound to independent encoding sub-regions of the three-level rendering encoding channel. The GPU encoding hardware cores are split and bound according to the bitrate ratio requested by the two applications, and core scheduling affinity is configured. The encoding configuration information of the two applications is synchronized using a shared memory encoding parameter area. A round of regular parameter synchronization is completed at set intervals of several frames. After the application adjusts parameters such as bitrate and encoding quality, the entire domain is updated in real time, and the encoding rules of the two applications are adapted in conjunction. Before using parameters, the encoding module performs parameter compliance verification to determine whether the bitrate and quantization parameters are within the hardware value range. Illegal parameters are automatically replaced with the system default encoding parameters.

[0062] The encoding module uses the image data processed by the secondary format adaptation channel and stores it in the dedicated encoding sub-area of ​​the tertiary channel. It then starts the GPU hardware encoding program to complete the closed-loop encoding operation, sequentially performing pixel prediction, interpolation calculation, transform compression, quantization noise reduction, and entropy encoding operations. Finally, it generates a standard video bitstream and stores it in the tertiary channel bitstream buffer, synchronously recording the bitstream size and frame type information of each frame. The actual encoding bitrate is counted every set frame, and the data is retained in the bitrate monitoring buffer. The quantization parameters are adjusted in a coordinated manner to balance the bitrate and image quality. If the bitrate of a single channel exceeds the threshold, the parameters are fine-tuned to compress the data volume. When the total bitrate of the entire device exceeds the hardware limit, the target encoding bitrate is synchronously reduced according to the bitrate ratio of the two application channels. The generation interval of key frames for the two application channels is uniformly managed, and the key frame generation sequence is coordinated by using shared memory synchronization markers.

[0063] Create a high-priority storage read / write queue to enhance the scheduling permissions for video file read / write, and adapt the read / write block size and pre-read capacity to the storage hardware sector specifications. The encoded video streams are uniformly written to the device's camera-specific directory, and the video files are named according to time and application identifier. The system monitors storage read / write latency in real time. When the read / write latency exceeds the threshold, the newly added streams are temporarily stored in the three-level channel emergency encoding cache area. The system manages the occupancy ratio of the emergency encoding area, and continuously caches streams in a low-occupancy state. When the emergency encoding area is full, the existing old non-critical frame streams are discarded, and the critical frames are retained. The storage read / write frame loss identifier is registered simultaneously, prompting the application that storage resources are busy and that there is a risk of recording stuttering.

[0064] Step 6: Pre-build a rendering command chain template. The template includes pipeline binding, sampled resource binding, viewport settings, image clipping rendering commands, and single-frame dynamic rendering parameters, including the current frame texture address, window transformation matrix, and window transparency parameters. The host application reuses the rendering command chain template, fills in the single-frame dynamic rendering parameters, marks the frame timestamp and application identifier of the captured frame, and generates the rendering command buffer for the current frame. The native rendering commands of the container application are converted into the host's standard GPU rendering command format, and the frame timestamp and application identifier are marked synchronously to generate the rendering command buffer for the current frame. The rendering command chains generated by the host and container applications are stored in the application process's command chain cache, waiting for scheduling and submission.

[0065] According to the set batch submission window, the rendering command chains in the window are collected from the command chain cache area of ​​the host and container applications, with the generation time of the first command chain as the timing benchmark. These rendering command chains are in the same batch. When the command chains of the same timestamp of the two applications have been collected and the previous batch of rendering command chains has been executed, the rendering command chains are submitted.

[0066] Furthermore, this invention merges and submits rendering commands in units of a set number of frames, merges and eliminates redundant and duplicate rendering commands, and submits them in order from first to last according to the frame timestamp, so as to reduce the frequency of GPU context switching and reduce the computational overhead of rendering scheduling.

[0067] Furthermore, rendering commands that fail to arrive on time within the batch submission window are identified as delayed rendering command chains, and their delay duration and application identifier are recorded. When the delay duration is less than or equal to the threshold, the arrived delayed command chains are stored in the delayed command chain buffer, awaiting merging and submission in the next batch submission window. When the delay duration exceeds the threshold, the rendering command chain of the current frame is discarded, the application is notified, and at the same time, valid data from the previous frame is read from the three-level channel rendering sub-region to generate a compensation frame. The rendering command chain of the compensation frame is included in the current batch submission to avoid black screen stuttering in the application window. When the same application experiences consecutive excessive rendering delays, the window rendering resolution is gradually reduced and high-level rendering effects are turned off. If the delay standard still cannot be restored, the rendering frame rate is reduced. After the application's rendering sequence returns to normal for multiple consecutive frames, the window resolution, rendering effects, and frame rate are gradually restored, returning to the initial rendering configuration.

[0068] Step 7: The compositing pipeline obtains the refresh rate of the display controller and forcibly binds the compositing frame rate to the refresh rate; enables the GPU's vertical synchronization mechanism to synchronize the frame output of the compositing pipeline with the vertical synchronization signal; sets a rendering latency counter during the frame buffer direct write stage to record the total compositing time for each frame. If the compositing time for a single frame exceeds the first threshold, the compositing precision is reduced; if the compositing time recovers to no more than the second threshold, the compositing precision is gradually restored; if the device is connected to multiple screens, an independent compositing sub-pipeline is created for each screen, using the vertical synchronization signal of the built-in screen as the synchronization source to synchronize the vertical synchronization signals of other screens; each compositing sub-pipeline generates a compositing frame corresponding to the screen resolution and writes it to the frame buffer of each screen through the multi-screen data distribution channel to complete the synchronized output of multi-screen images.

[0069] Furthermore, when an application switches to the background, the host window management module pushes a background switching event. Upon receiving the event, the application initiates a pause request. The host responds within a set time, marks the corresponding application as paused, and stops distributing new image frames to the application. The host terminates the push of secondary and tertiary channel data to the application, stores unprocessed image frames along with metadata in an independent memory pause buffer, and arranges all frames in ascending order of their sequence numbers, retaining temporary storage time markers. The host simultaneously saves the runtime context, storing the rendering instruction template, texture parameters, and window position in the rendering context cache, and storing the encoding bitrate, keyframe position, and quantization parameters in the encoding context cache. At the same time, it retains the camera exposure, focus, and white balance hardware configurations and reclaims idle memory resources such as texture cache and command cache.

[0070] After the application returns to the foreground, the host pushes a foreground wake-up event. The application then initiates a recovery request and reports the desired screen resolution and frame rate. The host prioritizes restoring the camera's hardware parameters, then restores the rendering and encoding environments in sequence, reallocates the secondary and tertiary channel memory buffers, rebuilds texture resources and rendering instruction templates, and rebinds the recording service to the GPU encoding core. The host resends cached images from the pause buffer in order of frame number, and synchronously refreshes the frame timestamps to avoid timing conflicts with the real-time captured images. After resending is complete, the application is notified to start reading the real-time images, and the timestamp of the first frame is calibrated to maintain continuity with the current captured frame. If the number of frames retained in the cache is insufficient, the last frame before the pause is reused to generate a transition image, and a text watermark is added to the center of the image, which is kept for a maximum of the set duration to fill data gaps and prevent the window from going black.

[0071] The host uses a multi-tasking parallel approach to synchronously restore the rendering context, encoding context, and camera hardware parameters. Multiple tasks are allocated to idle CPU cores for parallel execution, and the order of tasks is controlled by semaphores. After restoration, a stability monitoring window is opened for a set duration. At set intervals, buffer read latency, actual rendering frame rate, and encoding bitrate fluctuations are detected. If all indicators meet the standards, the system is considered to have restored normally. If the performance indicators do not meet expectations, a second restoration is automatically performed, and the window resolution is reduced synchronously. The system will retry a maximum of a set number of times. If the system still cannot run stably after multiple optimizations, the system will report the anomaly to the upper-layer application and retain the fault record.

[0072] Furthermore, if a corrupted context file is detected during the recovery process, the context backup file stored in the system directory is read and reloaded. If a cached image is lost, a black screen transition screen with text prompts is automatically generated until a new capture frame is output normally. If the overall recovery takes longer than the set time, the application frame rate is temporarily reduced to ensure the wake-up process is completed smoothly. The frame rate configuration will be gradually restored after the business is running stably.

[0073] Furthermore, the application-initiated camera stop command is divided into two modes: single-application stop and dual-application full-domain stop. When a single application stops, the cross-system camera management module marks the corresponding application as stopped and stops distributing new image frames to that application, ensuring that the camera service of the other application is not interfered with and continues to operate normally. When the last application initiates a stop command, or when the system triggers a full-domain forced stop command such as low power or process destruction, the management module issues a full-domain stop command to the acquisition service, encoding module, and rendering module, initiating a full-link collaborative shutdown process. After receiving the stop command, each business module completes a shutdown response within a time limit. If the response times out, a full-domain forced shutdown is executed to avoid business thread blocking and lag. After the single application is shut down, the other application is simultaneously notified of business change information, and the CPU, GPU encoding, and rendering resource quotas of the retained application are dynamically increased to exclusively enjoy hardware computing resources.

[0074] Upon receiving a stop command, the video recording module prioritizes completing the encoding of the current full video frame, writing the remaining video stream in the buffer to the storage medium, and supplementing the recording file's duration, frame rate, and bitrate header index information to ensure the video file can be parsed and read / written normally. If the storage read / write latency is too high, it prioritizes shutting down unrelated background read / write tasks, exclusively using the storage read / write channel to complete the stream writing to disk, thus closing the loop of the recording file read / write process. After the entire service chain is shut down, each service module collects the entire process data of this camera session, generates a structured operation log, and retains the session time period, service parameters, frame loss ratio, peak hardware resources, and information on abnormal events throughout the process. The log is archived and stored by natural day and retained for a fixed period of thirty days for subsequent business fault investigation and performance iteration optimization.

[0075] After the business shutdown verification is completed, the system performs a full-link memory resource reclamation operation layer by layer. This involves debinding the cross-process virtual address binding mapping relationship of each level of video memory channel, classifying and reclaiming high-speed video memory and ordinary business memory, clearing various business buffers, index areas and scheduling cache data in shared memory, and simultaneously inspecting all memory nodes to identify and warn of memory leaks. The system also unbinds dedicated GPU encoding and computing cores, cleans up encoding and rendering programs in the hardware cache, destroys the cross-system unified rendering environment, shuts down the image compositing pipeline, and reclaims screen frame cache read and write permissions. The system orderly exits acquisition, transmission, and cross-process collaborative worker threads, waits for threads to safely terminate and exit, reclaims thread stack resources, and releases kernel-level hardware resources such as hardware interrupts and DMA transfers.

[0076] After the entire resource reclamation is completed, the system resets all business status markers, session information, format adaptation ledgers, and scheduling parameters of the camera management module; restores the factory default parameters for camera hardware exposure, focus, and white balance; shuts down the hardware acquisition stream; and releases the permissions occupied by the camera hardware device. It also restores the default scheduling priority and core binding affinity scheduling rules for the entire CPU and GPU, and clears the historical load scheduling ledger. After a single application is shut down, a business completion notification is pushed, simultaneously providing feedback on business information such as preview duration and video storage path. After a dual-application full-domain shutdown, a system pop-up notification is pushed, indicating camera shutdown and file storage results. In cases of business anomalies, storage anomalies are simultaneously marked, providing users with intuitive business feedback.

[0077] Example:

[0078] This embodiment employs a cross-system camera concurrent access optimization method based on multi-level video memory channels provided by the present invention. Using the OpenHarmony operating system as the host system and the Android system as the container system, it achieves cross-system camera concurrent access through request permission control, hardware parameter negotiation, three-level video memory channel partitioning, DMA hardware acquisition and direct transmission, GPU zero-copy shadow mapping distribution, hardware format adaptation, rendering instruction synchronous scheduling, and vertical synchronous image compositing full-link collaborative management. The specific process includes:

[0079] To clearly and completely illustrate the modification process of the OpenHarmony system's underlying layer in this invention, the embodiments uniformly use the OH_ prefix to identify the enhanced interfaces after the original interface extension and modification. These interfaces rely on the HarmonyOS native Camera, GPU, memory, and DMA underlying interfaces for implementation. They add scheduling logic such as three-level video memory management, multi-application concurrency negotiation, hardware encoding computing power allocation, and zero-copy frame data distribution on top of the native calling logic, representing a functional extension and process modification of the existing system interfaces. Other business APIs without the OH_ prefix and cross-system collaborative middleware are new business components added in this invention.

[0080] S1. Cross-system camera startup request reception and permission verification.

[0081] S1.1. Add a cross-system camera collaboration manager, CrossSysCameraManager, to the OpenHarmony framework layer. Within this manager, add a request classification and processing module. This module distinguishes request types through the following process:

[0082] Parse the usage_type field in the application request parameters, where 0 indicates preview and 1 indicates recording. Preview requests are required to include resolution, frame rate (fps), and display orientation (display_orientation), which can be 0, 90, 180, or 270°. Recording requests additionally include the encoding format (codec_format), such as 0 for H.264 and 1 for H.265, the requested bitrate, and the keyframe interval (gop_size).

[0083] For requests to Android container applications, the Camera request translation module performs format conversion through a parameter mapping table. For example, Android's ImageFormat.YUV_420_888 corresponds to OpenHarmony's CAMERA_FORMAT_YUV_420_888_P010, and Android's MediaCodecInfo.CodecProfileLevel.AVCProfileHigh corresponds to OpenHarmony's AVC_PROFILE_HIGH. The mapping table is stored in / system / etc / camera / cross_sys_param_map.xml and supports dynamic updates.

[0084] Establish a FIFO request queue. If the number of requests exceeds the system's maximum concurrency (default 2, i.e., 1 for OpenHarmony and 1 for Android), return the ERR_CAMERA_BUSY error code and indicate that the application's current camera resources are occupied, and please try again later.

[0085] S1.2, Dual verification of permissions and compatibility:

[0086] Permission verification: A layered verification mechanism is adopted. First, the CAMERA permission is verified through the ohos.permission.CAMERA permission interface of OpenHarmony. The recording request additionally verifies the STORAGE permission ohos.permission.WRITE_MEDIA_STORAGE. Both applications must satisfy the corresponding permissions at the same time. If an application lacks permission, the ERR_PERMISSION_DENIED error code is returned, and the user is prompted to grant authorization through a system pop-up window.

[0087] Compatibility check:

[0088] Read the hardware capability list from the camera HAL layer capability description file / vendor / etc / camera / capability.xml, including maximum resolution such as 4096*2160, maximum encoding bitrate such as 50Mbps, supported frame rate range such as 15 to 60fps, and multi-device concurrency capability such as whether it supports simultaneous activation of front and rear cameras.

[0089] Verify the sum of parameters for both applications: When both applications are recording, the total bitrate should not exceed the maximum bitrate of the hardware; the resolution should not exceed the maximum resolution of the hardware; and the frame rate should not exceed the maximum frame rate of the hardware. If the thresholds are exceeded, the parameters of the application that starts first should be downgraded to those of the application that starts later. For example, if the application that starts first requests 1080P, 60fps, and 20Mbps, and the application that starts later requests 1080P, 60fps, and 20Mbps, and the maximum bitrate of the hardware is 35Mbps, then the bitrate of the application that starts later will be reduced to 15Mbps. At the same time, the application will be notified via API that the bitrate has been adjusted to 15Mbps to adapt to the hardware capabilities.

[0090] If the hardware does not support multiple devices concurrently, such as only supporting single camera activation, the device selection request of the application launched later will automatically be downgraded to be consistent with the application launched earlier, and a WARN_DEVICE_DOWNGRADE warning code will be returned.

[0091] S2. Camera device selection and cross-system parameter negotiation for multi-camera scene adaptation.

[0092] S2.1 Enumeration and selection of multiple camera devices.

[0093] Device enumeration process:

[0094] CrossSysCameraManager calls the OH_Camera_Device_Enumerate() interface to enumerate currently available camera devices and returns a list of devices including device ID, device type (0 for front, 1 for rear, 2 for ultra-wide, 3 for macro), sensor model, list of supported resolutions, and list of encoding formats.

[0095] For each device, detailed capabilities are obtained through the OH_Camera_Device_GetCapability() interface, such as exposure range of 100 to 10000us and focus mode such as automatic or manual. A list of device capabilities is generated and stored in the device information area of ​​shared memory for querying by both applications.

[0096] Device selection logic:

[0097] Both applications specify the target device type through the custom API setCameraDeviceType(). If the selection is consistent, such as both selecting the latter, the device is directly determined and the OH_Camera_Device_Open() interface is called to open the device.

[0098] If there is a conflict, such as selecting "front" for OpenHarmony applications and "back" for Android applications:

[0099] If the hardware supports concurrent multi-device operation, the OH_Camera_Device_Open_Multi() interface is called to open both devices simultaneously and allocate independent acquisition links for each application, based on the multi_device_support field in the capability list.

[0100] If the hardware does not support multiple devices concurrently, the application priority will be used to determine the priority. The foreground application will obtain the most recently interacted application through the window management service interface and select it first. Applications launched later will automatically be downgraded to the device type of the foreground application and return the WARN_DEVICE_CONFLICT warning code.

[0101] If the specified device type is unavailable (e.g., no ultra-wide-angle lens), an alternative device of the same type will be automatically selected. If the ultra-wide-angle lens is unavailable, the rear camera will be selected, and the application will be notified that the specified device is unavailable and that the camera has been switched to the rear camera.

[0102] S2.2 Cross-system parameter negotiation and confirmation.

[0103] The negotiation session process is as follows:

[0104] CrossSysCameraManager creates a negotiation session ID in the shared memory negotiation area. The ID format is YYYYMMDDHHMMSS plus a random number. Both applications join the session through the custom API joinNegotiationSession(sessionID). If the parameter request is not submitted within 100ms, the default parameters will be used. The default settings are 1080P, 30fps, and preview.

[0105] Negotiation Rules:

[0106] Resolution: Take the maximum value of the resolution requested by both applications. For example, if application A is 1080P and application B is 720P, then negotiate to 1080P. If the maximum value exceeds the hardware limit, then take the maximum hardware resolution.

[0107] Frame rate: Take the maximum value of the frame rate requested by the two applications. For example, if application A is 60fps and application B is 30fps, then negotiate to 60fps. If the maximum value exceeds the hardware limit, then take the maximum frame rate of the hardware.

[0108] Encoding format: Use the most efficient format supported by the hardware. If the hardware supports H.265, choose it first; otherwise, choose H.264. Dual applications need to be adapted to the same format.

[0109] Bitrate: The recording application allocates the maximum hardware bitrate according to the requested bitrate ratio. The preview application allocates the bitrate according to the resolution ratio by default. For example, the 1080P preview bitrate means the maximum hardware bitrate * 20%, and the total bitrate ≤ the maximum hardware bitrate.

[0110] The negotiation result is sent to both applications via a custom negotiationResultCallback() callback function and written to the parameter configuration area of ​​CameraService as the basis for subsequent acquisition and encoding. If the two applications disagree with the result or do not accept the resolution downgrade, they can initiate a second negotiation within 50ms. If the second negotiation still fails to reach an agreement, the negotiation result will be enforced.

[0111] S3, the core aspects of three-level memory channel design and cross-system resource pre-allocation preparation.

[0112] S3.1, refined design of three-level video memory channels.

[0113] Based on the negotiated parameters, a first-level homogeneous acquisition channel, a second-level format adaptation channel, and a third-level rendering encoding channel are constructed in the shared video memory pool. The buffer management and data flow logic of each channel are shown in the table below.

[0114] Table 1 Channel Buffer Management and Data Flow Logic Table Level 1 Channel Homogeneous acquisition channel Store raw collected data A circular buffer with a capacity of 16 frames: A read pointer is used for GPU reading, and a write pointer is used for CameraService writing, synchronized via atomic variables; the pointer increments by more than 16 frames and then wraps around; when the buffer is empty, the GPU waits; when the buffer is full, CameraService discards new frames. Camera driver to DMA and then to this channel Only the CameraService write permission bit is 0b10, and the GPU read permission bit is 0b01. When a frame is dropped due to a full buffer, the FRAME_DROP_COLLECT error code is recorded, triggering dynamic scheduling. Secondary channel Format adaptation channel Data after storage format conversion Double buffering, with 8 frames in each buffer: A ready flag is set to 1 (GPU write complete) and a read flag is set to 0 (application read complete). The GPU only writes when the read flag is 0. During switching, the pointers to the front and back buffers are swapped via atomic variables. Primary channel to GPU conversion and then to this channel GPU writes 0b10, application or encoding module reads 0b01. When a buffer switching conflict occurs, the data is temporarily stored in the conflict buffer, which has a capacity of 2 frames. Level 3 Channel Rendering encoding channel Storing rendered textures / encoded data The three-buffer system includes two frames for rendering and one frame for encoding: buffers are allocated according to frame number, with the rendering buffer Frame0 or Frame1 used cyclically, and the encoding buffer Frame2 exclusively used; a usage flag is set, where 0 indicates idle, 1 indicates rendering in progress, and 2 indicates encoding in progress. From secondary channel to application rendering, then to encoding, and back to this channel Application reads rendering 0b01, encoding module reads recording 0b01 When the buffer is full, the encoding buffer is released first, followed by the frames already written to storage.

[0115] S3.2, Three-level channel capacity calculation and memory allocation.

[0116] S3.2.1 The detailed rules for capacity calculation are as follows:

[0117] The first-level channel capacity represents the size of a single frame's raw data multiplied by 16 frames; the size of a single frame's raw data represents the negotiated resolution width * height * number of pixels in YUV_420_888_P010 format: width * height * 1.5 bytes; Example: negotiated resolution 1920*1080 (1080P), single frame size is 1920*1080*1.5, which equals 3110400 bytes, approximately 2.97MB, and the first-level channel capacity is 2.97MB * 16, approximately 47.5MB; the second-level channel capacity represents the size after conversion by application A and the size after conversion by application B multiplied by 8 frames; Example: after conversion by application A, it becomes 1080P / RGB_8888, specifically 1920*1... 080*4 equals 8294400 bytes, approximately 7.91MB. Application B is 720P / NV21, specifically 1280*720*1.5 equals 1382400 bytes, approximately 1.32MB. The secondary channel capacity represents the sum of 7.91 and 1.32*8, approximately 73.8MB. The tertiary channel capacity represents the sum of the rendering frame size of application A, the rendering frame size of application B, the encoding frame size of application A, and the encoding frame size of application B multiplied by 3 frames. Example: The rendering size is the same as the secondary channel, and the encoding frame H.265 / 1080P is approximately 0.5MB / frame. The tertiary channel capacity represents 7.91+1.32+0.5+0.5*3≈30.7MB.

[0118] S3.2.2 Memory allocation process:

[0119] The memory management subsystem calls the OH_Memory_Alloc_Contiguous() interface to request contiguous high-bandwidth memory HBM, specifying the memory type as MEM_TYPE_HBM and the priority as MEM_PRIORITY_HIGH.

[0120] If consecutive HBM requests fail and return NULL, then the request is downgraded to MEM_TYPE_NORMAL normal memory, and set to write back to the cache through the OH_Memory_Set_Cache_Policy() interface to reduce CPU-GPU data interaction latency. At the same time, dynamic scheduling is triggered to reduce the number of buffered frames for the application to start after startup to 6 frames.

[0121] After memory allocation is complete, the physical memory is mapped to the virtual address space of CameraService, application process, and Android container process through the OH_GPU_MMU_Map() interface. During mapping, page table permission bits are set according to the channel permission control table configuration, and the mapping relationship is recorded in the memory mapping table and stored in kernel mode to prevent user mode tampering.

[0122] S3.3, Pre-allocation of encoding and rendering resources.

[0123] S3.3.1, Pre-allocation of encoding resources:

[0124] The OH_GPU_Encode_Core_Alloc() interface is called to allocate GPU hardware encoding cores to both applications. The allocation ratio is the ratio of the application's requested bitrate to the total requested bitrate of both applications. For example, if application A requests 20Mbps and application B requests 15Mbps, for a total of 35Mbps, then application A will occupy approximately 57% (20 / 35), which is 1.2 cores. The Mali-G720 has a total of 2 encoding cores, and application B will occupy 43%, which is 0.8 cores.

[0125] The H.265 encoded shader is pre-compiled via the OH_GPU_Shader_Compile() interface. It includes Intra4*4 prediction, Integer Discrete Cosine Transform (DCT), and Entropy Encoding (CAVLC) instructions. After compilation, it is cached in the GPU's instruction cache with a capacity of 16MB. Its priority is higher than that of ordinary shaders, and the cache validity period is the duration of camera usage.

[0126] Allocate a bitrate monitoring buffer with a capacity of 10 frames of encoded data to the encoding module for real-time calculation of bitrate fluctuations.

[0127] S3.3.2, Pre-allocation of rendering resources:

[0128] Call the OH_GPU_Sampler_Create() interface to create a dedicated texture sampler VkSampler for dual applications. Configure the parameters as follows: filter mode VK_FILTER_LINEAR, addressing mode VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE, anisotropic filtering level 4x, and the anisotropic filtering level can be dynamically adjusted.

[0129] A pre-created rendering command chain template, VkCommandBuffer, contains basic commands: texture binding VK_CMD_BIND_TEXTURE, vertex shading VK_CMD_BIND_PIPELINE, and window clipping VK_CMD_SET_SCISSOR. The template is stored in the command chain buffer of the application process, and only dynamic parameters, such as the texture address and window coordinates of the current frame, need to be filled in for each frame, reducing the command creation overhead.

[0130] Allocate a rendering latency monitoring buffer for both applications to store the rendering time of the last 10 frames for dynamic scheduling triggering.

[0131] S4, Same-source data acquisition and GPU DMA copy-free writing to fill primary channel data.

[0132] S4.1 Initialization of multi-camera acquisition link.

[0133] S4.1.1 Link Configuration Process:

[0134] The CameraService calls the OH_Camera_Stream_Create() interface to create a capture stream based on the selected device and configures the capture timing: by setting the camera driver registers, it is configured to 16666ns, or 1 / 60s, in a 60fps scene to ensure a stable frame interval;

[0135] Configure the data output interface to DMA mode: Call the OH_Camera_Device_Set_Output_Mode() interface to set the output mode to CAMERA_OUTPUT_DMA, and specify that the DMA channel is allocated an idle channel through the OH_DMA_Channel_Alloc() interface to avoid conflicts with the DMA channels of other devices such as the display screen;

[0136] If a dual-camera device is enabled, such as both the front and rear cameras, two independent acquisition streams are created and bound to different DMA channels, such as binding the front camera to channel 0 and the rear camera to channel 1. The channel priority is set to DMA_PRIORITY_HIGH through the OH_DMA_Channel_Set_Priority() interface to avoid channel preemption.

[0137] S4.1.2, Initialization of the acquisition thread:

[0138] Create a data collection thread named CameraCollectThread, bind it to a large CPU core such as the Cortex-X4 core 0 of a Snapdragon 8 Gen3 through the OH_Thread_Set_Affinity() interface, set the scheduling policy to SCHED_FIFO, and set the priority to 1, the highest priority of the system, to avoid being preempted.

[0139] Initialize a temporary acquisition buffer with a capacity of 2 frames, stored in the CPU's L2 cache to reduce memory access latency. The buffer format is consistent with the primary channel: YUV_420_888_P010.

[0140] S4.2 Raw data acquisition and DMA transfer triggering.

[0141] S4.2.1, Data Acquisition and Verification Process:

[0142] The acquisition thread calls the OH_Camera_Stream_Start() interface to start acquisition. The camera driver outputs raw YUV data according to the configured timing sequence. The data is first written to the acquisition temporary buffer.

[0143] The verification module performs dual verification: frame integrity verification, which calculates the length of the received data and compares it with the negotiated resolution width * height * 1.5 bytes. If the deviation exceeds 1%, it is judged as incomplete. Pixel value verification, the Y channel pixel value must be in the range of 0 to 255, and the U or V channel must be in the range of 16 to 240 to avoid color abnormalities. If the percentage of pixels outside the range is greater than 0.1%, it is judged as invalid.

[0144] After successful verification, the acquisition thread generates frame metadata: timestamp system monotonic time with a precision of 1μs, frame sequence number starting from 0 and incrementing, device ID (e.g., 1 for subsequent device IDs), and CRC32 checksum calculation covering the entire frame data.

[0145] S4.2.2, DMA transfer process:

[0146] The acquisition thread calls the OH_GPU_DMA_Transfer() interface to send a DMA transfer request. The request parameters include: source address, the physical address of the acquisition temporary buffer obtained through the OH_Memory_Get_Physical_Addr() interface; target address, the current write position of the first-level channel circular buffer read through an atomic variable; data length, the size of the raw data of a single frame; and metadata address, the storage address of the frame metadata in shared memory.

[0147] The GPU DMA engine responds to requests and initiates DMA transfer: it directly reads data from the source address and writes it to the target address through hardware, without CPU involvement, and the transfer latency is controlled within 500μs.

[0148] After the transmission is completed, the GPU triggers a DMA completion interrupt. The interrupt number can be configured, such as 0x123. The interrupt handler updates the write pointer atomic variable of the first-level channel by 1. If more than 16 frames have passed, it wraps back to 0 and notifies the acquisition thread to prepare for the next frame.

[0149] S4.3, Handling of data acquisition anomalies.

[0150] S4.3.1 Handling verification failures:

[0151] If the frame integrity check fails, the acquisition thread discards the frame and calls the OH_Camera_Stream_Request_Retry() interface to send a frame retransmission request to the camera driver. It can retry a maximum of 3 times, with a retry interval of 100μs.

[0152] If pixel value verification fails, the acquisition thread marks the frame as invalid, does not trigger DMA transfer, discards it directly, and updates the acquisition exception counter to record the number of invalid frames.

[0153] S4.3.2 Backup link switching:

[0154] If three consecutive frames fail to be verified or retransmitted, the acquisition thread determines that the current link is faulty and calls the OH_Camera_Stream_Switch_Backup() interface to switch to the backup link. If the back link fails, it switches to the ultra-wide-angle link.

[0155] During the switching process, the DEVICE_SWITCHING status, including the estimated switching time (e.g., 500ms), is sent to both applications through the status notification interface. A transition frame is generated, and the watermark from the device switching process is overlaid using the valid data from the previous frame. The watermark is located in the lower right corner of the screen with 50% transparency to avoid a black screen.

[0156] After the switch is complete, notify the application DEVICE_SWITCH_COMPLETE to resume normal data acquisition.

[0157] S5, dual-buffered shadow mapping GPU multi-path distribution for primary to secondary channel data flow.

[0158] S5.1 Distribution Triggering and Shadow Mapping Metadata Construction:

[0159] S5.1.1 Distribution Trigger Mechanism:

[0160] After the GPU's DMA completion interrupt is triggered, the interrupt handler wakes up the dispatch and scheduling module running in the kernel-mode thread of the GPU driver. This module activates the pre-cached ComputeShaderCrossSysDataDistributor through the OH_GPU_Shader_Activate() interface.

[0161] After the shader starts, it first reads the read pointer of the atomic variable of the first-level channel to determine the frame sequence number and data address to be distributed.

[0162] S5.1.2 Metadata Construction and Indexing:

[0163] typedef struct {

[0164] uint64_tframe_timestamp; / / Frame timestamp, consistent with the acquisition time.

[0165] uint32_tframe_seq; / / Frame sequence number

[0166] uint32_tdevice_id; / / Camera device ID

[0167] void *original_data_ptr; / / Physical address of the original data in the primary channel, read-only reference

[0168] uint32_ttarget_channel_id; / / Target secondary channel ID, 1 for OHOS, 2 for Android

[0169] uint8_trw_status; / / Read / write status: 0 indicates idle, 1 indicates GPU write, 2 indicates application read.

[0170] uint8_tmap_status; / / Mapping status: 0 indicates no mapping, 1 indicates mapping in progress, 2 indicates mapping complete.

[0171] uint8_ttimeout_flag; / / Timeout flag: 0 indicates normal operation, 1 indicates timeout due to failure to read.

[0172] uint32_tcrc32_checksum; / / Metadata checksum to prevent tampering

[0173] ShadowMapMetadata;

[0174] The shader reads the frame metadata from the first-level channel and constructs the shadow mapping metadata, with the following structure, adding a mapping status bit and a timeout flag:

[0175] Shadow mapping metadata is created for the two applications respectively. The value of target_channel_id for application A is 1 and that for application B is 2. The metadata index area in shared memory is stored, and a hash index table from frame sequence number to metadata address is established. The hash function is frame_seq%128, which achieves O(1) lookup efficiency.

[0176] The shader calculates the CRC32 checksum of the metadata, overwrites all fields except crc32_checksum, and writes it to the crc32_checksum field to prevent the metadata from being tampered with.

[0177] S5.2, Double-buffered shadow mapping address remapping:

[0178] The address remapping process is as follows:

[0179] The Shader queries the application format requirement table for shared memory, which is populated by the request classification and processing module to confirm whether the two applications need format conversion: if the application requirement format is consistent with the primary channel format, such as both being YUV_420_888_P010, then address remapping is performed; if they are inconsistent, then it is marked as needing conversion, and subsequent format conversion is triggered.

[0180] Address remapping is achieved through page table updates of the GPU MMU: The shader calls the OH_GPU_MMU_Update_PageTable() interface to copy the physical address page table entries of the original data of the first-level channel to the virtual address page table entries of the corresponding application buffer of the second-level channel, so that the address reference does not move the data; when updating the page table, read-only permissions are set so that the application can only read and cannot modify the original data, and the page table update timestamp is recorded.

[0181] After remapping is complete, the shader updates the map_status of the shadow mapping metadata to 2 to indicate that the mapping is complete, and sets the rw_status of the secondary channel back buffer to 1 to indicate that the GPU is writing and prevents the application from reading.

[0182] If the page table update fails due to a virtual address conflict, the shader temporarily stores the frame in a remapping failure buffer of 4 frames and triggers page table conflict handling to reallocate the virtual address. The remapping can be retried up to 2 times.

[0183] S5.3, Double Buffer Switching and Conflict Handling.

[0184] S5.3.1 Buffer switching mechanism:

[0185] After the application finishes reading the data from the secondary channel pre-buffer, it sends a read completion signal through the OH_Camera_Buffer_Read_Complete() interface and sets the atomic variable of the read flag bit of the shared memory to 0;

[0186] The shader monitors the read flag and rw_status in real time through the OH_GPU_Atomic_Read() interface: when the read flag is 0, it means the application has finished reading the front buffer and rw_status is 2, it means the back buffer has been mapped, thus triggering a buffer switch; during the switch, the shader calls the OH_GPU_Buffer_Swap_Ptr() interface to atomically swap the address pointers of the front and back buffers, and resets the rw_status of the back buffer to 0 to indicate that it is idle; after the switch is completed, the shader sends a buffer ready notification to the application through the file descriptor epoll event, notifying the application to read the new front buffer data.

[0187] S5.3.2 Conflict Resolution Strategy:

[0188] If the application has not finished reading the front buffer read flag (1) and the back buffer has been mapped (rw_status 2), the shader will temporarily store the back buffer data in the conflict buffer with a capacity of 2 frames in the spare area of ​​the secondary channel. If the conflict buffer is full, i.e., both frames are temporarily stored, the oldest frame is discarded in descending order of frame number, i.e., the newest frame is kept, and the BUFFER_CONFLICT_DROP error code is recorded, triggering dynamic scheduling to reduce the application's read latency threshold.

[0189] After the application finishes reading the front buffer, the shader first maps the data in the conflict buffer to the back buffer before processing new frames, ensuring data continuity.

[0190] S6, cross-system format conversion and GPU hardware acceleration, i.e., secondary channel data adaptation.

[0191] S6.1, Format conversion trigger judgment and shader loading.

[0192] S6.1.1, Conversion Triggering Process:

[0193] When an application reads data from a secondary channel buffer, it calls the OH_Camera_Buffer_Get_Format() interface to obtain the buffer data format and compares it with its required format.

[0194] typedef struct {

[0195] uint32_trequest_id; / / Request ID, auto-incrementing

[0196] uint32_tapp_id; / / Application ID

[0197] void *src_buf_addr; / / Source buffer address, secondary channel back buffer

[0198] void *dst_buf_addr; / / Target buffer address, secondary channel translation sub-region

[0199] uint32_tsrc_format; / / Source format, e.g., 0 represents YUV_420_888_P010

[0200] uint32_tdst_format; / / Target format, e.g., 1 represents RGB_8888

[0201] uint8_tpriority; / / Request priority, 0 indicates normal, 1 indicates high.

[0202] }FormatConvertRequest;

[0203] If they match, the data is read directly; if they do not match, a conversion request is submitted through the conversion request area of ​​shared memory. The request structure is as follows: the conversion request area uses a priority queue to process high-priority requests first. The queue capacity is 16. If it is full, the error code ERR_CONVERT_QUEUE_FULL is returned, and the application is advised to reduce the frame rate.

[0204] S6.1.2 Shader Loading and Delay Monitoring:

[0205] The GPU conversion queue with QueueID 1 monitors conversion requests through the OH_GPU_Queue_Poll() interface. Upon detecting a new request, it loads the corresponding pre-compiled shader from the format conversion rule library / system / etc / camera / format_convert_rules.xml.

[0206] The rule base stores the mapping relationship between the format conversion and the shader, such as converting YUV_420_888_P010 to the corresponding shader YUV2RGB_Shader.glsl for RGB_8888, and records the shader loading delay threshold such as 50μs;

[0207] If the shader loading latency exceeds the threshold calculated using the GPU instruction loading timestamp, the shader is marked as a high-frequency shader. The next load will directly read from the GPU's high-frequency cache capacity of 8MB, reducing loading latency.

[0208] S6.2, GPU parallel format conversion and data verification.

[0209] S6.2.1 Parallel conversion execution:

[0210] After the shader is loaded, the OH_GPU_Shader_Set_Params() interface is called to bind parameters: source buffer address, destination buffer address, conversion matrix such as the YUV to RGB conversion matrix [1.164,0,1.596;1.164,-0.392,-0.813;1.164,2.017,0], and color gamut correction parameters are read from the system display configuration file / system / etc / display / color_profile.xml, with the default value being BT.709;

[0211] The shader employs a pixel block parallel processing mechanism: it divides each frame of data into 32*32 pixel blocks. The selection criteria are: the parallel capability of the GPU computing unit, such as the fact that each computing unit of the Mali-G720 can process 32 pixels at a time. Each GPU computing unit processes one block, and a single instruction can simultaneously convert 32 pixels. SIMD characteristics, the conversion efficiency is 8 to 10 times higher than that of the CPU.

[0212] If the frame resolution is large, such as 4K, the pixel block size is dynamically adjusted to 64*64 to reduce the number of blocks and reduce scheduling overhead.

[0213] S6.2.2 Data Validation and Retry:

[0214] After the conversion is complete, the shader performs a CRC32 check on the target buffer data, and the calculation range is the entire frame data to avoid partial data corruption;

[0215] If the verification passes and the verification code matches the expectation, then update the conversion completion flag atomic variable of the secondary channel conversion sub-region to 1, and send a conversion completion interrupt to the application through the OH_GPU_Interrupt_Send() interface;

[0216] If the verification fails with the error code ERR_CRC_MISMATCH, the conversion will be retried up to 2 times. During the retry, the conversion parameters will be adjusted, such as reducing the pixel block size. If the retry fails, the frame will be discarded, the error code ERR_CONVERT_FAILED will be returned, and the application will be notified that the format conversion failed. The previous frame data will be used.

[0217] S6.3, Conversion Cache and Dynamic Adaptation.

[0218] S6.3.1, High-frequency conversion cache:

[0219] For frequently used format conversion rules, such as converting YUV_420_888_P010 to NV21, if the frequency of use is greater than 5 times / minute, the corresponding shader program and conversion parameters, conversion matrix, and pixel block size are cached in the high-frequency cache area of ​​the GPU. The cache validity period is the duration of camera usage, and it is automatically cleared after expiration.

[0220] The high-frequency cache uses the Least Recently Used (LRU) eviction policy, which evicts the least recently used shader when the capacity is full.

[0221] S6.3.2 Dynamic Adaptation Strategy:

[0222] Conversion latency monitoring: A conversion time counter records the conversion time for each frame from shader startup to verification completion. If the conversion time exceeds 8ms for three consecutive frames, the conversion accuracy is automatically reduced.

[0223] Convert a 10-bit format such as YUV_420_888_P010 to an 8-bit format such as YUV_420_888;

[0224] Turn off color gamut correction and only retain basic format conversion;

[0225] The pixel block size has been adjusted from 32*32 to 64*64;

[0226] When the conversion time recovers to no more than 5ms, the conversion accuracy is gradually restored to avoid long-term low accuracy affecting image quality. If the application explicitly requires high-precision conversion via setConvertPrecision(high), then precision degradation is prohibited, and dynamic scheduling is triggered to allocate more GPU cores to the conversion queue.

[0227] S7 features cross-system video encoding collaboration and GPU hardware-accelerated video scene performance optimization.

[0228] S7.1 Coding task allocation and parameter synchronization.

[0229] S7.1.1 Task Assignment Process:

[0230] CrossSysCameraManager assigns encoding task IDs to dual-recording applications in the format of app_id, device_id, and timestamp. These IDs are then bound to the corresponding three-level channel encoding sub-areas via the OH_Encode_Task_Assign() interface. Application A is bound to encoding sub-area 0, and application B is bound to encoding sub-area 1.

[0231] Call the OH_GPU_Encode_Core_Bind() interface to bind GPU encoding cores to encoding tasks according to the bitrate ratio: the encoding core ratio represents the ratio of the application's requested bitrate to the total requested bitrate of both applications. After binding, core affinity configuration prevents other tasks from preempting and modifying the GPU driver's core scheduling table.

[0232] S7.1.2 Parameter Synchronization Mechanism:

[0233] typedef struct {

[0234] uint32_ttask_id; / / Coding task ID

[0235] uint32_tcodec_format; / / Encoding format, 0 represents H.264, 1 represents H.265

[0236] uint32_tbitrate; / / Target bitrate, in bps

[0237] uint32_tfps; / / Encoded frame rate, in fps

[0238] uint32_tgop_size; / / Keyframe interval (in frames)

[0239] uint8_trc_mode; / / Rate control mode, 0 indicates CBR, 1 indicates VBR

[0240] uint32_tintra_qp; / / Intra-frame quantization parameter 10⁻⁵¹

[0241] uint32_tinter_qp; / / Inter-frame quantization parameter 10⁻⁵¹

[0242] }EncodeParams;

[0243] The encoding parameters of the two applications are synchronized through the encoding parameter area of ​​shared memory. The parameter structure is as follows:

[0244] The parameter synchronization frequency is once every 10 frames. If the application modifies the parameters, such as adjusting the bitrate, it will update them in real time through the OH_Encode_Params_Update() interface and notify another application that the parameters have been updated and need to be synchronized and adapted.

[0245] When the encoding module reads parameters, it performs parameter validity checks. For example, if the bitrate is not greater than the maximum hardware bitrate and the QP value is in the range of 10 to 51, illegal parameters are automatically replaced with default values, such as the default QP value of 25.

[0246] S7.2, GPU hardware-accelerated encoding and bitrate coordination.

[0247] S7.2.1, Shader Execution Flow:

[0248] The encoding module reads the converted YUV data from the secondary channel and calls the OH_Encode_Data_Input() interface to write it into the tertiary channel encoding sub-region.

[0249] The steps to trigger GPU to execute encoding are as follows:

[0250] Intra-frame prediction uses Intra4*4 prediction mode to calculate the correlation between adjacent pixels and generate a prediction block; residual calculation uses the difference between the original block and the prediction block to generate a residual block; integer DCT is performed on the residual block to reduce data redundancy; quantization is performed on the transform coefficients according to the intra_qp / inter_qp parameters to reduce the amount of data; entropy coding uses CAVLC context adaptive variable-length coding to encode the quantization coefficients and generate a bitstream.

[0251] The encoded bitstream is written into the bitstream buffer of the three-level channel encoding sub-region, and the bitstream length and frame type (I / P / B frame) are recorded.

[0252] S7.2.2, Rate Coordination Mechanism:

[0253] Bitrate monitoring: The encoding module calculates the actual bitrate every 5 frames. The actual bitrate is represented by the total length of the 5-frame bitstream / 5 / fps and is stored in the bitrate monitoring buffer.

[0254] Bitrate adjustment:

[0255] If the actual bitrate is greater than the target bitrate * 1.2, the quantization parameter is increased to intra_qp / inter_qp + 2 to reduce the bitrate; if the actual bitrate is less than the target bitrate * 0.8, the quantization parameter is decreased to intra_qp / inter_qp - 2 to improve image quality; if the total actual bitrate of the two applications is greater than the maximum hardware bitrate, the target bitrate of the two applications is reduced proportionally according to their bitrate percentages. For example, if application A accounts for 60% and application B accounts for 40%, and the total bitrate exceeds 10Mbps, then application A is reduced by 6Mbps and application B is reduced by 4Mbps.

[0256] Keyframe synchronization:

[0257] Unify the keyframe interval for dual applications, such as one I-frame every 30 frames, and coordinate the generation timing through keyframe synchronization atomic variables in shared memory.

[0258] When application A reaches the keyframe interval, the atomic variable is set from 0 to 1. When application B detects that the atomic variable is 1, it synchronously generates an I-frame. After generation, the atomic variable is reset from 1 to 0 to avoid the simultaneous generation of I-frames by both applications, which would cause a sudden increase in the encoding load. The encoding time of an I-frame is 3 to 5 times that of a P-frame.

[0259] S7.3, Coordinated Encoding and Data Storage.

[0260] S7.3.1 Storage I / O queue configuration:

[0261] CrossSysCameraManager calls the OH_Storage_IO_Queue_Create() interface to create a high-priority IO queue for encoded data, sets the IO scheduling class to rt to indicate real-time scheduling, and sets the priority to 1, which is higher than ordinary file IO.

[0262] The block size of the IO queue is configured to be 4KB, which is the same as the sector size of the storage device, to reduce IO fragmentation. The pre-read size is 32KB to improve continuous IO performance.

[0263] S7.3.2, Data Writing Process:

[0264] The encoding module calls the OH_Storage_IO_Write() interface to write the bitstream data of the three-level channel encoding sub-region to a storage device such as UFS3.1. The write path is / storage / emulated / 0 / DCIM / Camera / , and the file name format is YYYYMMDD_HHMMSS_appid.mp4.

[0265] If the storage IO latency exceeds 10ms and is obtained through the OH_Storage_IO_Get_Latency() interface, the encoded data will be temporarily cached in the emergency encoding area of ​​the third-level channel with a capacity of 100MB, storing the most recent 20 frames of bitstream data.

[0266] S7.3.3 Management strategy for emergency coding areas:

[0267] Continue caching when the emergency coding area usage rate is less than 80%;

[0268] When the utilization rate is not less than 80%, a batch write is triggered to write all bitstream data to the cache at once, and the application's encoding bitrate is reduced by 20% until the IO latency is no more than 10ms.

[0269] If the emergency coding area is 100% full, the oldest non-critical frame bitstream is discarded, while the critical frames are retained to ensure that the video file can be repaired. The STORAGE_IO_DROP error code is also recorded to notify the application that storage I / O is busy and recording may be stuttering.

[0270] S8, cross-system rendering command chain synchronization and batch submission of three-level channel rendering data preparation.

[0271] S8.1 Rendering command chain assembly and template reuse.

[0272] S8.1.1, Command Chain Template Composition:

[0273] The OpenHarmony application's rendering command chain template VkCommandBuffer contains the following basic commands that remain unchanged, requiring only dynamic parameters to be updated per frame:

[0274] VK_CMD_BIND_PIPELINE: Binds the rendering pipeline, including vertex shaders and fragment shaders;

[0275] VK_CMD_BIND_DESCRIPTOR_SETS: Binding descriptor set containing texture samplers and Uniform variables;

[0276] VK_CMD_SET_VIEWPORT: Sets the viewport window coordinates, width, and height;

[0277] VK_CMD_SET_SCISSOR: Sets the clipping area to be the same size as the window.

[0278] Dynamic parameter single-frame update:

[0279] The physical address of the current frame texture address in the three-level channel rendering sub-region;

[0280] The window transformation matrix is ​​calculated based on the screen rotation angle;

[0281] The transparency parameter applies the window transparency setting, from 0 to 100%.

[0282] S8.1.2 Command Chain Assembly Process:

[0283] OpenHarmony applications reuse command chain templates through the CameraRenderContext interface of ArkUI, call the OH_Render_CmdBuf_Update_Dynamic() interface to fill dynamic parameters, generate the VkCommandBuffer of the current frame, and mark the frame timestamp and application ID consistent with the captured frame.

[0284] Android applications generate rendering command chains through the SurfaceFlinger's RenderScript interface, which are then converted into VkCommandBuffers by the command chain conversion module via the OH_Render_CmdBuf_Convert() interface.

[0285] Parse RenderScript command opcodes such as RS_OP_BIND_TEXTURE and map them to the corresponding Vulkan instruction VK_CMD_BIND_TEXTURE;

[0286] Convert command chain parameters, such as converting RenderScript texture addresses to Vulkan device addresses;

[0287] Mark the frame timestamp and application ID with the same identifier as the OpenHarmony application;

[0288] Once the command chain is generated, it is stored in the application process's command chain cache, awaiting submission.

[0289] S8.2 Command Chain Synchronization and Batch Submission Window.

[0290] S8.2.1 Synchronization Scheduling Mechanism:

[0291] The command chain synchronization scheduler is deployed and runs on the Compositor process. This scheduler collects command chains from the command chain buffer of the two applications through the OH_Render_CmdBuf_Poll() interface, sets a 2ms batch submission window, starts timing from the arrival of the first command chain, and collects all command chains with the same timestamp within 2ms.

[0292] Timing start point determination: Taking the generation timestamp of the first command chain as the starting point, if the generation timestamp of subsequent command chains differs from the starting point by no more than 2ms, they are included in the same batch; if the difference is greater than 2ms, they are included in the next batch.

[0293] Synchronization barrier settings:

[0294] Bind a VkFence synchronization object to each batch of command chains and set the trigger conditions: both application command chains with the same timestamp have been collected and the command chain count must be equal to 2; the rendering of the previous batch of command chains has been completed and the VkFence VK_SUCCESS status has been determined.

[0295] The command chain commit is triggered only when both conditions are met, thus avoiding premature commit of the single application command chain, which could lead to desynchronization between the two windows.

[0296] S8.2.2 Batch Submission Optimization:

[0297] Instead of committing the command chain every 3 frames, this reduces the scheduling overhead of the GPU command queue. Single-frame commits require frequent context switching, while batch commits can merge context switching.

[0298] Command chain compression: Duplicate instructions are merged through the OH_Render_CmdBuf_Compress() interface. For example, if texture sampling instructions from two applications have the same parameters, such as filtering mode or addressing mode, they are merged into one instruction, which only updates the texture address parameters.

[0299] Submission order: Submit batch command chains in ascending order of frame timestamps to ensure that the images are rendered in the order of capture and to avoid frame out-of-order rendering.

[0300] S8.3 Delayed command chain processing strategy.

[0301] S8.3.1 Delay Detection and Handling:

[0302] If an application's command chain fails to reach the point where the difference between the generation timestamp and the starting point is greater than 2ms within the batch submission window, the scheduler determines it as a delayed command chain and records the delay duration and the application ID.

[0303] Delay threshold processing:

[0304] Delay no more than 15ms: The arrived command chains are temporarily stored in the delayed command chain buffer with a capacity of 10 frames, waiting for the next batch window merging and submission;

[0305] If the delay is greater than 15ms: Discard the command chain of this frame, call the OH_App_Notify() interface to broadcast a notification to the application that the rendering command chain is delayed and the frame has been discarded. It is recommended to reduce the window resolution.

[0306] Frame compensation mechanism:

[0307] After discarding a frame, the scheduler reads the valid data of the previous frame from the three-level channel rendering sub-region, generates a compensation frame that is consistent with the data of the previous frame, and marks the compensation frame identifier.

[0308] Incorporate the command chain of the compensation frame into the current batch submission to avoid a black screen.

[0309] S8.3.2 Delayed Retry and Degradation:

[0310] If the same application experiences a command chain delay greater than 15ms three times in a row, dynamic scheduling will be triggered: automatically reduce the application's window resolution, such as from 1080P to 720P, and disable performance-intensive rendering features such as anisotropic filtering.

[0311] If the latency is still greater than 15ms after downgrading, further reduce the frame rate, such as from 60fps to 30fps, until the latency is no greater than 15ms.

[0312] When the application returns to normal (i.e., 10 consecutive frames without delay), the dynamic scheduling will automatically restore the resolution and frame rate.

[0313] S9, GPU unified compositing pipeline and dual-window synchronous rendering output core components.

[0314] S9.1, Optimization of the unified synthesis pipeline stage.

[0315] The cross-system dual-window composition pipeline built on the Vulkan API is refined into five hardware acceleration stages. Performance monitoring points are set for each stage to record the time consumption for dynamic scheduling, as shown in the table below.

[0316] Table 2 List of performance monitoring points for cross-system dual-window synthetic pipeline Geometric processing Window coordinate calibration and scaling Read the window position information from the window management service, including x / y coordinates and width / height; calculate the scaling matrix, specifying the resolution or the original resolution; perform matrix transformations on the window vertices. Stage duration, target no more than 1ms Scaling matrix pre-computed, updated only when the window changes per frame. Texture depth sampling Texture reading and depth value marking Texture data is read from the three-channel rendering sub-region using VkCmdBindImage; anisotropic filtering is applied, 4x / 8x, dynamically adjusted; depth values ​​are marked according to the window's Z-order, with 0 for the foreground and 1 for the background. Stage duration, target no more than 2ms Texture data preloaded into the GPU texture cache Cropping of obscured areas Overlapping region detection and invalid pixel removal Compare the coordinates of the two windows to detect overlapping areas. For example, if window A [0,0,960,1080] and window B [960,0,1920,1080] do not overlap, perform a depth test on the overlapping areas and retain pixels with smaller depth values. Remove occluded pixels that are not included in subsequent compositing. Stage duration, target no more than 1ms Overlapping region caching allows for reuse of detection results when the window position remains unchanged. Color consistency adjustment Brightness or contrast unification and color gamut correction Read standard color parameters from the system display configuration: brightness 50%, contrast 100%; adjust the dual-window colors to standard parameters using a color correction shader; ensure a unified color gamut of BT.709. Stage duration, target no more than 1ms Color parameters are pre-stored to avoid reading configuration files every frame. Framebuffer direct write Synthesized data is written to the display controller frame buffer. Call OH_Display_FrameBuffer_Get_Addr() to get the physical address of the Frame Buffer; write the composite data directly to the Frame Buffer using VkCmdCopyBufferToImage; trigger the display ready interrupt. Stage duration, target no more than 3ms Frame buffer address caching, reused when display resolution remains unchanged.

[0317] S9.2, Dual-window synchronous rendering control.

[0318] S9.2.1 Vertical synchronization and frame rate binding:

[0319] The compositing pipeline obtains the display controller's refresh rate, such as 60Hz, through the OH_Display_Get_VSync_Freq() interface, forces the compositing frame rate to be bound to the refresh rate of 60fps, and outputs one frame every 16.67ms;

[0320] Enable the GPU's vertical synchronization (VSync) mechanism. Synchronize the frame output of the compositing pipeline with the VSync signal through the OH_GPU_VSync_Enable() interface to avoid screen tearing. The VSync signal is triggered once every 16.67ms and is only written to the FrameBuffer when the signal is triggered.

[0321] S9.2.2 Rendering Delay Monitoring and Precision Adjustment:

[0322] During the framebuffer direct writing stage, a rendering delay counter is set to record the total compositing time for each frame, from geometry processing to the completion of direct writing.

[0323] If the single-frame synthesis time exceeds 10ms, which is more than 60% of 16.67ms, the synthesis accuracy will be automatically reduced.

[0324] The anisotropic filtration level was reduced from 8x to 4x;

[0325] Turn off gamut correction in color consistency adjustment;

[0326] The accuracy of occluded area cropping is reduced from pixel-level detection to 16*16 block-level detection;

[0327] When the synthesis time is restored to no more than 8ms, the synthesis accuracy is gradually restored.

[0328] S9.2.3, Multi-screen adaptation:

[0329] If the device is connected to multiple screens, such as an external monitor connected via HDMI, the OH_Display_Enumerate() interface is called to enumerate all display devices and create an independent compositing sub-pipeline for each screen. Using the VSync signal of the main screen's built-in screen as the synchronization source, the OH_Display_VSync_Sync() interface is used to ensure that the VSync signals of other screens are synchronized with the main screen with an error of no more than 1ms. Each compositing sub-pipeline generates a composite frame corresponding to the screen resolution, such as 2560*1440 for the main screen and 1920*1080 for the external screen. This frame is written to the FrameBuffer of each screen through the multi-screen data distribution channel to ensure synchronized output of the multi-screen images.

[0330] S10. Rule-based dynamic resource scheduling and load balancing are optimized in real time throughout the entire process.

[0331] S10.1, End-to-end load monitoring.

[0332] The cross-system load monitoring module runs in kernel mode with a priority of SCHED_FIFO of 2. It collects full-link metrics every 5ms and stores them in a shared memory load monitoring area ring buffer with a capacity of 100 frames through the OH_Load_Monitor_Collect() interface. The metrics are detailed in the table below.

[0333] Table 3 List of Monitoring Dimensions Collection Dimensions Collect frame loss rate, DMA transfer time, number of multi-link handovers, and number of frame retransmissions. Read the Camera Service's acquisition status register and the GPUDMA transfer timestamp. The loss rate is no more than 1%, and the time taken is no more than 500 μs. Encoding Dimensions Encoding time, bitrate fluctuation range, keyframe generation delay, and encoding core utilization Read the GPU encoding core's execution timestamp, bitrate monitoring buffer, and encoding core utilization register. The processing time should not exceed 8ms, and the fluctuation should not exceed 10%. Rendering Dimensions Command chain latency, composition time, framebuffer write latency, texture sampling time Read the delay counter of the command chain synchronization scheduler, the stage time records of the synthesis pipeline, and the sampling timestamps of the GPU texture units. Compositing time should not exceed 10ms, and the number of delays should not exceed 1 per 100 frames. Resource Dimension GPU encoding core utilization, GPU rendering core utilization, storage I / O bandwidth, and contiguous memory fragmentation. Read the GPU driver's core usage statistics, the storage I / O queue's bandwidth counter, and the memory management subsystem's fragmentation statistics. Utilization should not exceed 85%, bandwidth should not be less than 120% of demand, and fragmentation should not exceed 40%. Application Dimension Application read buffer latency, application rendering frame rate, application CPU utilization Read the application process's read completion signal timestamp, application frame rate statistics, and CPU scheduler process utilization. Read latency no greater than 2ms, frame rate no less than 90% of the negotiated value.

[0334] S10.2, Refine the dynamic scheduling rules.

[0335] Based on the deviation between the monitored indicators and the normal thresholds, preset scheduling rules are matched. These rules are stored in / system / etc / camera / dynamic_schedule_rules.xml. The rules are executed in descending order of priority, with the highest priority rule being triggered first, as follows:

[0336] S10.2.1 Multi-camera load balancing rules, with a priority of 1:

[0337] Rule 1: If the frame loss rate of a certain camera link is greater than 3% or the DMA transfer time is greater than 800μs, the link is considered to be overloaded.

[0338] Execution action: Migrate the preview format conversion task of this link to the GPU core of the idle link. If the load of the rear link is high, migrate it to the GPU rendering core of the ultra-wide-angle link. The migration is implemented through the OH_GPU_Task_Migrate() interface.

[0339] Rule 2: If the load of both dual-camera links is greater than 85%, the load is the ratio of the sum of acquisition time and processing time to the frame interval, and the system is judged to be overloaded.

[0340] Actions performed: Reduce the resolution of the application upon startup, such as from 1080P to 720P, and simultaneously reduce its frame rate, such as from 60fps to 30fps. Also, notify the application that the system load is too high and that the resolution and frame rate have been adjusted.

[0341] S10.2.2, Coding resource scheduling rules, with a priority of 2:

[0342] Rule 1: If the GPU encoding core utilization rate is greater than 90% for 3 consecutive monitoring cycles of 15ms;

[0343] Actions performed: Temporarily disable adaptive quantization to reduce computation, and set rc_mode to 0 (CBR) until the usage rate is no greater than 75%.

[0344] Rule 2: If the bitrate fluctuation range is greater than 20% for 5 consecutive monitoring periods, i.e. 25ms;

[0345] Action performed: Adjust the bitrate control mode from Dynamic Bitrate (VBR) to Constant Bitrate (CBR), and set the maximum bitrate fluctuation to 5%;

[0346] Rule 3: If the keyframe generation delay is greater than 15ms, it should normally be no greater than 8ms;

[0347] Actions performed: Increase the keyframe interval, such as from 30 frames to 45 frames, reduce the keyframe generation frequency, and increase the keyframe quantization parameter to intra_qp+3 to reduce encoding time.

[0348] S10.2.3 Storage I / O scheduling rules, priority 3:

[0349] Rule 1: If the storage I / O bandwidth is less than 100% of the application's required bandwidth, the required bandwidth means that the total bitrate of the two applications is 1.2 times, and this will continue for 3 monitoring periods.

[0350] Action to be taken: Pause non-essential background I / O tasks, such as photo thumbnail generation and application update download, and allocate an additional 30% bandwidth to encoding I / O;

[0351] Rule 2: If the storage I / O latency is less than 20ms, it should not exceed 10ms under normal circumstances, and this should be maintained for two monitoring cycles.

[0352] Actions performed: Trigger storage cache expansion, increasing the capacity of the Level 3 channel emergency coding zone from 100MB to 200MB, while simultaneously reducing the coding bitrate of both applications, such as by 15% for each.

[0353] Rule 3: If the utilization rate of the emergency coding area is greater than 90%;

[0354] Action to be performed: Trigger emergency batch write, pause the current encoding task for no more than 500μs, write all data in the emergency encoding area at once, and resume encoding after the write is complete.

[0355] S10.2.4 Rendering resource scheduling rules, priority 4:

[0356] Rule 1: If the synthesis time is greater than 12ms, it should not exceed 10ms under normal circumstances, and this should be continued for 3 monitoring cycles;

[0357] Actions performed: Reduce compositing precision, disable anisotropic filtering (set to VK_SAMPLER_ADDRESS_MODE_CLAMP_TO_EDGE), and disable color consistency adjustment;

[0358] Rule 2: If the command chain delay count is greater than 3 times per 100 frames;

[0359] Actions performed: Shorten the batch submission window from 2ms to 1ms to reduce command chain wait time, and at the same time lower the application's read buffer latency threshold from 2ms to 1ms to notify the application;

[0360] Rule 3: If the GPU rendering core utilization is greater than 90%;

[0361] Actions to be performed: Allocate additional GPU rendering cores to the rendering queue, such as increasing from 4 cores to 6 cores, while restricting the GPU usage of background applications to only 1 core.

[0362] S10.2.5 Memory resource scheduling rules, priority 5:

[0363] Rule 1: If the fragmentation level of contiguous memory exceeds 50%, it should not exceed 40% under normal circumstances;

[0364] Actions performed: Trigger lightweight memory defragmentation, defragment memory blocks related to the shared video memory pool, merge fragmented memory, defragmentation time is no more than 1ms, and data processing is not interrupted;

[0365] Rule 2: If the shared video memory pool utilization rate is greater than 90%;

[0366] Actions performed: Following the principle of prioritizing applications launched later, the buffer frame rate is reduced from 8 frames to 6 frames, and buffered data is compressed, such as YUV data being compressed to YUV_420_888, reducing memory usage by 1 / 4.

[0367] S10.3, Scheduling result feedback and adaptation.

[0368] S10.3.1 Feedback Mechanism:

[0369] After the scheduling rules are executed, the dynamic scheduling module sends a scheduling configuration update notification to the relevant modules. The notification includes: scheduling type such as encoding bitrate adjustment, adjustment parameters such as reducing from 20Mbps to 17Mbps, and effective time: immediate or effective in the next frame.

[0370] Each module registers a scheduling configuration callback function, and completes parameter adaptation within 10ms after receiving the notification.

[0371] The CameraService adjusts the capture frame rate and resolution, the encoding module updates the bitrate and quantization parameters, the rendering module adjusts the compositing precision, and the application process receives notifications via broadcast and updates its own rendering parameters, such as window resolution.

[0372] S10.3.2, Adaptation Verification:

[0373] After the adaptation is completed, each module returns an adaptation completion confirmation to the dynamic scheduling module, including the adapted parameter values;

[0374] The dynamic scheduling module verifies the adjustment effect during the next load monitoring, such as whether the bitrate has dropped to the target value and whether the synthesis time has been reduced. If the effect does not meet expectations, i.e. the deviation is greater than 10%, the next priority scheduling rule is triggered until the indicators return to the normal threshold.

[0375] S11, Camera pause and resume collaborative application adapted to background scenarios.

[0376] S11.1 Suspend request processing and resource storage.

[0377] S11.1.1 Pause Request Triggering and Response:

[0378] When an application is switched to the background, such as when a user presses the Home button and the application is switched to the background, the window management service sends the APP_BACKGROUND event. After receiving the event, the application calls the OH_Camera_Pause() interface to send a pause request to CrossSysCameraManager. The request includes the pause reason, such as 0 indicating user operation and 1 indicating system forced pause.

[0379] After receiving the request, CrossSysCameraManager returns a PAUSE_ACCEPTED response within 50ms, and marks the application status as PAUSED, prohibiting the distribution of new frame data to the application.

[0380] S11.1.2 Resource Temporary Storage Process:

[0381] Data Temporary Storage: Stop the secondary or tertiary channel data distribution of the application, and temporarily store the unprocessed frame data and metadata in the channel into the pause buffer. This buffer is an independent memory area with a capacity of 50MB, stored in ordinary memory to avoid occupying HBM. During temporary storage, the frames are arranged in ascending order of the frame number and the temporary storage timestamp is recorded.

[0382] Context saving:

[0383] The rendering context stores the application's VkCommandBuffer template, texture sampler parameter filtering mode, addressing mode, and window coordinates in an 8MB rendering context buffer.

[0384] The encoding context recording application saves the current encoding bitrate, the frame number of the next key frame at the key frame position, the quantization parameters, and the length of the encoded bitstream in the encoding context buffer with a capacity of 4MB.

[0385] Device parameters, including camera exposure value, focus position, and white balance parameters, are stored in the device parameter cache area;

[0386] Resource release: Release non-essential resources such as the application's texture cache and command chain cache, and reclaim memory through the OH_Memory_Free() interface to free up resources for the foreground application.

[0387] S11.2, Recovery Request Processing and Data Resumption.

[0388] S11.2.1 Recovery Request Triggering and Resource Recovery:

[0389] When an application switches to the foreground window management service and sends the APP_FOREGROUND event, the application calls the OH_Camera_Resume() interface to send a recovery request, which includes the resolution or frame rate to be restored.

[0390] After receiving the request, CrossSysCameraManager calls the OH_Camera_Device_Restore_Params() interface to restore the camera parameters exposure, focus, and white balance, and calls the OH_Render_Context_Restore() and OH_Encode_Context_Restore() interfaces to restore the rendering or encoding context.

[0391] Reallocate resources: Reallocate secondary or tertiary channel buffers for the application, restore to the frame number before the pause, recreate texture samplers and command chain templates, and bind GPU encoding cores if it is a recording application.

[0392] S11.2.2, Data Resumption and Synchronization:

[0393] Read the temporarily stored frame data from the pause buffer and continue transmitting it to the secondary channel in ascending order of frame number. When resuming transmission, update the frame timestamp to the current system time to avoid conflict with the timestamp of newly acquired frames.

[0394] Once the resumed transmission is complete, notify the application that the data resumed transmission is complete, and the application will begin reading new frame data.

[0395] Synchronization calibration: Align the timestamp of the first frame after recovery with the timestamp of the current acquisition frame. For example, the timestamp of the first frame after recovery represents the difference between the timestamp of the current acquisition frame and the interval of 1 frame, to ensure continuous image and avoid frame skipping.

[0396] S11.2.3, Transition Frame Processing:

[0397] If the pause buffer data is less than 3 frames, a transition frame is generated, which uses the last valid frame data before the pause and overlays it during the recovery process. The watermark position is in the center of the screen, the font size is 24, and the transparency is 70%. The transition frame continues to be displayed until the data resumes and is completed, up to a maximum of 100ms.

[0398] The generation of transition frames is achieved through the OH_Camera_Frame_Create_Transition() interface to avoid black screen during the recovery process.

[0399] S11.3, Pause and resume performance optimization.

[0400] The steps to optimize recovery latency are as follows:

[0401] Context recovery employs a parallel recovery strategy: rendering context, encoding context, and device parameters are recovered simultaneously. Three parallel recovery tasks are created through the OH_Parallel_Restore_Create() interface, corresponding to rendering, encoding, and device parameters respectively. These tasks are bound to idle CPU cores, such as the Cortex-A720 core of the Snapdragon 8 Gen3. The tasks synchronize their dependencies through semaphores. For example, the rendering / encoding context is only allowed to use the parameters after the device parameters have been recovered. This reduces the total recovery time from 50ms for serial recovery to less than 20ms.

[0402] Stability assessment after recovery:

[0403] After recovery, a stability monitoring window is started for 500ms, and key indicators are collected every 100ms: application read buffer latency target is no more than 2ms, rendering frame rate target is no less than 95% of the negotiated frame rate, and encoding bitrate fluctuation target is no more than 10%.

[0404] If all indicators meet the target within 500ms, the system is considered to have recovered stably, and the application is notified to use normally via the OH_Camera_Resume_Stable() interface.

[0405] If any indicators fail to meet the standards, such as the rendering frame rate being only 80% of the negotiated value, a secondary recovery is triggered: the context is reloaded, the application resolution is reduced (e.g., from 1080P to 720P), and monitoring is performed again, with a maximum of two recovery attempts. If the situation remains unstable, the application is notified that the recovery has failed, and it is recommended to restart the camera and record the RESUME_UNSTABLE error code.

[0406] Pause recovery exception handling:

[0407] If the context data is corrupted during the recovery process, such as rendering template verification failing, the context backup created synchronously from the backup context cache is stored in / system / cache / camera / context_backup / . The backup data is then loaded and the recovery is performed again.

[0408] If the pause buffer data is lost due to storage medium error, a default transition frame with a black background and a "Data Recovery in Progress" message will be generated. The font size is 20, and the frame will be centered. The frame will continue to be displayed until a new acquisition frame is reached, which will last for a maximum of 200ms to avoid prolonged abnormal screen display.

[0409] If the recovery time exceeds 300ms, which is beyond the user's acceptable latency, the application's frame rate will be temporarily reduced, such as from 60fps to 30fps, prioritizing the completion of the recovery and gradually restoring the frame rate after stabilization.

[0410] S12, Camera stops request processing and releases full-link resources to conclude application usage.

[0411] S12.1 Stop Request Classification and Coordinated Response.

[0412] S12.1.1, Stop requesting classified reception:

[0413] Single application stop: The application sends a stop request through the OH_Camera_Stop() interface, carrying the stop type 0 to indicate normal stop and 1 to indicate forced stop, such as when the application crashes; after receiving the request, CrossSysCameraManager marks the application's status as STOPPING, prohibits the distribution of new frames to it, and at the same time preserves the normal data flow of another application;

[0414] Dual application shutdown: When the last application sends a shutdown request, or the system triggers a forced shutdown such as low battery shutdown or user manual termination of the process, CrossSysCameraManager sends a full-link shutdown command to CameraService, encoding module, and rendering module to start a collaborative shutdown process.

[0415] S12.1.2, Confirmation of Stop Response:

[0416] After receiving the stop command, each module returns a STOP_ACK response within 50ms. If no response is received within the timeout period, and the encoding module is processing a keyframe, CrossSysCameraManager sends a forced stop signal through the OH_Force_Stop_Signal() interface to ensure that the stop process is not blocked.

[0417] When a single application stops, it notifies another application that it has stopped and that resources have been released. It also dynamically schedules up the application's resource quota, such as increasing the proportion of coding cores from 57% to 100%.

[0418] S12.2 Data storage and log recording.

[0419] After receiving the stop command, the encoding module completes the encoding of the current frame to avoid bitstream corruption, calls the OH_Encode_Flush() interface to write the remaining data in the bitstream buffer to the storage device, and generates the index information of the video file, such as duration, frame rate, and bit rate, and stores it in the file header;

[0420] If the storage I / O latency is too high, exceeding 20ms, then emergency write mode is enabled: other non-essential I / O tasks are paused, such as background file synchronization, a dedicated I / O channel is allocated for encoded data to ensure complete data writing; after writing is complete, the file is closed through the OH_Storage_File_Close() interface to avoid data loss.

[0421] Each module writes key logs from the camera usage process to the cross-system camera log file / data / log / camera / cross_sys_camera.log. The log content includes: usage duration, timestamp from start to stop, frame drop rate, total number of dropped frames or total number of captured frames, peak resource utilization, abnormal events such as device switching, recovery failure and error codes.

[0422] The logs are in JSON structured format, as shown in the example below:

[0423] {

[0424] session_id:20251105143025_12345,

[0425] start_time:2025-11-0514:30:25.123,

[0426] stop_time:2025-11-0514:35:40.567,

[0427] app_info:[

[0428] {app_id:com.ohos.camera,type:preview,resolution:1920x1080,fps:60},

[0429] {app_id:com.android.camera,type:recording,resolution:1280x720,fps:30}

[0430] ],

[0431] drop_frame_rate: 2.3%,

[0432] resource_peak:{cpu:75%,gpu_encode:82%,gpu_render:78%,memory:68%},

[0433] events:[{type:device_switch,time:2025-11-0514:32:10.789,code:SUCCESS},{type:resume,time:2025-11-0514:33:50.123,code:STABLE}]

[0434] }

[0435] Log files are archived daily and retained for 30 days for subsequent troubleshooting and performance optimization analysis.

[0436] S12.3, Release of full-link resources.

[0437] S12.3.1, Releasing video memory resources:

[0438] Remove the virtual address mapping of the three-level video memory channel in CameraService, application process, and Android container process, and mark the physical memory as waiting to be reclaimed;

[0439] Recycle contiguous HBM memory in the first, second, and third channels; if it is ordinary memory that has been downgraded, then recycle it.

[0440] Clear the metadata area, conflict buffer, and emergency coding area of ​​the shared video memory pool, release the associated memory blocks, and detect memory leaks. If the leak rate is greater than 0.1%, output the MEM_LEAK_WARN warning log.

[0441] S12.3.2, Releasing Encoding and Rendering Resources:

[0442] Encoding resources: Release the bound GPU encoding core, remove the pre-compiled encoding shader from the GPU instruction cache, and close the bitrate monitoring buffer;

[0443] Rendering resources: Destroy the unified rendering context VkDevice and VkInstance, release Vulkan resources; reclaim texture samplers and rendering command chain templates, and delete the rendering latency monitoring buffer;

[0444] Composition Pipeline: Stop the GPU unified composition pipeline, clear temporary data during the composition stage, and release write access to the FrameBuffer.

[0445] S12.3.3, Thread and Process Resource Release:

[0446] Terminate the acquisition thread and DMA transfer thread of CameraService, wait for the threads to exit safely, and reclaim the thread stack memory;

[0447] Close the CameraHAL translation thread of the Android container and the rendering coordinating thread of SurfaceFlinger to release shared memory for cross-process communication;

[0448] Terminate system service threads such as the cross-system load monitoring module command chain synchronization scheduler, and release kernel-mode resources, such as interrupt numbers and DMA channels.

[0449] S12.4, Status Reset and User Feedback.

[0450] S12.4.1, State Reset:

[0451] Reset all state variables of CrossSysCameraManager, such as negotiation session ID, application status flag, and device selection result, and clear the application format requirement table coding parameter area.

[0452] Reset camera driver state: Restore the camera's default parameters such as exposure value and focus mode, close the acquisition stream, and release the camera device handle;

[0453] Reset dynamic scheduling configuration: Restore the default scheduling policy of CPU or GPU, such as process priority and core affinity, and clear the historical data in the load monitoring area and scheduling configuration area.

[0454] S12.4.2, User Feedback:

[0455] When a single application stops, a STOP_SUCCESS notification is sent to the stopped application, including key statistics of the current usage such as preview duration and video file path.

[0456] When both applications stop, a system notification will pop up for the user, indicating that the camera is off and the video file has been saved to DCIM or Camera. If there are any abnormalities, such as data loss or encoding failure, the notification will indicate that some data has not been saved and ask the user to check the storage device.

[0457] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing cross-system concurrent access to cameras based on multi-level video memory channels, characterized in that, Specifically, the following steps are included: The application initiates a camera usage request; if it is a container application, it is converted into a host request format; it obtains the available camera hardware capabilities and determines the corresponding camera for the application. Create a parameter negotiation session to determine the camera configuration parameters based on the application's parameter requirements, and push them to the host and container; Based on the configuration parameters, a primary channel for storing the original image output by the camera and a secondary channel for storing the format-converted image are established in the shared video memory pool. GPU encoding and rendering hardware resources are pre-allocated, a texture sampler is created for the application, and a rendering command chain template is pre-generated. Create a capture stream for the camera, configure the capture timing, and bind the DMA transfer channel; Create a data acquisition thread and bind it to a CPU performance core, and allocate a temporary acquisition buffer with the same data format as the primary channel; The acquisition thread drives the camera to acquire raw images according to the acquisition sequence and writes them into the acquisition temporary buffer to form frame data. This triggers the DMA engine to write the raw images into the first-level channel and wakes up the distribution and scheduling module. The distribution and scheduling module activates the compute shader, reads the frame metadata of the first-level channel, creates shadow mapping metadata, and establishes a hash index table of frame sequence numbers and frame metadata addresses. When the image format required by the application is consistent with that of the first-level channel, physical memory is mapped to the second-level channel of the host and container application, allowing the application to read the image. When they are inconsistent, memory address remapping is performed, and the GPU completes the format conversion within the second-level channel, allowing the application to read the image. The host application reuses the rendering command chain template, fills in the dynamic rendering parameters of a single frame to generate the rendering command buffer for the current frame; the rendering instructions of the container application are converted into the host rendering command format to generate the rendering command buffer for the current frame. A rendering command chain is formed and submitted, and the compositing pipeline outputs the image.

2. The method for optimizing cross-system concurrent access to cameras according to claim 1, characterized in that, Based on the configuration parameters, a three-level channel is established in the shared video memory pool. The three-level channel includes a rendering buffer for storing texture previews and an encoding buffer for storing images and video streams to be compressed. Generate encoding task identifiers for the host and container recording applications, bind them to independent encoding sub-regions of the three-level channels, split and bind GPU encoding hardware cores according to the application request bitrate ratio, and configure core scheduling affinity; periodically synchronize the encoding configurations of the two applications, and update them in real time after the application adjusts the parameters; The encoding module stores the images from the secondary channel into the tertiary channel encoding sub-region, initiates GPU hardware encoding to complete the closed-loop encoding operation, generates a video stream, and stores it in the tertiary channel stream buffer. The actual encoding bitrate is calculated every set frame and saved to the bitrate monitoring buffer. If the bitrate of a single channel exceeds a threshold, parameters are adjusted to compress the data volume. When the total bitrate exceeds the hardware upper limit threshold, the target encoding bitrate is synchronously reduced according to the bitrate ratio of the two application channels. The keyframe generation interval for the two application channels is adjusted, and a shared memory synchronization flag is used to set the keyframe generation sequence. Create a storage read / write queue, elevate the scheduling authority for reading and writing video files, and write the encoded video stream to the device directory; when the read / write latency exceeds the threshold, the newly added video stream will be temporarily stored in the emergency encoding area of ​​the third-level channel; if the occupancy rate of the emergency encoding area is lower than the threshold, the stream will continue to be cached; after the emergency encoding area is full, discard the non-critical frame stream with the earliest storage time, register the storage read / write frame loss flag, and alert the application to the risk of busy storage resources.

3. The method for optimizing cross-system concurrent access to cameras according to claim 1, characterized in that, The method of outputting the image from the compositing pipeline is as follows: the compositing pipeline obtains the refresh rate of the display controller and binds the compositing frame rate to the refresh rate; the GPU's vertical synchronization mechanism is enabled to synchronize the frame output of the compositing pipeline with the vertical synchronization signal; a rendering latency counter is set during the frame buffer direct write stage to record the total compositing time for each frame. If the compositing time for a single frame is greater than a first threshold, the compositing precision is reduced; if the compositing time is not greater than a second threshold, the compositing precision is gradually increased; if the device is connected to multiple screens, an independent compositing sub-pipeline is created for each screen, using the vertical synchronization signal of the built-in screen as the synchronization source to synchronize the vertical synchronization signals of other screens; each compositing sub-pipeline generates a compositing frame corresponding to the screen resolution and writes it to the frame buffer of each screen through the multi-screen data distribution channel to complete the synchronous output of the multi-screen image.

4. The method for optimizing cross-system concurrent access to cameras according to claim 1, characterized in that, The method for determining the corresponding camera for an application is as follows: the host or container application specifies the type of camera to be used. If the selected camera types are the same, the system occupies the camera hardware. If they are different, the hardware capability identifier is read to determine whether multiple cameras are supported concurrently. If they are supported, independent acquisition links are allocated to the two types of applications respectively. Otherwise, the current interactive application takes priority and the application that initiates the request switches to the camera selected by the current interactive application and issues a device conflict warning. If the specified camera is not found, a generic camera of the same type will be used as a substitute, and a device switching prompt will be sent to the application.

5. The cross-system camera concurrent access optimization method according to claim 1, characterized in that, The method for creating a parameter negotiation session to determine camera configuration parameters based on the application's parameter requirements is as follows: The application's parameter requirements include resolution, frame rate, encoding, and bitrate requirements. The resolution and frame rate are both set to the maximum values ​​among the parameter requirements. When the maximum value is greater than the hardware threshold, the maximum value supported by the hardware is used. The optimal compression standard that is compatible with the hardware is uniformly selected for encoding. The total hardware bandwidth is allocated according to the ratio of the bitrate requested by both parties for recording requests, and a fixed bandwidth is allocated according to the resolution ratio for preview requests. The total bitrate of all requests is less than the hardware peak value.

6. The cross-system camera concurrent access optimization method according to claim 1, characterized in that, The method for allowing the application to read images is as follows: the application is allowed to read images only after the front and back buffers in the secondary channel have finished switching. After the application reads the image data in the front buffer in the secondary channel, the shared memory read flag is modified to an idle state. The compute shader monitors the read flag and buffer read / write status in real time. When the current buffer is idle and the back buffer has been mapped, a buffer switch is triggered. The address pointers of the front and back buffers in the secondary channel are atomically swapped, the back buffer is reset to an idle and available state, and a buffer ready notification is pushed synchronously to inform the application to read the latest front buffer image.

7. The cross-system camera concurrent access optimization method according to claim 6, characterized in that, If the application has not finished reading the front buffer while the back buffer has been mapped, the back buffer frame data will be temporarily stored in the spare area conflict buffer of the secondary channel. When the conflict buffer is full, the frame with the smallest sequence number will be discarded in descending order of the frame number. The buffer conflict frame loss flag will be recorded synchronously, triggering the system's dynamic scheduling. After the application releases the front buffer, the frame data in the conflict buffer will be bound to the back buffer first.

8. The method for optimizing concurrent access to cameras across systems according to claim 1, characterized in that, The GPU performs the format conversion as follows: The GPU polls and monitors conversion tasks. Upon detecting a new conversion request, it reads the format conversion rules and retrieves the matching pre-compiled shader to complete the format adaptation. It records the loading latency thresholds for various shaders. Shaders with loading latency greater than the threshold are classified as a type of program and stored in a GPU cache. After the shader is retrieved, it binds the source buffer address, the target conversion sub-region address, the color conversion matrix, and the color gamut correction parameters. The GPU uses a pixel-block parallel approach to complete the image format conversion, dividing a single frame image into a set number of pixel units for synchronous batch processing to complete the conversion.

9. The method for optimizing concurrent access to cameras across systems according to claim 2, characterized in that, Rendering commands that fail to arrive on time within the batch submission window are identified as delayed rendering command chains, and their delay duration and application identifier are recorded. If the delay duration is less than or equal to the threshold, the arrived delayed command chains are stored in the delayed command chain buffer, waiting to be merged and submitted in the next batch submission window. If the delay duration exceeds the threshold, the rendering command chain of the current frame is discarded, the application is notified, and at the same time, valid data from the previous frame is read from the three-level channel rendering sub-region to generate a compensation frame. The rendering command chain of the compensation frame is included in the current batch submission. If the same application experiences consecutive over-limit rendering delays, the window rendering resolution is gradually reduced or high-level rendering effects are turned off. If the latency standard still cannot be restored, the rendering frame rate is reduced. After the rendering timing of multiple consecutive frames is normal, the window resolution, rendering effects and frame rate are restored step by step to return to the initial rendering configuration.

10. The cross-system camera concurrent access optimization method according to claim 1, characterized in that, When an application switches to the background, the host window management module pushes a background switching event. After receiving the event, the application initiates a pause request. The host completes the response within a set time, marks the corresponding application as paused, and no longer distributes new image frames to the application. The host stops pushing secondary and tertiary channel data to the application, stores unprocessed image frames and metadata into a memory pause buffer, and arranges all frames in ascending order of sequence number and retains temporary storage time markers. The host synchronously saves the runtime context, stores the rendering instruction template, texture parameters and window position in the rendering context cache, and stores the encoding bitrate, keyframe position and quantization parameters in the encoding context cache. At the same time, it retains the camera hardware configuration and reclaims idle memory resources. After the application switches to the foreground, the host pushes a foreground wake-up event, and the application initiates a recovery request and reports the desired screen resolution and frame rate; The host prioritizes restoring the camera's hardware parameters, then restores the rendering and encoding environments in sequence, reallocates the secondary and tertiary channel memory buffers, rebuilds texture resources and rendering instruction templates, and rebinds the recording service to the GPU encoding core. The host resends cached images from the pause buffer in order of frame number, synchronously refreshes the frame timestamp, and notifies the application to read the real-time screen; if the number of images remaining in the cache is less than the set number of frames, the last frame before the pause is reused to generate a transition screen. The host uses a multi-task parallel approach to synchronously restore the rendering context, encoding context, and camera hardware parameters. Multiple tasks are allocated to idle CPU cores for parallel execution, and the task order is set according to the semaphore. After the restoration is completed, a stability monitoring window with a set duration is opened. At set intervals, the buffer read latency, actual rendering frame rate, and encoding bit rate fluctuations are detected. If all indicators meet the requirements, the restoration is considered normal. If the requirements are not met, a second restoration is performed, and the window resolution is reduced synchronously.

Images