Cross-platform yuv hardware encoding photographing method and system, intelligent terminal and medium

Abstract

The application discloses a cross-platform YUV hardware coding photographing method and system, an intelligent terminal and a medium, relates to the technical field of mobile terminal image processing, and comprises the following steps: when detecting a user operation instruction to trigger photographing, acquiring original YUV image data generated by a camera hardware abstraction layer in an Android camera application layer; through a pre-established IPC channel, YUV image data is sent to a camera hardware abstraction layer service process by a file descriptor memfd shared memory combined with a file descriptor transmission mechanism; the camera hardware abstraction layer service process calls a platform hardware JPEG encoder through a platform abstraction layer set in the camera hardware abstraction layer service process; the YUV image data is encoded and converted to generate JPEG image data, and the JPEG image data is returned to the application layer through the memfd shared memory. The application has the advantages of reducing inter-process data transmission overhead, improving hardware coding efficiency, reducing photographing delay, and realizing cross-platform compatibility.

Description

Technical Field

[0001] This invention relates to the field of mobile terminal image processing technology, and in particular to a cross-platform YUV hardware encoding photography method, system, smart terminal, and storage medium. Background Technology

[0002] With the continuous evolution of mobile terminal technology, users' performance requirements for camera functions have significantly increased. In the current Android platform's camera shooting process, raw YUV image data needs to be converted into JPEG format after algorithmic processing. Traditional solutions suffer from multiple technical bottlenecks: when the application layer uses software encoding libraries for CPU processing, the encoding efficiency is low in high-resolution image scenarios, resulting in sluggish user operation responses; while hardware JPEG encoders have performance advantages, they are difficult for the application layer to directly call due to the Android system's process isolation mechanism. The camera application and hardware abstraction layer services belong to independent process spaces, and the hardware encoder can only run within the service process, preventing the application process from accessing underlying hardware resources across process boundaries.

[0003] Meanwhile, system security policies strictly restrict inter-process communication, and establishing a reliable data channel requires overcoming SELinux access control constraints. Hardware encoders on different chip platforms differ fundamentally in interface specifications, buffer management, and rotation processing mechanisms. For example, some platforms rely on the mobile display processor to complete image rotation before encoding, while others achieve rotation through the encoder's native parameters. This fragmentation makes it difficult to achieve cross-platform reuse through unified interface encapsulation. Furthermore, high-resolution YUV image data is massive, and using conventional copying methods for inter-process transmission incurs significant overhead, severely diminishing the performance gains of hardware encoding. In the existing workflow, image algorithm processing and encoding conversion are executed serially on the main thread. When applying algorithms such as beauty filters, the processing time accumulates, causing blocking during photo taking, limiting continuous shooting experience, and fundamentally restricting algorithm scalability. These issues collectively prevent hardware encoding performance from being fully utilized, leaving users facing the dual dilemma of operational latency and functional limitations.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The technical problem this invention aims to solve is to address the shortcomings of the prior art by providing a cross-platform YUV hardware encoding photography method, system, smart terminal, and storage medium. This invention provides a cross-platform photography method and system that achieves efficient YUV-to-JPEG conversion on the Android platform by calling the HAL layer hardware encoder through the application layer. This invention can shield underlying hardware differences through a cross-platform abstraction layer, enabling the construction of an efficient and portable hardware encoding channel. It reduces inter-process data transmission overhead, improves hardware encoding efficiency, reduces photography latency, and achieves cross-platform compatibility.

[0006] The technical solution adopted by this invention to solve the problem is as follows: A cross-platform YUV hardware encoding imaging method, comprising: In the Android camera application layer, when a user operation command is detected to trigger a photo capture, the raw YUV image data generated by the camera hardware abstraction layer is obtained; The application layer uses a pre-established IPC channel between the abstract namespace Unix domain socket and the camera hardware abstraction layer service process, and combines file descriptor memfd shared memory with file descriptor passing mechanism to send the YUV image data to the camera hardware abstraction layer service process with zero copy. The camera hardware abstraction layer service process calls the platform hardware JPEG encoder through its internally configured platform abstraction layer. The platform hardware JPEG encoder encodes and converts the YUV image data to generate JPEG image data, and then sends the JPEG image data back to the application layer via memfd shared memory.

[0007] The cross-platform YUV hardware encoding photography method, wherein, prior to the step of obtaining the raw YUV image data generated by the camera HAL layer when a user operation command is detected to trigger photography at the Android camera application layer, includes: An inter-process communication channel for IPC is pre-established between the Android camera application layer and the camera hardware abstraction layer service process through Unix domain sockets in the abstract namespace. This channel is used to achieve zero-copy cross-process transmission of YUV image data by using memfd file descriptor shared memory combined with SCM_RIGHTS file descriptor passing mechanism. In addition, a platform abstraction layer is set up in advance within the camera hardware abstraction layer service process. The platform abstraction layer defines a unified buffer management and encoding operation interface. Hardware JPEG encoders of different chip platforms complete platform adaptation by implementing the unified interface. The platform abstraction layer abstracts the operations of the hardware JPEG encoder into the following function interfaces: buffer allocation, buffer release, buffer attribute query, cache synchronization, format conversion, encoding capability query, and encoding execution.

[0008] The cross-platform YUV hardware encoding photography method further includes, before the step of sending the YUV image data to the camera hardware abstraction layer service process with zero copy, the step of performing image processing algorithm operations on the YUV image data at the application layer, wherein the image processing algorithm includes at least one of beauty filter and image enhancement. The image processing algorithm is executed asynchronously in the background, and the camera application layer continuously responds to the user's shooting operation during the algorithm processing. The algorithm processing and shooting process are executed in parallel. The IPC channel uses Unix domain sockets with abstract namespaces and transmits image data between the application layer and the camera hardware abstraction layer service process through the memfd shared memory mechanism.

[0009] The cross-platform YUV hardware encoding photography method, wherein the camera hardware abstraction layer service process caches the allocated hardware encoding buffer through the Linux kernel memory management component (ION) buffer pool. Before the first encoding, the Linux kernel memory management component (ION) buffer is pre-allocated and cached through a warm-up protocol, and subsequent encoding requests directly reuse the cached buffer.

[0010] The cross-platform YUV hardware encoding imaging method further includes the following steps: When rotation is required, the MTK platform uses a two-stage encoding method: first, the YUV data is rotated by the mobile display processor (MDP) hardware, and then encoded by the hardware JPEG encoder; the Unisoc and Qualcomm platforms complete the rotation encoding in one step using the rotation parameters natively supported by the hardware JPEG encoder. The encoding monitoring module is loaded into the camera HAL service process as a shared library. The monitoring thread is automatically started through the shared library constructor, without requiring modification of the camera HAL service source code.

[0011] The cross-platform YUV hardware encoding and photography method described herein includes an IPC channel that supports multiple data transfer modes: transferring YUV byte array data via shared memory through the memfd file descriptor; directly transferring the dma-buf file descriptor of the camera Image via the SCM_RIGHTS file descriptor; and pre-allocating the Linux kernel memory management component (ION) buffer pool in a warm-up mode without performing actual encoding.

[0012] The cross-platform YUV hardware encoding imaging method further includes the following steps: A time-slot-based circular buffer management structure is constructed to divide the buffer pool into a fixed number of consecutive time slots. Each time slot is pre-bound to a complete ION buffer (including a source YUV buffer, a target JPEG buffer, and a platform-specific rotation buffer), forming a logical circular queue. Each time slot independently manages the occupancy status of its buffer (idle, filling, encoding, and transmission). Continuously captured image frames occupy different time slots in chronological order, physically isolating the buffer resources of adjacent frames and eliminating resource conflicts when multiple frames are concurrent. A high-water mark and low-water mark monitoring mechanism is set up in the circular buffer; the encoding request density per unit time is monitored in real time. When a high-frequency continuous shooting mode is detected and the number of idle time slots is lower than the low-water mark, the extension protocol is automatically triggered to request an additional buffer group from the system ION buffer memory and inject it into the preparation queue; when the shooting frequency decreases and the idle time slots exceed the high-water mark, the excess buffer is gradually released to the low-water mark level to dynamically balance memory usage and concurrent processing capacity. Establish buffer inheritance relationships between adjacent shooting frames to achieve zero-release reuse; after the previous frame completes hardware encoding, the ION buffer group it occupies is not immediately recycled to the general pool or released, but directly enters a hot standby state, and the memory mapping is kept valid through the reference counting mechanism, and is preferentially allocated to the next frame encoding request; the real release and reallocation are only triggered when the buffer size is mismatched or the shooting interval exceeds the threshold. A hardware encoder is introduced to complete the signal-driven buffer state machine. When the hardware encoder completes JPEG compression and triggers an interrupt, the buffer state of the corresponding time slot is immediately transferred from the encoder to a reusable state, so that it can be immediately occupied by subsequent frames. The operation of JPEG data being sent back to the application layer uses an independent temporary mapping buffer or directly establishes a read-only mapping in the original buffer for asynchronous transmission. By using a time-slotted ring structure to isolate multi-frame resources in the spatial dimension, a dual-water level mechanism to dynamically adapt to traffic fluctuations in the temporal dimension, an inter-frame inheritance protocol to eliminate unnecessary memory allocation overhead, and asynchronous state transition to compress resource occupation cycle, the four elements work together to allocate ready buffer resources for each frame encoding request with constant low latency in continuous high-speed shooting scenarios.

[0013] A cross-platform YUV hardware encoding and imaging system, wherein the system includes: The application layer encoding interface module is used to provide a Java interface for hardware JPEG encoding in the Android camera application layer. It loads the local encoding library through JNI and establishes an IPC connection with the camera hardware abstraction layer service process. The IPC communication module is used to transfer image data between the application layer and the camera hardware abstraction layer service process via Unix domain sockets and shared memory. The platform abstraction layer module is set in the camera hardware abstraction layer service process. It is used to define a unified hardware encoding operation interface and adapt to the hardware JPEG encoder implementation of different chip platforms. The hardware encoding execution module is used to call the platform's hardware JPEG encoder to complete the YUV to JPEG encoding conversion; The algorithm pipeline management module is used to manage image processing algorithm nodes at the application layer, supporting asynchronous execution and overlay processing of algorithms. The algorithm pipeline management module works with the application layer encoding interface module to trigger hardware encoding after the algorithm processing is completed. The cross-platform YUV hardware encoding photography system uses any of the above-described cross-platform YUV hardware encoding photography methods to perform photography processing.

[0014] A smart terminal includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs comprising the method for performing any of the above.

[0015] A computer-readable storage medium wherein, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in any of the preceding descriptions.

[0016] The beneficial effects of this invention are as follows: This invention provides a cross-platform YUV hardware encoding and photography method, system, smart terminal, and storage medium; by establishing a zero-copy IPC channel between the application layer and the hardware abstraction layer service process to directly transmit YUV data to the hardware encoder, it avoids data copying overhead, achieves efficient cross-platform encoding, reduces inter-process data transmission overhead, improves hardware encoding efficiency, reduces photography latency, and achieves cross-platform compatibility.

[0017] This invention establishes a high-efficiency IPC channel between the application layer and the camera HAL service process, utilizes the platform's hardware JPEG encoder to complete the YUV-to-JPEG encoding conversion, employs a modular platform abstraction layer design to achieve cross-platform reuse, and uses an asynchronous algorithm pipeline to ensure that algorithm processing does not block the image capture process. Furthermore, this invention also has the following advantages: 1) Significantly improved encoding performance. By directly calling the platform's hardware JPEG encoder, the performance bottleneck of CPU software encoding is avoided. Real-world test data shows that on the MTK6789 platform, the entire image capture process is reduced from approximately 950 milliseconds for software encoding to approximately 600 milliseconds for hardware encoding, an improvement of about 37%; on the Unisoc T615 platform, the time is significantly reduced from 3.5-4 seconds for software encoding to 600-800 milliseconds for hardware encoding, an improvement of over 75%, demonstrating extremely significant performance improvements.

[0018] 2) Strong cross-platform portability. This invention, through a modular platform abstraction layer design, completely decouples the application layer code, JNI communication layer, IPC protocol, etc., from the platform. When porting to a new platform, only 12 standard function interfaces need to be implemented in the platform abstraction layer, without modifying any application layer code. It has been successfully verified and implemented on MTK and Unisoc platforms, and Qualcomm platforms can be accessed using the same platform adaptation method.

[0019] 3) Asynchronous Algorithm Processing Enhances the Photography Experience. This invention changes image algorithm processing to be executed asynchronously in the background. The camera application can continuously respond to the user's shooting operation while processing algorithms in the background. Algorithm processing no longer blocks the shooting process, allowing more image processing algorithms (such as beautification, filters, watermarks, etc.) to be layered without affecting the shooting speed, resulting in better photo quality.

[0020] 4) Zero-code intrusion integration. The HAL layer monitoring module of this invention is loaded into the camera HAL service process as a shared library. The monitoring thread is automatically started through the shared library constructor. During integration, only one line of shared library dependency needs to be added to the compilation script. There is no need to modify the source code of the camera HAL service or the init configuration file.

[0021] 5) Intelligent buffer pool management. Through the ION buffer pool warm-up mechanism, the ION buffer required for hardware encoding is pre-allocated and cached before the first encoding. Subsequent encoding requests directly reuse the cached buffer, avoiding the overhead of reallocating ION memory each time encoding (about 25 milliseconds), which significantly improves continuous encoding performance.

[0022] Furthermore, the present invention has achieved the following unexpected technical effects: (i) On the Unisoc T615 platform, the platform-aware buffer pool strategy—that is, deciding whether to allocate a rotation buffer based on whether the platform natively supports rotation—has significantly reduced the warm-up time from the estimated 476 milliseconds to about 80 milliseconds and the memory usage from about 47MB to about 32MB. This optimization effect far exceeds the benefits brought by simple buffer caching.

[0023] (ii) By directly passing the dma-buf file descriptor of the camera Image object through SCM_RIGHTS, approximately 23MB of memory copying operations are eliminated on the client side (saving about 42 milliseconds), making the overhead of transferring large image data across processes close to the level of direct operation within the process. This effect cannot be achieved in traditional data copying schemes.

[0024] (iii) The zero-code intrusion integration scheme of automatically starting the listening thread by the shared library constructor allows the present invention to be integrated into the camera HAL service process without modifying any existing source code, which greatly reduces the cost of cross-platform porting and version maintenance. This integration method has not been reported in the existing camera HAL architecture. Attached Figure Description

[0025] Figure 1 This is a functional principle block diagram of the cross-platform YUV hardware encoding and imaging system provided in Embodiment 1 of the present invention.

[0026] Figure 2 This is a flowchart illustrating the cross-platform YUV hardware encoding and imaging method provided in Embodiment 2 of the present invention.

[0027] Figure 3 This is a schematic diagram of the IPC data interaction process based on memfd for the cross-platform YUV hardware encoding and photography method provided in this embodiment of the invention.

[0028] Figure 4 This is a flowchart of the IPC data interaction process of the cross-platform YUV hardware encoding and imaging method based on dma-buffd provided in the embodiments of the present invention.

[0029] Figure 5 This is a diagram illustrating the asynchronous processing algorithm and timing structure of the cross-platform YUV hardware encoding photography method provided in this embodiment of the invention.

[0030] Figure 6 This is the ION cache pool management and preheating process of the cross-platform YUV hardware encoding photography method provided in this embodiment of the invention.

[0031] Figure 7 This is a schematic diagram of an embodiment of the cross-platform YUV hardware encoding and imaging system provided by the present invention.

[0032] Figure 8 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0034] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0035] In Android camera application development, due to the system process isolation mechanism, the application process and the hardware abstraction layer service process run in independent address spaces, preventing the application layer from directly accessing the hardware JPEG encoder within the hardware abstraction layer. SELinux's mandatory access control policy, by default, prohibits socket communication between the application process and the hardware abstraction layer service process, requiring additional security policy configuration to establish cross-process data channels. Hardware JPEG encoders on different chip platforms differ in buffer management, encoding interfaces, and rotation processing mechanisms. For example, the MTK platform uses two-stage rotation encoding, while the Unisoc platform supports native rotation; the MTK platform uses a proprietary image buffer allocator, while the Unisoc platform uses a DMA-BUF heap allocator. These differences make it difficult to reuse encoding logic across platforms. When high-resolution YUV image data is transferred between processes, using traditional data copying methods increases transmission overhead and system latency. Furthermore, image algorithm processing and encoding conversion are executed serially in the main shooting thread. When multiple image processing algorithms are superimposed, the processing time accumulates, directly blocking the shooting response speed and limiting the number and complexity of algorithms that can be superimposed.

[0036] For example, in a smartphone's camera application, when a user presses the shutter button, the application layer retrieves the raw YUV image data generated by the camera's hardware abstraction layer. Due to the isolation between the application process and the hardware abstraction layer service process, the application layer needs to transfer data through inter-process communication mechanisms, but the SELinux policy prevents default socket connections. On the MTK platform, the hardware JPEG encoder requires the YUV data to be rotated by the mobile display processor before encoding; while on the Unisoc platform, the rotation parameters can be directly passed to the hardware encoder. Therefore, if the application layer attempts to uniformly call the hardware encoder, it must customize the implementation for each platform. Furthermore, if memory copying is used when transferring high-resolution YUV image data between processes, it will consume system resources. In addition, image algorithms such as beautification and filters are processed in the application layer's main thread, and cannot respond to new shooting commands during processing, causing users to wait for the algorithm to complete before taking the next shot when taking continuous photos.

[0037] If the above issues are not resolved, the camera application's shooting response time will increase, especially in scenarios with high-resolution images and multiple algorithms overlayed, resulting in user lag. Continuous shooting functionality will not execute smoothly due to main thread blocking, reducing user satisfaction. Adaptation work for different chip platforms will require repeated development, increasing maintenance costs. The performance advantages of the hardware JPEG encoder cannot be fully utilized, leading to wasted system resources and overall inefficiency in the shooting process.

[0038] To address this, this application proposes a cross-platform YUV hardware encoding imaging method and system. For example... Figure 1As shown in Embodiment 1 of the present invention, a cross-platform YUV hardware encoding photography method is provided. The system architecture is divided into three layers: application layer, JNI communication layer and HAL service layer.

[0039] The application layer includes: a hardware encoding interface module, which loads the native encoding library via JNI (Java Native Interface) and supports three encoding modes: shared memory transfer (memfd), direct transfer (DMA-buffd), and buffer pool warm-up; a YUV (a color encoding method) encoding management module, which manages the lifecycle of hardware encoding interface instances, prioritizes hardware encoding, and automatically falls back to software encoding when hardware encoding is unavailable; an algorithm pipeline management module, which manages the registration and execution of image processing algorithm nodes, supports direct processing of camera Image objects, and avoids the extra overhead of first converting them to byte arrays; and a filter processing module, which completes filter processing on shared memory and then directly triggers hardware encoding, achieving a one-time completion of filtering and encoding.

[0040] The JNI communication layer includes a client communication module responsible for establishing Unix domain socket connections and passing memfd or dma-buf file descriptors via the SCM_RIGHTS mechanism to achieve zero-copy or near-zero-copy image data transmission. SCM_RIGHTS is a control message type in Unix domain sockets used to pass file descriptors (FDs) between processes on the same host. The HAL (Hardware Abstraction Layer) service layer includes an encoding and listening module, which is automatically loaded into the camera's HAL service process as a shared library. This module automatically starts a listening thread through its library constructor, listening for client connections on the Unix domain socket in the abstract namespace. Upon receiving an encoding request, it allocates an ION buffer, copies the YUV data from the client into the ION buffer, calls the platform's hardware JPEG encoder to complete the encoding, and then sends the JPEG result back to the client via memfd. The ION buffer is a physical memory block used for multimedia and graphics processing in the Android / Linux system; YUV is a color encoding method that separates luminance (Y) and chrominance (U / V).

[0041] like Figure 2 As shown, a cross-platform YUV hardware encoding photography method according to Embodiment 2 of the present invention includes the following steps: S100: In the Android camera application layer, when a user operation command is detected to trigger a photo capture, the raw YUV image data generated by the camera hardware abstraction layer is obtained. S200. The application layer sends the YUV image data to the camera hardware abstraction layer service process with zero copy through the pre-established IPC channel between the abstract namespace Unix domain socket and the camera hardware abstraction layer service process, using file descriptor memfd shared memory combined with the file descriptor transfer mechanism. S300, The camera hardware abstraction layer service process calls the platform hardware JPEG encoder through the platform abstraction layer set within it; S400: The platform hardware JPEG encoder encodes and converts the YUV image data to generate JPEG image data, and then sends the JPEG image data back to the application layer via memfd shared memory.

[0042] The Android camera application layer in this embodiment of the invention refers to the user space portion of the camera application running on the Android operating system, which is responsible for user interface interaction, receiving shooting commands, and displaying and saving the final JPEG image.

[0043] The Camera Hardware Abstraction Layer (HAL) in this embodiment is an interface layer in the Android system, located between the application framework and the hardware driver. It defines the standard interfaces that camera hardware devices must implement, allowing upper-layer applications to access different camera hardware through a unified API.

[0044] The YUV image data in this embodiment is a color-coded format used for video and image processing. It stores luminance information (Y) and chromaticity information (U, V) separately. Compared to the RGB format, it generally has advantages in storage and transmission efficiency and is a common raw data format output by camera sensors.

[0045] The abstract namespace Unix domain socket in this embodiment is an inter-process communication (IPC) mechanism that allows different processes on the same host to exchange data. Abstract namespace sockets do not correspond to actual paths in the file system, and therefore are not restricted by file permissions. They are commonly used for secure inter-process communication in systems such as Android.

[0046] In this embodiment, the IPC channel refers to the inter-process communication channel, which is a mechanism provided by the operating system that allows different processes to exchange data or synchronize.

[0047] This embodiment uses the memfd file descriptor for shared memory: memfd is an anonymous file descriptor provided by the Linux kernel, which can be used to create shared memory regions. Through the file descriptor passing mechanism, one process can pass the file descriptor of the shared memory region corresponding to memfd to another process, thereby enabling two processes to share access to the same memory region and avoiding data copying.

[0048] The file descriptor passing mechanism in this embodiment is an IPC technology that allows one process to pass an open file descriptor (e.g., a file descriptor pointing to shared memory, a pipe, or a socket) to another process. The receiving process can directly use the file descriptor without reopening the file or re-establishing resources.

[0049] The zero-copy approach described in this embodiment refers to avoiding the CPU's operation of copying data from one memory region to another during data transmission. Through techniques such as direct mapping or shared memory, data can be directly transferred from the source location to the destination location, thereby significantly improving data transmission efficiency and reducing CPU overhead.

[0050] In this embodiment, the camera hardware abstraction layer service process refers to an independent process running in the Android system, responsible for managing the camera hardware abstraction layer and providing camera services to upper-layer applications. This process typically has permission to access the underlying hardware.

[0051] The platform abstraction layer is a software design pattern designed to provide a unified programming interface for different hardware platforms or operating systems. By introducing an abstraction layer on top of the hardware, upper-layer applications do not need to concern themselves with the specific implementation details of the underlying hardware, thereby improving the cross-platform compatibility and portability of the software.

[0052] The platform hardware JPEG encoder in this embodiment refers to a dedicated hardware module integrated on a specific chip platform (such as MTK, Unisoc, Qualcomm, etc.) for efficiently encoding and converting YUV format image data into JPEG format. Compared to software encoding, hardware encoders typically offer higher speed and lower power consumption.

[0053] The JPEG image data in this embodiment is a lossy image compression format widely used in digital image storage and transmission. It reduces file size by removing image information that is not sensitive to the human eye, while maintaining acceptable image quality.

[0054] This embodiment provides a cross-platform YUV hardware encoding method for taking pictures. Specifically, in the Android camera application layer, when a user operation command triggers a picture, the raw YUV image data generated by the camera hardware abstraction layer is obtained. For example, the user can trigger the picture operation by touching the shutter button on the screen, or by issuing a picture request through voice commands, gesture recognition, etc. Once the picture command is recognized, the application layer requests the current image frame data from the camera hardware abstraction layer. In one implementation, the application layer can proactively send a request to the camera hardware abstraction layer to obtain the latest generated YUV image data. In another implementation, the camera hardware abstraction layer can periodically push image data to the application layer, which then processes the received data.

[0055] Secondly, in this embodiment, the application layer uses a pre-established IPC channel between the abstract namespace Unix domain socket and the camera hardware abstraction layer service process. It employs a file descriptor (memfd) shared memory mechanism combined with file descriptor passing to send the YUV image data to the camera hardware abstraction layer service process with zero copying. To achieve efficient data transmission and avoid the data copying overhead common in traditional IPC mechanisms, this method utilizes the memfd shared memory mechanism provided by the Linux kernel. Specifically, the application layer can create a memfd shared memory region and write the YUV image data into it. Subsequently, the file descriptor of this memfd is passed to the camera hardware abstraction layer service process via the Unix domain socket. After receiving the file descriptor, the camera hardware abstraction layer service process can directly map the shared memory region, thus accessing the YUV image data without data copying. As an alternative, the application layer can also directly copy the YUV image data to a pre-allocated shared memory region and then notify the camera hardware abstraction layer service process for processing.

[0056] Furthermore, in this embodiment, the camera hardware abstraction layer service process calls the platform hardware JPEG encoder through its internally configured platform abstraction layer. Since hardware JPEG encoders on different chip platforms (e.g., MTK, Unisoc, Qualcomm) have different programming interfaces and operating methods, this method introduces a platform abstraction layer to achieve cross-platform compatibility. This platform abstraction layer provides a unified encoding interface to the upper layers, while internally encapsulating specific implementations for different hardware platforms. For example, when the camera hardware abstraction layer service process needs to perform JPEG encoding, it calls the general encoding function defined by the platform abstraction layer. The platform abstraction layer converts this general call into instructions that can be recognized and executed by the hardware JPEG encoder of the corresponding platform, based on the currently running hardware platform type. In one implementation, the platform abstraction layer can contain multiple platform-related encoding modules, loading and calling the corresponding modules by detecting the current platform at runtime. In another implementation, the platform abstraction layer can adopt a unified interface definition and require each hardware manufacturer to provide driver implementations that conform to this interface.

[0057] Finally, in this embodiment, the platform hardware JPEG encoder encodes and converts the YUV image data to generate JPEG image data, and then sends this JPEG image data back to the application layer via the memfd shared memory. Once the platform abstraction layer successfully invokes the hardware JPEG encoder, the encoder receives the YUV image data and performs efficient hardware-accelerated compression to convert it to JPEG format. After encoding, the generated JPEG image data needs to be returned to the application layer for further processing or storage. To maintain the zero-copy characteristic of data transmission, the hardware JPEG encoder can directly write the generated JPEG data into a memfd shared memory area provided by the application layer or the camera hardware abstraction layer service process. Subsequently, the file descriptor of this shared memory area is sent back to the application layer again via the IPC channel. After receiving the file descriptor, the application layer can directly read the JPEG image data from this shared memory area, avoiding additional memory copying operations. As an alternative, the hardware encoder can write the JPEG data into its internal buffer, and then the camera hardware abstraction layer service process can copy the data to a new memfd shared memory area and pass the file descriptor of this memfd to the application layer.

[0058] The following example will provide a more detailed explanation of the above technical solution: Suppose user A is using an Android smartphone to take a high-resolution photo at location A. User A wants to capture images quickly and doesn't want to miss any exciting moments due to delays in image processing or encoding.

[0059] When user A taps the shutter button on the camera application interface, the Android camera application layer detects the user's action and triggers the shutter process. At this time, the camera hardware abstraction layer has already generated the raw YUV image data. In order to efficiently transfer this high-resolution YUV image data (for example, a 40-megapixel image may reach 23MB) to the camera hardware abstraction layer service process for hardware encoding, the application layer does not use the traditional data copying method.

[0060] Instead, the application layer utilizes a pre-established IPC channel with the camera hardware abstraction layer (HAL) service process. This IPC channel is built on Unix domain sockets based on abstract namespaces and combines memfd shared memory with a file descriptor passing mechanism. Specifically, the application layer creates a memfd shared memory region and writes the YUV image data generated by the camera HAL into this shared memory. Subsequently, the application layer passes the file descriptor of this memfd to the camera HAL service process via a Unix domain socket. Upon receiving the file descriptor, the camera HAL service process can directly map it into its own address space, thus achieving zero-copy transmission of the YUV image data. This means that there is no need to copy 23MB of data from the application process's memory to the camera HAL service process's memory, significantly reducing data transfer time and CPU overhead.

[0061] Once the camera hardware abstraction layer service process obtains the YUV image data, it needs to call the platform hardware JPEG encoder for encoding. Since user A's phone may be equipped with chips from different manufacturers such as MTK, Unisoc, or Qualcomm, the hardware encoder interfaces of these chips differ. To solve this cross-platform compatibility issue, the camera hardware abstraction layer service process internally sets up a platform abstraction layer. This platform abstraction layer provides a unified set of interfaces; the camera hardware abstraction layer service process only needs to call these unified interfaces without needing to concern itself with the specific implementation details of the underlying hardware encoder. For example, it calls the platform abstraction layer's "encode" function, and the platform abstraction layer automatically selects and calls the corresponding MTK, Unisoc, or Qualcomm platform hardware JPEG encoder based on the current phone's chip type.

[0062] Subsequently, the platform's hardware JPEG encoder receives YUV image data and, utilizing its dedicated hardware acceleration capabilities, quickly encodes it into JPEG image data. After encoding, the generated JPEG image data (e.g., compressed to only a few MB) also needs to be efficiently returned to the application layer. Again, this JPEG image data is returned via the memfd shared memory mechanism. The hardware encoder can directly write JPEG data to a pre-negotiated or application-layer-provided memfd shared memory region. The camera hardware abstraction layer service process then passes the memfd file descriptor back to the application layer via an IPC channel. Upon receiving the file descriptor, the application layer can directly read the JPEG image data from this shared memory region to save or display the image, again avoiding data copying throughout the process.

[0063] Through the above process, user A can see the shooting result almost instantly after clicking the photo button, because the encoding and transmission of high-resolution YUV images are hardware-accelerated and zero-copy optimized. The entire method reduces the encoding process, which might have taken several seconds, to the millisecond level, significantly improving shooting speed and user experience, while solving several technical challenges such as cross-platform compatibility, process isolation, and data transmission efficiency.

[0064] Based on the above examples, the technical concept of this embodiment demonstrates a significant technical contribution in solving the problems of the prior art.

[0065] In existing technologies, software encoding of high-resolution images is time-consuming; for example, encoding a 40-megapixel photo may take 3.5 to 4 seconds, severely impacting user experience. This embodiment significantly shortens encoding time by efficiently calling the platform's hardware JPEG encoder at the application layer and combining it with a zero-copy data transfer mechanism, shifting the encoding process from a CPU-intensive task to a dedicated hardware acceleration module. For example, in the aforementioned user A's photo-taking scenario, the encoding process, which might have taken several seconds, is compressed to the millisecond level, enabling the user to obtain faster photo-taking response speeds and a smoother continuous shooting experience.

[0066] Furthermore, in existing solutions, the hardware JPEG encoder interfaces differ significantly across different chip platforms, making it difficult to reuse algorithms and encoding processes across platforms. This embodiment introduces a platform abstraction layer, providing a unified hardware encoding operation interface for upper-layer applications and camera hardware abstraction layer service processes. This means that regardless of whether the underlying hardware is an MTK, Unisoc, or Qualcomm platform, the upper-layer logic can be called through the same interface, thus achieving cross-platform compatibility and portability of the hardware encoding logic. This contrasts sharply with existing technologies that require separate development or adaptation of encoding logic for each platform, significantly reducing development and maintenance costs.

[0067] Furthermore, the process isolation mechanism and SELinux mandatory access control of the Android system are the main obstacles to enabling the application layer to directly call the HAL layer hardware encoder. This embodiment establishes an IPC channel using an abstract namespace Unix domain socket, combined with shared memory (memfd) and a file descriptor passing mechanism, successfully establishing an efficient and secure zero-copy data transmission link between the application layer and the camera hardware abstraction layer service process. This mechanism effectively avoids the data copying overhead of traditional IPC, solves the problem of low transmission efficiency caused by the large data volume of high-resolution YUV images, and ensures that the performance advantages of hardware encoding are not offset by the data transmission bottleneck.

[0068] In summary, this embodiment provides a high-efficiency, cross-platform, and user-friendly YUV hardware encoding method for image capture by integrating zero-copy IPC, platform abstraction layer, and hardware encoder calls. This method not only solves the problem of slow high-resolution image encoding speed but also overcomes multiple technical challenges such as cross-platform compatibility, process isolation, and data transmission efficiency, bringing significant technological advancements to the camera functionality of mobile terminals.

[0069] When the Android camera application layer detects a user's photo-taking command, some embodiments of this application propose a method to acquire raw YUV image data and send it to the camera hardware abstraction layer service process via a zero-copy IPC channel, thereby invoking the platform's hardware JPEG encoder for encoding conversion. However, in the actual implementation of this efficient cross-platform hardware encoding photo-taking process, ensuring that the IPC channel is ready before taking the photo to achieve real-time, zero-copy data transmission, and how to uniformly manage and adapt the hardware JPEG encoders on different chip platforms are key to ensuring system response speed and cross-platform compatibility.

[0070] To address this, this application further proposes that, before the step of obtaining the raw YUV image data generated by the camera HAL layer when a user operation command is detected triggering a photo capture at the Android camera application layer, an IPC inter-process communication channel is established between the Android camera application layer and the camera hardware abstraction layer service process through a Unix domain socket in the abstract namespace. This channel is used to achieve zero-copy cross-process transmission of YUV image data by using memfd file descriptor shared memory combined with the SCM_RIGHTS file descriptor transfer mechanism. Furthermore, a platform abstraction layer is pre-set within the camera hardware abstraction layer service process. This platform abstraction layer defines a unified buffer management and encoding operation interface, and hardware JPEG encoders on different chip platforms achieve platform adaptation by implementing this unified interface. The platform abstraction layer abstracts the operations of the hardware JPEG encoder into the following function interfaces: buffer allocation, buffer release, buffer attribute query, cache synchronization, format conversion, encoding capability query, and encoding execution.

[0071] The pre-establishment of an inter-process communication (IPC) channel between the Android camera application layer and the camera hardware abstraction layer service process means that the system has already established the communication link between the application layer and the camera hardware abstraction layer service process before the actual photo-taking operation occurs. This pre-established channel ensures that the data transmission path is immediately available when the photo-taking command is issued, avoiding the delay caused by dynamically establishing the channel. This channel can be implemented in several ways. For example, during system startup or camera service initialization, the camera hardware abstraction layer service process can create a Unix domain socket in an abstract namespace and listen for connection requests, while the Android camera application layer actively connects to this socket. Once the connection is successful, both parties can exchange file descriptors through this channel, such as using the SCM_RIGHTS mechanism to pass the memfd file descriptor, thereby achieving zero-copy data transmission of shared memory. Another implementation method is to introduce an independent IPC management module to coordinate the connection establishment and file descriptor exchange between the application layer and the camera hardware abstraction layer service process, thereby further decoupling the communication logic.

[0072] Setting up a platform abstraction layer (PAL) beforehand within the camera hardware abstraction layer service process means that an abstraction layer is loaded and configured when the camera hardware abstraction layer service process starts. This PAL is used to uniformly manage and call the hardware JPEG encoders of different underlying chip platforms. This PAL defines a standardized set of interfaces, so that upper-layer services do not need to concern themselves with the implementation details of specific hardware encoders. For example, this PAL can be designed as an interface class or a set of function pointers. Different hardware platforms (such as Qualcomm, MediaTek, Unisoc, etc.) will provide their own implementation modules, which are dynamically loaded and registered into the PAL at runtime. When encoding operations are required, the camera hardware abstraction layer service process only needs to call the unified interface provided by the PAL, and the PAL will automatically route to the appropriate hardware encoder implementation based on the current device's hardware type.

[0073] The platform abstraction layer abstracts the operations of the hardware JPEG encoder into a series of function interfaces, including buffer allocation, buffer release, buffer attribute query, cache synchronization, format conversion, encoding capability query, and encoding execution. These interfaces are defined to provide a common operational paradigm, shielding the underlying hardware differences. For example, the buffer allocation interface can allocate buffers for YUV or JPEG data from system memory (such as ION memory) and return a handle for upper layers to use; the buffer release interface is used to reclaim these buffer resources. The cache synchronization interface ensures data consistency between the CPU and the hardware encoder, such as flushing or invalidating the cache after data is written or read. The format conversion interface handles conversion requirements between different YUV formats (such as NV12, NV21, YV12, etc.) to adapt to the input requirements of the hardware encoder. The encoding capability query interface allows upper layers to query information such as the resolution, frame rate, and encoding format supported by the hardware encoder. The encoding execution interface is the core interface that triggers the hardware encoder to perform the actual encoding operation; it receives the input YUV buffer, the output JPEG buffer, and encoding parameters, and starts the encoding process. These interfaces can be implemented as virtual functions in C++, overridden by platform-specific encoder classes, or as a set of C-style function pointers, implemented by platform-specific libraries.

[0074] This application's solution establishes an inter-process communication (IPC) channel between the Android camera application layer and the camera hardware abstraction layer (HAL) service process using Unix domain sockets in the abstract namespace before the user triggers the photo-taking operation. This channel employs a combination of memfd file descriptor shared memory and SCM_RIGHTS file descriptor passing mechanism, laying a high-efficiency and low-latency foundation for subsequent zero-copy cross-process transmission of YUV image data. Simultaneously, a platform abstraction layer is pre-configured within the HAL service process. This platform abstraction layer, acting as a unified adaptation layer for hardware encoders, defines a series of standardized buffer management and encoding operation interfaces, such as buffer allocation, buffer release, buffer attribute query, cache synchronization, format conversion, encoding capability query, and encoding execution. When the camera HAL service process receives YUV image data from the application layer, it does not need to concern itself with the specific implementation details of the underlying hardware JPEG encoder. Instead, it calls the unified interfaces provided by the platform abstraction layer, which then translates these general operation requests into instructions recognizable and executable by the specific chip platform's hardware JPEG encoder. Hardware JPEG encoders on different chip platforms achieve platform adaptation to this method by implementing these unified interfaces. This pre-established communication mechanism and abstraction layer design enable the entire system to immediately and efficiently transmit data in a zero-copy manner after detecting the user's photo-taking command and acquiring YUV image data, and seamlessly call the adapted hardware encoder for processing, greatly improving the response speed and cross-platform compatibility of the photo-taking process.

[0075] This invention designs a platform abstraction layer, which completely separates platform-related hardware coding logic from general IPC processing logic by defining a unified operation interface.

[0076] The platform abstraction layer defines a unified set of operation interfaces, including the following functions: YUV buffer allocation, JPEG output buffer allocation, buffer release, buffer row step size lookup, image plane number lookup, buffer virtual address acquisition, image format lookup, cache synchronization operation, JPEG output size acquisition, Android format to platform format conversion, hardware encoding support capability lookup, and encoding execution. These interfaces are organized in the form of function pointers, and each chip platform completes platform adaptation by implementing all interface functions.

[0077] Taking the MTK platform as an example, the platform adaptation layer encapsulates the MTK image buffer pointer. The buffer is allocated through the MTK image buffer allocator, and the encoding uses the MTK hardware JPEG encoding interface. When rotation is required, a two-stage encoding is used—the first stage rotates the YUV data through MDP hardware, and the second stage performs JPEG encoding through a hardware JPEG encoder.

[0078] Taking the Unisoc platform as an example, the platform adaptation layer encapsulates the memory allocated by the DMA-BUF heap. The buffer is allocated through the DMA-BUF heap allocator, and the encoding is achieved by dynamically loading the Unisoc JPEG hardware encoding shared library and calling its encoding interface. The Unisoc platform's JPEG encoder natively supports rotation parameters, eliminating the need for an additional rotation buffer and two-stage encoding. During warm-up, only two buffers (approximately 32MB) need to be allocated: the source buffer and the target buffer. This saves approximately 15MB of memory compared to the three buffers (approximately 47MB) on the MTK platform.

[0079] Taking the Qualcomm platform as an example, the platform adaptation layer encapsulates the Qualcomm platform's hardware encoding buffer. The buffer is allocated through a DMA-BUF heap allocator, and encoding is completed through the Qualcomm platform's hardware JPEG encoding interface. The Qualcomm platform's JPEG encoder also natively supports rotation parameters, employing a single-rotation encoding method similar to the Unisoc platform. The Qualcomm platform adaptation process uses the same platform abstraction layer interface as the Unisoc platform; only the corresponding platform's data type conversion and encoding interface calls need to be implemented to complete the integration.

[0080] The general-purpose code layer includes: abstract namespace socket creation and listening, data transfer mode processing logic based on memfd, direct transfer mode processing logic based on dma-buffd, buffer pool warm-up mode processing logic, buffer pool management, YUV data copying helper functions, JPEG result extraction helper functions, and a shared library constructor self-starting mechanism. The general-purpose code layer does not reference any platform-specific header files or data types; all platform differences are masked through a unified operation interface.

[0081] In other embodiments, this application proposes a cross-platform YUV hardware encoding method for image capture. While the aforementioned embodiments achieve zero-copy transmission and hardware encoding of YUV image data, in practical applications, users' demands for image processing effects are increasing. Performing image processing algorithms synchronously before encoding may cause delays in the image capture process, affecting user experience and system response speed.

[0082] To address this, this application further proposes that before the step of sending YUV image data to the camera hardware abstraction layer service process with zero copy, the process includes: performing image processing algorithm operations on the YUV image data at the application layer, wherein the image processing algorithm includes at least one of beautification filters and image enhancement; the image processing algorithm is executed asynchronously in the background, and the camera application layer continuously responds to the user's shooting operation during the algorithm processing, with the algorithm processing and shooting process being executed in parallel; the IPC channel uses Unix domain sockets in an abstract namespace and transmits image data between the application layer and the camera hardware abstraction layer service process through the memfd shared memory mechanism.

[0083] Specifically, the steps of performing image processing algorithms on YUV image data at the application layer aim to optimize or correct the visual effects of the original YUV image data to meet users' personalized needs for image quality and style. For example, the image processing algorithm may include beauty filters, which use facial recognition technology to locate facial regions in the image and then apply algorithms such as skin smoothing, skin retouching, face slimming, and eye enlargement to improve the appearance of portrait photos; or, the image processing algorithm may include image enhancement, which adjusts parameters such as brightness, contrast, saturation, and sharpness of the image, or applies algorithms such as noise reduction and HDR (High Dynamic Range) mapping to improve the overall visual quality and detail of the image. Furthermore, it may also include color correction, stylized filters, etc., implemented through pixel-level transformations or deep learning-based models.

[0084] The image processing algorithm executes asynchronously in the background, while the camera application layer continuously responds to user photo-taking operations during algorithm processing. This parallel execution of algorithm processing and photo-taking ensures that image processing operations do not block the camera application's main thread, thus maintaining the smoothness and responsiveness of the user interface and allowing users to continue other operations while algorithm processing is in progress. For example, the application layer can create a separate background thread or process specifically for executing the image processing algorithm. Once the YUV image data is ready, it is passed to this background thread or process for processing, while the main thread immediately returns to continue processing user input and camera preview. Alternatively, the application layer can encapsulate the image processing task into an executable unit and place it in a task queue, with one or more worker threads retrieving tasks from the queue and executing them asynchronously. When the task is completed, the result is returned to the main thread via a callback or event notification mechanism. For computationally intensive image processing algorithms, the GPU on the mobile device can be used for acceleration. The application layer submits image data and algorithm instructions to the GPU, which processes them in parallel in the background, and the processing results are then returned to the CPU or used directly for subsequent operations.

[0085] The IPC channel uses Unix domain sockets with an abstract namespace and transmits image data between the application layer and the camera hardware abstraction layer service process via the memfd shared memory mechanism. This ensures that the processed YUV data can be efficiently and securely transferred from the application layer to the camera hardware abstraction layer service process for subsequent hardware encoding, while maintaining the advantage of zero-copy. For example, after processing the YUV data, the application layer can write the processed data to a shared memory region created using memfd. Then, it passes the file descriptor of this memfd to the camera hardware abstraction layer service process via the Unix domain socket. Upon receiving the file descriptor, the service process can directly map the shared memory region without copying the data. Alternatively, the SCM_RIGHTS mechanism can be used to directly transmit existing dma-buf file descriptors. If the image processing algorithm operates directly on dma-buf, then the file descriptor of that dma-buf can be directly transmitted, further reducing data copying.

[0086] This application's solution introduces asynchronous execution of image processing algorithms at the application layer. This allows the camera application layer to acquire raw YUV image data immediately after the user triggers a photo-taking command, and then immediately start a background thread or process to perform operations such as beauty filters and image enhancement on that data. During this process, the main thread of the camera application layer is not blocked and can continuously respond to other user photo-taking operations, such as continuous shooting. Once the image processing algorithm is complete, the processed YUV image data is efficiently sent to the camera hardware abstraction layer service process in a zero-copy manner through a pre-established IPC channel, utilizing the Unix domain sockets and memfd shared memory mechanism of the abstract namespace. The camera hardware abstraction layer service process then calls the platform hardware JPEG encoder through its internally configured platform abstraction layer to encode and convert the processed YUV image data, generating JPEG image data, and finally sending it back to the application layer via memfd shared memory. This design allows image processing and the photo-taking process to execute in parallel, effectively solving the photo-taking delay and application lag problems that may occur when introducing image processing functions, while maintaining zero-copy data transmission efficiency, ensuring the smoothness and high performance of the entire photo-taking process.

[0087] The following is a concrete example. When a user taps the shutter button in the Android camera app, the application layer first obtains the raw YUV image data generated by the camera hardware abstraction layer. At this point, the application layer does not immediately send this data to the hardware encoder, but instead submits the YUV image data to a background image processing module. This module executes asynchronously in a separate thread, for example, applying a preset beauty filter and image enhancement algorithm to the YUV data. While the image processing module is working, the main thread of the camera app can still respond to other user operations, such as the user quickly tapping the shutter button again to take a burst of photos. Once the background image processing module completes the processing of the YUV image data, it provides the processed YUV data to the application layer through a memfd shared memory region. The application layer then passes the file descriptor of this memfd to the camera hardware abstraction layer service process through a Unix domain socket in an abstract namespace. After receiving the file descriptor, the service process directly maps the shared memory region, obtains the processed YUV data, and sends it to the platform hardware JPEG encoder for encoding. After encoding, the generated JPEG data is also sent back to the application layer through the memfd shared memory.

[0088] The above technical solution, building upon the zero-copy hardware encoding of YUV image data, further introduces the ability to perform asynchronous image processing on YUV image data at the application layer. This allows camera applications to provide rich image processing functions, such as beauty filters and image enhancement, without sacrificing user experience and system responsiveness. The background asynchronous execution of image processing algorithms and their parallel execution with the shooting process effectively avoid shooting delays and application lag caused by algorithm processing. Simultaneously, the use of Unix domain sockets and memfd shared memory mechanisms in the abstract namespace for data transfer ensures that even with the added image processing steps, the transmission of YUV image data from the application layer to the hardware abstraction layer service process maintains high efficiency with zero copies, thus providing superior image quality while preserving high-speed shooting performance.

[0089] In some of the embodiments described above in this application, zero-copy transmission of YUV image data is proposed between the Android camera application layer and the camera hardware abstraction layer service process, and encoding conversion is performed by the platform hardware JPEG encoder. However, in actual implementation, if the memory buffer required for hardware encoding is dynamically allocated and released for each encoding request, especially in high-frequency continuous shooting or fast shooting scenarios, frequent memory allocation and reclamation operations will introduce significant system overhead and latency, thereby affecting the shooting response speed and overall performance.

[0090] In response, this application further proposes that the camera hardware abstraction layer service process caches the allocated hardware encoding buffer through the Linux kernel memory management component (ION) buffer pool. Before the first encoding, the Linux kernel memory management component (ION) buffer is pre-allocated and cached through a warm-up protocol, and subsequent encoding requests directly reuse the cached buffer.

[0091] The Linux Kernel Memory Management Component (ION) buffer pool is a memory management framework provided by the Linux kernel, designed to efficiently manage and share physically contiguous memory regions, especially suitable for hardware devices requiring DMA (Direct Memory Access). The ION buffer pool can be understood as a collection of memory resources maintained in the kernel by the ION driver; applications or drivers can request memory allocation from it through specific interfaces. As one implementation, the ION driver can maintain multiple memory heaps with different attributes in the kernel, such as a dedicated heap for graphics processing or a general-purpose system heap, to meet the memory characteristics requirements of different hardware. Another implementation is that ION can provide file descriptors to reference these allocated memory regions, thereby allowing zero-copy data exchange between user space and kernel space. Caching allocated hardware encoding buffers refers to keeping memory regions allocated through ION for hardware encoding operations in memory instead of releasing them immediately after each use. These buffers can be used to store raw YUV image data, encoded JPEG image data, or other intermediate data that may be needed during the encoding process. One implementation is to maintain a data structure (such as a list or queue) within the camera hardware abstraction layer service process to manage these allocated but currently idle buffers. Another implementation is to use a memory pool design pattern, pre-allocating a large block of memory and logically dividing it into multiple fixed-size buffer units, managing the occupancy status of these units. The pre-allocation and caching of the Linux kernel memory management component (ION) buffers is an optimization strategy. Its core idea is to proactively and in advance allocate a certain number of ION buffers and place them in the cache pool before the actual encoding task begins. This aims to reduce latency during initial encoding or high-frequency encoding. As one implementation, a pre-warming process can be triggered when the camera application starts or first enters shooting mode, based on preset configuration parameters (e.g., the expected maximum concurrent encoding frames, image resolution, etc.), allocating a batch of buffers at once.

[0092] Another implementation approach is for the system to dynamically trigger warm-up based on user behavior patterns or system load predictions. For example, when it detects that a user is frequently opening the camera or preparing for burst shooting, additional buffers can be allocated in advance to cope with the upcoming high load. Subsequent encoding requests directly reuse cached buffers. This means that when a new encoding request arrives, the system no longer performs time-consuming memory allocation operations, but directly retrieves an available buffer from the pre-cached buffer pool. As one implementation, when an encoding request requires a buffer, the camera hardware abstraction layer service process retrieves a buffer currently marked as free from the cache pool; after encoding is complete, the buffer is marked as available and returned to the cache pool, awaiting reuse by the next encoding request. Another implementation approach is to combine reference counting mechanisms or more complex buffer state machine management to ensure that the buffer is not incorrectly reused before the data is fully processed (e.g., JPEG data has been sent back to the application layer and consumed), thereby avoiding data conflicts or corruption.

[0093] This application's solution addresses the performance bottleneck caused by frequent memory allocation and deallocation by introducing a buffer management mechanism based on the Linux kernel memory management component (ION) into the aforementioned cross-platform YUV hardware encoding and imaging method. Specifically, the camera hardware abstraction layer service process no longer dynamically allocates memory for each YUV image data encoding request. Instead, through a warm-up protocol, it proactively requests and allocates a certain number of hardware encoding buffers from the ION buffer pool before the first encoding operation. These pre-allocated buffers are then cached, forming a reusable buffer pool. When the application layer sends a zero-copy copy of YUV image data to the camera hardware abstraction layer service process via the IPC channel, triggering the platform hardware JPEG encoder to perform encoding conversion, the service process directly obtains a free buffer from the cached buffer pool to receive the YUV data and allocates another free buffer as the storage target for the encoded JPEG data. After encoding is complete, these buffers are not immediately released back to the system but are remarked as available and returned to the cache pool for direct reuse in subsequent encoding requests. This mechanism ensures that the platform's hardware JPEG encoder always has available memory resources, thereby avoiding latency caused by memory allocation waiting and greatly improving encoding efficiency and system response speed. In this way, this application further optimizes memory resource utilization based on zero-copy data transmission, enabling the entire image capture process, especially in continuous shooting scenarios, to operate with lower latency and higher throughput.

[0094] The following is a concrete example. When a user first opens the camera application or switches from other modes to photo mode, the camera hardware abstraction layer service process initiates a warm-up process. During this process, the service process requests, for example, eight buffers from the system via the Linux kernel memory management component (ION) to store raw YUV image data. The size of each buffer is calculated based on the current camera preview resolution (e.g., 1920x1080 pixels) and the YUV format (e.g., YUV420). Simultaneously, it requests eight buffers to store encoded JPEG image data, their sizes estimated based on the expected JPEG compression ratio and maximum image size. Once these buffers are successfully allocated, they are maintained in an internal list of free buffers. When the user clicks the photo button, the YUV image data acquired by the application layer is sent to the camera hardware abstraction layer service process via memfd shared memory. The service process retrieves a YUV buffer from the free list to receive the incoming YUV data and a JPEG buffer as the output target for the platform's hardware JPEG encoder. After the platform's hardware JPEG encoder completes encoding, both buffers are marked as available and returned to the free list. If a user takes multiple photos in a short period of time, subsequent photo requests will directly retrieve the pre-allocated buffer from the free list for reuse, without having to perform time-consuming memory allocation operations again.

[0095] Through the above technical solution, the camera hardware abstraction layer service process can efficiently manage the hardware encoding buffer through the Linux kernel memory management component (ION) buffer pool. By pre-allocating and caching the buffer before the initial encoding using a warm-up protocol, and by directly reusing the cached buffer in subsequent encoding requests, the system overhead and latency caused by memory allocation and release operations are significantly reduced. This allows the system to respond to user shooting commands faster and provide consistently stable encoding performance in high-load scenarios such as continuous shooting or rapid image capture, thereby greatly improving the user experience and ensuring the smoothness and efficiency of the shooting process. This solution, combined with a zero-copy data transfer mechanism, together constructs a high-efficiency, low-latency image processing pipeline.

[0096] In some embodiments described above, this application proposes a cross-platform YUV hardware encoding method for image capture. This method achieves rapid transmission and hardware encoding of YUV image data through a zero-copy mechanism. However, in practical applications, image rotation is a common requirement in image capture functions, but different chip platforms have varying capabilities in handling image rotation. For example, some platforms' hardware JPEG encoders may not support native rotation and require additional processing steps, which may affect encoding efficiency. Furthermore, to enhance the system's scalability and flexibility, how to introduce additional functional modules without intrusively modifying the existing camera hardware abstraction layer service process source code is also a problem that needs to be solved.

[0097] In response, this application further proposes that when rotation is required, the MTK platform adopts a two-stage encoding method: first, the YUV data is rotated by the mobile display processor (MDP) hardware, and then encoded by the hardware JPEG encoder; the Unisoc and Qualcomm platforms complete the rotation encoding in one step using the rotation parameters natively supported by the hardware JPEG encoder; the encoding monitoring module is loaded into the camera hardware abstraction layer service process as a shared library, and the monitoring thread is automatically started through the shared library constructor without modifying the source code of the camera hardware abstraction layer service process.

[0098] This technical solution aims to optimize the image rotation processing flow based on the characteristics of different hardware platforms and provides a non-intrusive method for extending the functionality of the camera hardware abstraction layer service process. The Mobile Display Processor (MDP) hardware refers to a dedicated image processing unit integrated into a System-on-a-Chip (SoC), capable of efficiently performing image rotation, scaling, and other operations. When the hardware JPEG encoder lacks native rotation capabilities, the MDP hardware can be used as a pre-processing unit to pre-rotate the raw YUV data to ensure the correctness and efficiency of subsequent encoding. For example, the MDP hardware can be a standalone Image Signal Processor (ISP) module, responsible for completing all necessary preprocessing before the image data enters the encoder. For Unisoc and Qualcomm platforms, their hardware JPEG encoders natively support rotation parameters, meaning that the encoder can directly adjust the image orientation based on the input rotation parameters while performing compression. This is typically implemented through internal hardware logic or microcode within the encoder, avoiding additional data transmission and processing stages. For example, the encoder may receive rotation angle information through configuration registers and rearrange pixel data within its internal processing pipeline.

[0099] The encoding monitoring module is an independent software component whose main function is to monitor or respond to events related to image encoding. Loading it as a shared library means that the module is compiled into a dynamic link library (such as a .so file in Linux), which can be loaded and unloaded by the camera hardware abstraction layer service process at runtime. This approach allows the module to be developed, updated, and deployed independently of the main program. The shared library constructor is a special function in the shared library that executes automatically when the shared library is loaded into the process's memory space, without requiring explicit calling from the main program. Through this constructor, the encoding monitoring module can perform initialization operations, such as creating and starting an independent monitoring thread. This monitoring thread can run asynchronously in the background, performing tasks such as logging, performance statistics, and event notifications, without blocking the main thread of the camera hardware abstraction layer service process. This design avoids modifying the existing source code of the camera hardware abstraction layer service process, thereby reducing system coupling and improving system maintainability and scalability.

[0100] The solution in this application, through the aforementioned technical means, enables the system to intelligently select the optimal rotation processing strategy based on the current chip platform when image rotation is required. For the MTK platform, the raw YUV data is first sent to the mobile display processor (MDP) hardware for rotation, and the rotated YUV data is then encoded by the hardware JPEG encoder. This two-stage processing method ensures efficient image orientation adjustment even if the hardware JPEG encoder does not natively support rotation. For the Unisoc and Qualcomm platforms, since their hardware JPEG encoders natively support rotation parameters, the system can directly pass the rotation parameters to the encoder, which then completes both YUV data rotation and JPEG encoding in a single operation. This platform-adaptive rotation processing mechanism, combined with the aforementioned cross-platform YUV hardware encoding and imaging method, ensures that image rotation can be achieved in an optimized manner on different hardware platforms, avoiding unnecessary software processing overhead and thus improving encoding efficiency and performance. Furthermore, to enhance the system's scalability and flexibility, the encoding monitoring module is designed to be loaded into the camera hardware abstraction layer service process as a shared library. When the camera hardware abstraction layer service process starts or dynamically loads the shared library at runtime, its internal shared library constructor is automatically invoked. This constructor starts an independent listening thread. This listening thread can asynchronously perform listening tasks in the background, such as monitoring encoding status, collecting performance data, or triggering subsequent processing. This loading mechanism completely avoids any modification to the source code of the camera hardware abstraction layer service process, greatly reducing system coupling, improving system scalability and ease of upgrades, and making the entire cross-platform YUV hardware encoding and imaging method more robust and easier to manage.

[0101] Through the above technical solution, this application can intelligently select the optimal image rotation processing method according to the hardware characteristics of different chip platforms. For platforms that do not support native rotation by the hardware encoder, a two-stage rotation encoding is performed using the mobile display processor (MDP) hardware, ensuring the efficiency of rotation processing. For platforms that natively support rotation, the one-time rotation encoding capability of the hardware encoder is utilized to further improve encoding efficiency. This platform-adaptive rotation processing mechanism effectively solves the performance bottleneck caused by the difference in rotation capabilities of different hardware platforms, ensuring that the cross-platform YUV hardware encoding imaging method can achieve efficient and high-quality image output on various devices. At the same time, by loading the encoding monitoring module as a shared library and automatically starting the monitoring thread using the shared library constructor, the system functionality can be flexibly extended and enhanced without modifying the source code of the camera hardware abstraction layer service process. This not only reduces system maintenance costs and integration risks but also improves the modularity and upgradeability of the system, making the entire cross-platform YUV hardware encoding imaging method more robust and easier to manage.

[0102] In some embodiments described above, this application proposes to send YUV image data to the camera hardware abstraction layer service process via a pre-established IPC channel between the abstract namespace Unix domain socket and the camera hardware abstraction layer service process, using a file descriptor memfd shared memory combined with a file descriptor transfer mechanism. However, in practical applications, a single data transfer mode may not fully meet the flexible requirements for data type, transmission efficiency, and resource pre-allocation in different scenarios, especially when processing raw image data or needing to optimize resource utilization.

[0103] To address this, this application further proposes that the IPC channel supports multiple data transfer modes. The IPC channel is an inter-process communication mechanism that supports various data transfer strategies, enhancing the system's flexibility and adaptability, allowing it to select the most suitable transmission method based on specific data characteristics and performance requirements. For example, the data transfer mode can be dynamically switched based on data size, data structure complexity, or real-time requirements. Among these, transferring YUV byte array data via shared memory using the memfd file descriptor is an efficient inter-process communication mechanism, allowing different processes to share the same memory region. By transferring YUV byte array data, zero-copy transmission of image data can be achieved, avoiding the overhead of copying data between different process address spaces. This can be implemented by the application layer allocating memory in memfd, writing the YUV data, and then passing the memfd file descriptor to the service process, which then reads the data by mapping the same memory region using this descriptor; or the service process pre-allocating memfd memory, passing the descriptor to the application layer, and the application layer writing the data. Furthermore, the dma-buf file descriptor of the camera image can be directly transferred via the SCM_RIGHTS file descriptor. The SCM_RIGHTS file descriptor passing mechanism allows one process to pass a file descriptor (such as a dma-buf file descriptor) to another process. dma-buf is a mechanism provided by the Linux kernel for sharing memory between different hardware devices, commonly used in graphics and video processing to achieve zero-copy hardware acceleration. Directly passing the dma-buf file descriptor means that the application layer can directly pass the handle of the raw image buffer (usually dma-buf) generated by the camera hardware to the service process. The service process can directly access the hardware buffer without data copying, further improving transmission efficiency and reducing CPU overhead. This can be achieved by using the sendmsg and recvmsg system calls on Unix domain sockets and setting the SCM_RIGHTS auxiliary data. Furthermore, the Linux kernel memory management component (ION) buffer pool is pre-allocated in warm-up mode without performing actual coding. Warm-up mode is a performance optimization strategy designed to reduce latency in subsequent operations by pre-allocating and initializing resources. The Linux kernel memory management component (ION) buffer pool is a commonly used memory management framework in Android systems, used to manage the physical contiguous memory required by hardware accelerators. By pre-allocating the ION buffer pool in warm-up mode, the necessary memory resources can be prepared before the actual encoding request arrives, avoiding time-consuming memory allocation operations on the critical path. In this mode, although buffers are allocated, encoding is not performed immediately; instead, these buffers are placed in a standby state, waiting for subsequent encoding requests.This can be achieved by sending an allocation request to the ION driver during system startup or application initialization, obtaining a certain number of ION buffer handles, and caching them.

[0104] This application's solution expands the data transmission capabilities of the IPC channel, enabling it to flexibly select the most suitable transmission mechanism based on different application scenarios and data characteristics. When the application layer needs to transmit software-processed YUV byte array data, the IPC channel can utilize the memfd file descriptor shared memory mechanism to achieve zero-copy data transmission, effectively reducing the overhead of data copying. When processing raw image data directly generated by the camera hardware (e.g., dma-buf), the IPC channel can directly pass the dma-buf file descriptor from the application layer to the camera hardware abstraction layer service process via the SCM_RIGHTS file descriptor passing mechanism. This approach avoids any data copying of the raw hardware buffer, allowing the service process to directly access hardware memory, greatly improving transmission efficiency and reducing CPU load. Furthermore, to handle high-concurrency or low-latency encoding requirements, the IPC channel also supports a warm-up mode. In this mode, the system pre-allocates the Linux kernel memory management component (ION) buffer pool and caches these buffers without immediately performing encoding operations. When an actual encoding request arrives, these allocated buffers can be reused directly, eliminating memory allocation latency and ensuring a smooth and efficient encoding process. Through the synergistic effect of these multiple data transmission modes, the IPC channel can provide optimal transmission paths for image data from different sources and processing stages, significantly improving the flexibility, efficiency, and response speed of the entire imaging method, and effectively solving the limitations of a single transmission mode in diverse scenarios.

[0105] Please see Figure 3 and Figure 4 This invention defines the following IPC data interaction modes: The data transfer mode based on memfd is as follows: The client sends an encoding request header (format, width, height, quality, rotation). The server allocates an ION buffer, creates a memfd, and sends it to the client via SCM_RIGHTS. The client writes YUV data to the mmap-encoded memfd and sends a ready signal. The server copies the YUV data from the memfd to the ION buffer, encodes it using the hardware encoder, writes the JPEG result to a new memfd, and sends it back to the client via SCM_RIGHTS. After the client confirms the completion, the server releases the buffer.

[0106] The direct transmission mode based on dma-buf: The client obtains the dma-buf file descriptor from the camera's Image object via AHardwareBuffer_getNativeHandle, and sends it directly to the server along with the encoding request headers (format, width, height, quality, rotation, yStride, uvStride) via SCM_RIGHTS. The server then mmaps the file descriptor and directly reads the YUV data, saving the client approximately 23MB of memory copying operations (saving approximately 42 milliseconds).

[0107] Buffer pool warm-up mode: The client sends a warm-up request, and the server performs buffer allocation and release to fill the buffer pool, but does not perform actual encoding. Warming is performed asynchronously during the idle time after taking a picture, and the buffer pool can be hit during the first encoding, saving about 25 milliseconds of ION buffer allocation time.

[0108] Please see Figure 5 Another important core innovation of this invention is to change the image algorithm processing to a background asynchronous execution mode.

[0109] In traditional photo-taking processes, algorithm processing and encoding conversion are executed sequentially in the main photo-taking thread. After the user clicks to take a photo, the system completes the following in order: acquiring YUV data, performing algorithm processing, encoding JPEG, and saving the file, before it can respond to the next photo-taking attempt. When multiple algorithms are combined (such as BST filter + beauty mode + watermark), the total algorithm time accumulates, directly blocking the photo-taking response.

[0110] This invention moves the algorithm processing to a separate background working thread. When the user clicks to take a picture: (1) The camera HAL layer generates YUV data and submits it to the algorithm pipeline management module; (2) The algorithm pipeline management module distributes the YUV data to the background working thread to perform algorithm processing; (3) The camera application immediately returns to the ready state, and the user can continue to take pictures; (4) After the background thread completes the algorithm processing, it sends the processed YUV data to the HAL service process through the IPC channel for hardware encoding; (5) The encoded JPEG data is saved to a file.

[0111] With the asynchronous processing architecture described above, algorithm processing time no longer affects the camera's response speed. The camera can take photos continuously and quickly, while the algorithm processes and encodes each photo in the background. This allows for the layering of more image processing algorithms (such as multi-level beautification, complex filters, text watermarks, frame watermarks, etc.), significantly improving photo quality without affecting the shooting experience.

[0112] like Figure 6 As shown, this invention designs an ION buffer pool management mechanism in the HAL service process.

[0113] The buffer pool caches allocated ION buffers using image format, width, and height as keys. These include YUV input buffers, JPEG output buffers, and rotation buffers (allocated only on platforms requiring rotation, such as MTK; Unisoc and Qualcomm platforms do not require allocation because the encoder natively supports rotation). The buffer pool uses a single-entry design, directly reusing the allocated buffer when a hit occurs, and reallocating and replacing the old buffer when a miss occurs.

[0114] The preheating mechanism is triggered during the idle time after the photo is taken. The application layer background thread sends a preheating request to the HAL service process. The server performs buffer pool allocation and caching operations, completing the allocation and caching of the ION buffer, but does not perform actual encoding. After preheating is complete, subsequent encoding requests can hit the buffer pool, skipping the ION allocation phase.

[0115] On the Unisoc and Qualcomm platforms, since the hardware JPEG encoder natively supports rotation, there is no need to allocate a rotation buffer. During warm-up, only two buffers, the source buffer and the target buffer, need to be allocated (approximately 32MB on the Unisoc platform, taking about 80 milliseconds). On the MTK platform, however, three buffers need to be allocated: the source buffer, the target buffer, and the rotation buffer (approximately 47MB, taking about 476 milliseconds). Through the platform-aware buffer pool strategy, warm-up overhead and memory usage are effectively reduced.

[0116] Furthermore, this invention designs a complete security policy configuration for SELinux mandatory access control in the Android system: On the HAL server side, add Unix domain socket listening permissions, memfd creation and write permissions, and memfd security tag relabeling permissions to the SELinux security domain corresponding to the camera HAL service process. The memfd sets security context labels through the extended attribute interface to ensure consistent SELinux policies when passing data across processes.

[0117] On the application side, permissions are added to the SELinux security domain corresponding to the application process, including Unix domain socket connection permissions, socket file read / write permissions, Unix domain socket operation permissions, and shared memory read / write mapping permissions. All policies ensure normal communication across security levels through multi-level trusted object attributes.

[0118] In a further embodiment, the integration of the present invention with the HAL server adopts a zero-code intrusion scheme: The monitoring module is compiled as a shared library, and integration is achieved by adding a single line of shared library dependency to the camera HAL service's compilation script. The shared library is automatically loaded along with the camera HAL service process via ELF dynamic library dependencies.

[0119] The listening thread starts automatically via the shared library constructor mechanism, automatically initializing the platform operation interface when the library is loaded, and then creating the listening thread. The entire integration process requires no modification to the camera HAL service source code, init configuration file, or any existing files.

[0120] Regarding the performance test data for specific implementations of this invention, the invention has been tested and verified on multiple mainstream chip platforms, and the data is as follows: 1) MTK6789 platform (Dimensity 720, 4608x2592 NV21): The complete photo-taking process using software encoding takes approximately 950 milliseconds. Complete hardware coding process for taking a picture: approximately 600 milliseconds; Performance improvement: approximately 37%; The time consumed in each stage is as follows: warm-up (hidden in the background) about 43 milliseconds, filter processing about 67-72 milliseconds, copying memfd to ION about 6-15 milliseconds, hardware JPEG encoding (including rotation) about 50-62 milliseconds, and JPEG saving about 30-40 milliseconds.

[0121] 2) Unisoc T615 platform (UMS9230, 4096x2304 NV21, weight 95, rotation 90 degrees): The complete photo-taking process using software encoding takes approximately 3500-4000 milliseconds. Complete hardware coding process for taking a picture: approximately 600-800 milliseconds; Performance improvement: over 75%; Among the improvements: the warm-up buffer pool was optimized from an estimated 476 milliseconds to approximately 80 milliseconds (the Unisoc platform does not require a rotating buffer), hardware JPEG encoding was approximately 40 milliseconds, the total time for the first photo was optimized from approximately 1737 milliseconds to approximately 690 milliseconds, and memory usage was reduced from approximately 47MB to approximately 32MB.

[0122] 3) Qualcomm Platform: The Qualcomm platform's hardware JPEG encoder natively supports rotation parameters, and its buffer management mechanism is similar to that of the Unisoc platform, employing a DMA-BUF heap allocator. After adapting to the Qualcomm platform through the platform abstraction layer interface of this invention, all application layer code and IPC communication protocols can be reused. Given the high performance of the hardware JPEG encoder integrated into the Qualcomm platform SoC, it is expected to achieve encoding performance improvements comparable to or better than those of the Unisoc platform.

[0123] As can be seen from the above, this invention does not simply replace software coding with hardware coding. Instead, it designs a cross-platform, portable, and high-performance YUV hardware coding scheme based on the overall architecture of the image data processing flow. It achieves efficient data transfer between the application layer and the HAL service layer through Unix domain socket IPC channels, fully reuses coding logic across different chip platforms through a platform abstraction layer, decouples algorithm processing from the image capture process through asynchronous algorithm pipelines, optimizes continuous coding performance through an ION buffer pool warm-up mechanism, and reduces portability and maintenance costs through a zero-code-intrusion integration scheme.

[0124] This invention achieves a significant improvement in encoding performance (37% improvement on the MTK platform and over 75% improvement on the Unisoc platform) while ensuring encoding quality and display effect. At the same time, it gives the system good portability and scalability. It currently supports mainstream mobile terminal chip platforms such as MTK, Unisoc, and Qualcomm, providing reliable technical support for terminal devices to quickly integrate high-performance photography functions on different chip platforms.

[0125] Additionally, at the Android camera application layer, when a user operation command triggers a photo capture, the raw YUV image data generated by the camera hardware abstraction layer is obtained. This data is then sent to the camera hardware abstraction layer service process via a pre-established IPC channel between the abstract namespace Unix domain socket and the process using file descriptor memfd shared memory combined with a file descriptor transfer mechanism, resulting in a zero-copy transmission. The camera hardware abstraction layer service process then calls the platform hardware JPEG encoder through its internally configured platform abstraction layer to encode the YUV image data, generating JPEG image data, which is then sent back to the application layer via memfd shared memory. However, in scenarios involving continuous high-speed shooting, frequent buffer allocation and release, resource contention during multi-frame concurrent processing, and data return delays after encoding can still lead to system performance bottlenecks, affecting the smoothness and responsiveness of photo capture.

[0126] To address this, this application further proposes an optimized buffer management scheme, including: constructing a time-slot-based circular buffer management structure to divide the buffer pool into a fixed number of consecutive time slots, with each time slot pre-bound to a complete ION buffer (including a source YUV buffer, a target JPEG buffer, and a platform-specific rotation buffer), forming a logical circular queue; each time slot independently manages its buffer occupancy status (idle, filling, encoding, and transmission); continuously captured image frames occupy different time slots in chronological order, physically isolating the buffer resources of adjacent frames and eliminating resource conflicts during multi-frame concurrency; setting up high-watermark and low-watermark monitoring mechanisms within the circular buffer; real-time monitoring of the encoding request density per unit time, and automatically triggering an extension protocol when a high-frequency continuous shooting mode is detected and the number of idle time slots is lower than the low-watermark, requesting additional buffer groups from the system ION buffer memory and injecting them into the preparation queue; when the shooting frequency... When the rate decreases and idle time slots exceed the high watermark, excess buffers are gradually released to the low watermark level to dynamically balance memory usage and concurrent processing capabilities; a buffer inheritance relationship is established between adjacent shooting frames to achieve zero-release reuse; after the previous frame completes hardware encoding, the ION buffer group it occupies is not immediately recycled to the general pool or released, but directly enters a hot standby state, and the memory mapping is kept valid through a reference counting mechanism, and it is preferentially allocated to the next frame encoding request; the real release and reallocation are only triggered when the buffer size is mismatched or the shooting interval exceeds the threshold; a buffer state machine driven by the hardware encoding completion signal is introduced; when the hardware encoder completes JPEG compression and triggers an interrupt, the buffer state of the corresponding time slot is immediately migrated from encoding to reusable, so that it can be immediately occupied by subsequent frames; and the operation of JPEG data being sent back to the application layer uses an independent temporary mapping buffer or directly establishes a read-only mapping in the original buffer for asynchronous transmission.

[0127] The construction of a time-slot-based circular buffer management structure refers to logically dividing the memory region used to store image data into multiple independent, fixed-size units, called "time slots," according to preset rules. These time slots are organized into a circular sequence, forming a circular queue. This structure ensures the orderly flow of data during processing and provides independent storage space for concurrent operations. For example, an array or linked list data structure can be used to represent the circular queue, and the order and access of time slots can be managed through indexes or pointers.

[0128] Dividing the buffer pool into a fixed number of contiguous time slots means pre-dividing the entire available memory buffer area into several logically contiguous storage units of the same size. This partitioning method simplifies memory management and avoids the overhead and fragmentation problems caused by dynamic allocation at runtime. For example, a large block of contiguous physical memory can be allocated at system startup and then divided into multiple time slots, with the size of each time slot determined according to the expected image resolution and data format.

[0129] Pre-binding a complete ION buffer to each time slot means that each logical time slot is pre-allocated and associated with a complete set of physical memory blocks managed by the Linux kernel memory management component (ION) at creation. This set of memory blocks contains all the buffers needed to process a frame of image, such as a source buffer for storing raw YUV data, a target buffer for storing encoded JPEG data, and a platform-specific rotation buffer required for image rotation on some platforms (such as MTK). This pre-binding mechanism ensures that all necessary memory resources are ready at the start of the image processing flow, eliminating the need for temporary allocation and reducing latency. For example, during time slot initialization, three independent ION handles can be allocated for each time slot via the ION driver interface, corresponding to the YUV, JPEG, and rotation buffers respectively, and these handles can be stored in the time slot's data structure.

[0130] A logically circular queue refers to a sequence of time slots that logically form a continuous loop. After the last time slot is processed, the next processing request will start again from the first time slot, repeating the cycle. This queue structure enables circular reuse of the buffer, effectively managing limited memory resources, and is particularly suitable for processing continuous data streams. For example, a circular queue can be managed by maintaining a read pointer and a write pointer; when the write pointer reaches the end of the queue, it will wrap around to the beginning of the queue.

[0131] Managing the occupancy status of each time slot's buffer independently means that each time slot maintains an internal state variable to indicate its current usage. These states can include "idle" (unused), "filling" (writing YUV data), "encoding" (hardware encoder is processing), and "retransmission" (JPEG data is being transmitted back to the application layer), etc. Independent state management ensures that the lifecycle of each time slot is clear and controllable, avoiding resource conflicts between different processing stages. For example, these states can be defined using enumeration types, and updated via atomic operations or mutexes to guarantee thread safety.

[0132] The cyclical allocation of continuously captured image frames across different time slots means that when a user captures images continuously, each new image data frame is allocated to the next available time slot in a circular queue for processing. This cyclical allocation method ensures that different frames use different physical buffer resources, thus avoiding memory access conflicts that may occur when processing multiple frames concurrently. For example, the system can maintain an index of the "current write time slot," which is incremented and points to the next free time slot whenever a new image frame arrives, wrapping around if it reaches the end of the queue.

[0133] Physically isolating buffer resources between adjacent frames refers to allocating a separate physical memory region for each frame, ensuring that even when multiple frames are being processed at different stages simultaneously, their memory data will not overwrite or interfere with each other. This physical isolation is the foundation for achieving stability in multi-frame concurrent processing. For example, because each time slot is pre-bound to an independent ION buffer, these buffers are independent of each other in physical memory, thus achieving data isolation.

[0134] Eliminating resource conflicts during multi-frame concurrency refers to avoiding contention for the same memory region or hardware resources when multiple image frames are simultaneously performing data padding, hardware encoding, or data transmission, through the aforementioned physical isolation and state management mechanisms. This helps improve system stability and processing efficiency. For example, when one time slot is in the "encoding" state, other time slots can be in the "padding" or "transmission" state simultaneously without affecting each other.

[0135] The high-water level and low-water level monitoring mechanism involves setting two thresholds, namely the "high-water level" and the "low-water level," within the circular buffer. These thresholds are used to dynamically monitor the number of idle time slots and trigger corresponding buffer resource adjustment strategies based on the number of idle time slots. For example, the high-water level can be set to 80% of the total number of time slots, and the low-water level can be set to 20%.

[0136] Real-time monitoring of encoding request density per unit time refers to the system continuously tracking the number of image encoding requests received within a specific time window. By analyzing the request density, the system can determine whether it is currently in a high-frequency continuous shooting mode, thereby predicting future resource needs. For example, the number of newly entered image frames in the encoding queue can be counted every 100 milliseconds, and their average rate can be calculated.

[0137] When a high-frequency continuous shooting mode is detected and the number of idle time slots falls below the low watermark, the expansion protocol is automatically triggered. This involves requesting additional buffer groups from the system's ION buffer memory and injecting them into the reserve queue. This means that when the system determines that the current shooting intensity is high and available buffer resources are insufficient, it proactively requests more ION memory from the operating system and organizes it into new buffer groups, adding them to the buffer management system as backup resources. This is a proactive resource expansion strategy designed to cope with sudden high load situations. For example, when the number of idle time slots falls below the low watermark, the system can request an additional 8 time slots of ION buffer at once and mark them as "ready," waiting to be absorbed by the circular queue.

[0138] When the shooting frequency decreases and idle time slots exceed the high watermark, excess buffers are gradually released to the low watermark level. This dynamic balancing of memory usage and concurrent processing capacity means that when the system detects a decrease in shooting intensity and a large number of unused buffers, it gradually releases excess ION memory resources to reduce system memory usage. This dynamic adjustment mechanism ensures that the system meets performance requirements while maximizing memory resource conservation. For example, when idle time slots exceed the high watermark, the system can release an additional buffer group that is in an "idle" state at regular intervals until the number of idle time slots falls back to near the low watermark.

[0139] Establishing a buffer inheritance relationship between adjacent shooting frames to achieve zero-release reuse means that when processing consecutive image frames, buffers used in the previous frame are not immediately and completely released back to the system after completing their task, but are marked as available for direct reuse in the next frame. This "inheritance" relationship avoids frequent memory allocation and release operations, thereby significantly reducing system overhead and latency. For example, after a time slot completes JPEG encoding and data return, its internal ION buffer group can enter a "hot standby" state, waiting to be directly used by the next frame.

[0140] After the previous frame is hardware encoded, the ION buffer group it occupies is not immediately reclaimed to the general pool or released. Instead, it directly enters a hot standby state, maintaining the memory mapping validity through a reference counting mechanism. Prioritizing allocation to the next frame's encoding request means that after the hardware encoder completes encoding a frame, the ION buffer group used by that frame is not immediately destroyed or returned to the system memory pool. Instead, its memory mapping validity is maintained, and it enters a "standby" state. Reference counting ensures that even if it is not immediately used by the next frame, the memory will not be accidentally released. When a new encoding request arrives, the system prioritizes checking whether these hot standby buffers are available, thus achieving fast reuse. For example, when the reference count of a time slot drops to 1 (indicating that only the buffer manager holds a reference), it can enter a hot standby state, waiting to be acquired by the next encoding request.

[0141] The principle of triggering actual release and reallocation only when buffer sizes mismatch or shooting intervals exceed a threshold means that buffers in hot standby mode are only actually released and reallocated under two specific conditions: first, when a new encoding request requires a buffer of a different size (e.g., switching from shooting a small image to shooting a large image), causing the existing buffer to be unable to meet the demand; second, when the hot standby state lasts for too long, exceeding a preset threshold, indicating that the buffer has not been reused for a period of time, the system will release it to save memory. For example, a timeout threshold of 500 milliseconds can be set; if a hot standby slot is not reused after this time, the actual release of its internal ION buffer is triggered.

[0142] Introducing a hardware encoder completion signal-driven buffer state machine means that the buffer's state transitions no longer depend solely on software logic, but are directly driven by completion signals (such as interrupts) emitted by the hardware encoder. This mechanism ensures that buffer state updates are synchronized with hardware operations, minimizing latency caused by software processing. For example, after the hardware encoder completes JPEG compression, it triggers an interrupt, and the interrupt service routine immediately updates the state of the corresponding time slot.

[0143] When the hardware encoder completes JPEG compression and triggers an interrupt, the buffer state of the corresponding time slot is immediately transitioned from "encoding" to "reusable," allowing it to be immediately occupied by subsequent frames. This means that once the hardware encoder completes its task and sends a signal, the system immediately updates the buffer state of that time slot from "encoding" to "reusable" (e.g., "idle" or "hot standby"). This instantaneous state transition ensures that buffer resources can be released and used for encoding the next frame as quickly as possible, thus shortening the occupancy period of a single buffer. For example, the interrupt handler identifies the time slot that has completed encoding based on the interrupt source and updates its state to "hot standby," making it immediately available for allocation to the next encoding task by the scheduler.

[0144] The asynchronous transfer of JPEG data back to the application layer, using either a separate temporary mapped buffer or a read-only mapping directly on the original buffer, decouples the JPEG data transfer process from the release and reuse of the buffer itself to avoid blocking the rapid transition and reuse of the buffer state. The system can employ two methods: first, copying the JPEG data to a separate temporary buffer and then mapping this temporary buffer to the application layer; second, directly establishing a read-only mapping on the original JPEG target buffer for asynchronous reading by the application layer. Both methods ensure that the original buffer can quickly enter a reusable state without waiting for the data to be fully transferred to the application layer. For example, after hardware encoding is completed and an interrupt is triggered, the system can immediately initiate a DMA transfer to move the JPEG data from the ION buffer to a memory area accessible to the application layer, while simultaneously marking the ION buffer as reusable.

[0145] This application's solution isolates multi-frame resources spatially through a time-slotted ring structure, dynamically adapts to traffic fluctuations temporally through a dual-watermark mechanism, eliminates unnecessary memory allocation overhead through an inter-frame inheritance protocol, and compresses resource occupancy cycles through asynchronous state transitions. These four elements work together to allocate ready-to-use buffer resources for each frame encoding request with constant low latency in continuous high-speed shooting scenarios.

[0146] As a specific implementation, a circular buffer containing 32 time slots can be constructed. Each time slot pre-allocates a 1920x1080 pixel YUV420 format source YUV buffer, a JPEG target buffer with a maximum capacity of 2MB, and a platform-specific buffer for rotation. These buffers are allocated through the Linux ION memory manager, and their file descriptors are stored in the corresponding time slot structure. During system initialization, a buffer of 16 time slots can be pre-allocated, with the low watermark set to 8 free time slots and the high watermark set to 24 free time slots. When the user initiates continuous shooting mode, if the detected frame rate exceeds 10 frames per second and the number of free time slots is less than 8, the system can immediately request an additional 8 time slots from ION memory and add them to the reserve queue to handle sudden processing demands. When a frame of an image completes hardware encoding—for example, when the hardware JPEG encoder notifies the system of encoding completion via an interrupt—the time slot status occupied by that frame can be immediately updated from "encoding in progress" to "hot standby." At this point, the buffer for that time slot is not immediately released; instead, its memory mapping is kept valid through reference counting. If the request to capture the next frame arrives within 50 milliseconds, and the requested image size matches the buffer size of the current hot standby time slot, the buffer for that hot standby time slot will be preferentially allocated for the next frame, thus avoiding the overhead of memory reallocation. Only if the hot standby state lasts for more than 200 milliseconds, or if the image size requested for the next frame changes (e.g., from 1920x1080 to 3840x2160), will the actual release and reallocation of the buffer for that time slot be triggered. Meanwhile, the encoded JPEG data can be asynchronously transferred to the application layer via the DMA controller without blocking the fast multiplexing of that time slot.

[0147] Through the above technical solutions, this application effectively solves the problems of frequent memory allocation, resource conflicts, and increased encoding latency that may be caused by traditional buffer management mechanisms in high-speed continuous shooting scenarios. The time-slotted ring buffer structure spatially isolates the resources required for multi-frame processing, ensuring the stability of concurrent processing. The dual-watermark mechanism enables dynamic scaling of buffer resources, avoiding memory waste and resource exhaustion. The inter-frame inheritance protocol significantly reduces unnecessary memory allocation and release operations, lowering system overhead. The asynchronous state transition mechanism driven by the hardware encoding completion signal minimizes the buffer resource occupancy period, enabling rapid buffer reuse. These measures work together to ensure that in continuous high-speed continuous shooting mode, the system can provide ready buffer resources for each frame encoding request with constant low latency, thereby significantly improving the shooting response speed and user experience, especially in scenarios requiring rapid capture of continuous images.

[0148] Exemplary device like Figure 7 As shown, this embodiment of the invention provides a cross-platform YUV hardware encoding imaging system, including: Application layer encoding interface module 310 is used to provide a Java interface for hardware JPEG encoding in the Android camera application layer, load the local encoding library through JNI and establish an IPC connection with the camera hardware abstraction layer service process; IPC communication module 320 is used to transfer image data between the application layer and the camera hardware abstraction layer service process via Unix domain sockets and shared memory; Platform abstraction layer module 330 is set in the camera hardware abstraction layer service process to define a unified hardware encoding operation interface and adapt to the hardware JPEG encoder implementation of different chip platforms. The hardware encoding execution module 340 is used to call the platform's hardware JPEG encoder to complete the YUV to JPEG encoding conversion; The algorithm pipeline management module 350 is used to manage image processing algorithm nodes at the application layer, supporting asynchronous execution and overlay processing of algorithms. The algorithm pipeline management module works in conjunction with the application layer encoding interface module to trigger hardware encoding after the algorithm processing is completed; as described above.

[0149] The cross-platform YUV hardware encoding photography system uses any of the above-described cross-platform YUV hardware encoding photography methods to perform photography processing.

[0150] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 8 As shown, the smart terminal can be a smartphone, smart TV, smart large-screen smart terminal, etc. The smart terminal includes a processor, memory, network interface, display screen, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a cross-platform YUV hardware encoding photography method. The database of the smart terminal stores cross-platform YUV hardware encoding photography programs.

[0151] Those skilled in the art will understand that Figure 3 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0152] In one embodiment, a smart terminal is provided, including a memory and one or more programs. The programs are stored in the memory and configured to be executed by one or more processors. These programs include methods for performing image processing such as the cross-platform YUV hardware encoding method described above. This smart terminal achieves efficient image processing compatible with multiple platforms without modifying the hardware architecture by embedding the cross-platform YUV hardware encoding method into an executable program. Specifically, when the processor executes the program stored in the memory, the program calls the application-layer encoding interface module 8. This module loads the local encoding library via JNI and establishes an IPC connection with the camera hardware abstraction layer service process. Simultaneously, the algorithm pipeline management module 6 cooperates with the application-layer encoding interface module 8 to perform asynchronous processing of the YUV image data using beautification filters or image enhancement algorithms at the application layer, ensuring that the camera application layer continuously responds to the user's image capture operation during algorithm processing. Building upon this foundation, the IPC communication module 7 establishes an IPC channel through the abstract namespace Unix domain socket, using shared memory via the file descriptor memfd combined with the SCM_RIGHTS file descriptor transfer mechanism to achieve zero-copy cross-process transmission of YUV image data. The platform abstraction layer module 5 defines a unified buffer management and encoding operation interface, enabling hardware JPEG encoders on different chip platforms to adapt by implementing this unified interface. Specifically, the platform abstraction layer module 5 abstracts the operations of the hardware JPEG encoder into function interfaces for buffer allocation, buffer release, buffer attribute query, cache synchronization, format conversion, encoding capability query, and encoding execution.

[0153] The core innovation of this embodiment lies in integrating the cross-platform YUV hardware encoding photography method into the executable program of the smart terminal. This achieves efficient invocation of the hardware encoder and asynchronous processing of image algorithms without relying on specific platform APIs, resulting in a significant reduction in encoding time and support for the overlay of multiple algorithms. Specifically, when the processor executes the program, the smart terminal triggers the platform abstraction layer module 5 to call the platform hardware JPEG encoder 4. For the MTK platform, a two-stage encoding method is adopted: first, the YUV data is rotated by the mobile display processor MDP hardware before encoding. For the Unisoc and Qualcomm platforms, rotation encoding is completed in one step using the rotation parameters natively supported by the hardware JPEG encoder 4. Simultaneously, the hardware encoding execution module 3 utilizes the Linux kernel memory management component ION buffer pool to cache the allocated hardware encoding buffer. The buffer is pre-allocated and reused through a warm-up protocol to avoid the overhead of repeated allocation during high-frequency continuous shooting. In addition, the constructed time-slot-based ring buffer management structure divides the buffer pool into a fixed number of continuous time slots. Each time slot is pre-bound to a complete ION buffer group. The number of buffers is dynamically adjusted through a high-water level and low-water level monitoring mechanism. The buffer inheritance relationship between adjacent shooting frames is established to achieve zero-release reuse, thereby eliminating resource conflicts and maintaining low latency in continuous high-speed shooting scenarios.

[0154] Through the above technical solution, this smart terminal effectively solves the technical problems pointed out in the background: On the one hand, the abstract namespace Unix domain socket combined with the memfd shared memory mechanism breaks through the process isolation and SELinux restrictions of the Android system, realizing zero-copy transmission of YUV image data and avoiding the data copying overhead of 23MB-level high-resolution images; on the other hand, the platform abstraction layer module 5 shields the fundamental differences in buffer allocation methods and rotation processing mechanisms of platforms such as MTK, Unisoc, and Qualcomm, decoupling image algorithm processing from hardware encoding processes. The algorithm pipeline management module 6 supports the asynchronous execution of algorithms such as beauty filters in the background, allowing users to take the next photo without waiting for encoding to complete. Taking 40-megapixel image processing as an example, the hardware encoding time is reduced from 3.5 to 4 seconds in software encoding to the millisecond level, while the number of algorithms superimposed is increased by more than 3 times, significantly optimizing the continuous shooting experience and image processing capabilities. In summary, this smart terminal, through programmatic integration of a cross-platform YUV hardware encoding shooting method, fully leverages hardware encoding performance while ensuring cross-platform compatibility, providing an efficient and reliable technical implementation path for the camera function of mobile terminals.

[0155] In other embodiments, this application proposes a computer-readable storage medium that, when executed by a processor of an electronic device, enables the electronic device to perform the aforementioned cross-platform YUV hardware encoding imaging method. The core innovation of this embodiment lies in that, through the implementation of this computer-readable storage medium, the electronic device can efficiently execute the cross-platform YUV hardware encoding imaging method, thereby achieving efficient invocation of hardware JPEG encoders for various platforms at the application layer without relying on specific platform APIs, and supporting asynchronous execution of image processing algorithms to overcome the performance bottleneck of continuous imaging.

[0156] Specifically, when the instructions in the computer-readable storage medium are executed, the electronic device is guided to implement the following technical solutions: In the Android camera application layer, when a user operation command is detected to trigger a photo capture, the raw YUV image data generated by the camera hardware abstraction layer is obtained; the application layer sends the YUV image data to the camera hardware abstraction layer service process with zero copy via a pre-established IPC channel between the abstract namespace Unix domain socket and the camera hardware abstraction layer service process, using a file descriptor memfd shared memory combined with a file descriptor transfer mechanism; the camera hardware abstraction layer service process calls the platform hardware JPEG encoder through its internally set platform abstraction layer; the platform hardware JPEG encoder encodes and converts the YUV image data to generate JPEG image data, and then sends the JPEG image data back to the application layer via memfd shared memory.

[0157] This application's solution effectively solves the communication barriers caused by Android system process isolation and SELinux security policies by combining a zero-copy IPC mechanism with platform abstraction layer design, while also overcoming cross-platform compatibility issues caused by differences in hardware encoding interfaces across different chip platforms. Addressing the large data volume of high-resolution YUV images, a memfd shared memory and file descriptor transfer mechanism is used to achieve true zero-copy transmission, avoiding the performance loss associated with traditional data copying methods. Furthermore, by supporting asynchronous execution of image processing algorithms, algorithm processing and the image capture process can be performed in parallel, significantly improving the response speed and user experience of continuous image capture.

[0158] Through the above technical solution, electronic devices can complete hardware encoding of high-resolution images at millisecond speeds, representing an order-of-magnitude improvement in encoding efficiency compared to the 3.5 to 4 seconds required for software encoding in existing technologies. Simultaneously, the platform abstraction layer design allows the same encoding logic to seamlessly adapt to different chip platforms such as MTK, Unisoc, and Qualcomm, significantly reducing development and maintenance costs. In continuous high-speed shooting scenarios, this solution maintains stable low-latency performance, providing users with a smooth photography experience.

[0159] As a specific implementation, when a user triggers a photo-taking operation, the instructions in the computer-readable storage medium guide the electronic device to execute a cross-platform YUV hardware encoding photo-taking method. This allows 40-megapixel YUV image data (approximately 23MB) to be efficiently transmitted to the hardware encoder in a zero-copy manner and quickly converted to JPEG format with hardware acceleration. The entire process avoids the data copying overhead between the application layer and the HAL service process, while shielding the underlying hardware differences through the platform abstraction layer, achieving true cross-platform compatibility. Ultimately, users can obtain instant photo-taking feedback, and the continuous shooting experience is significantly improved.

[0160] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A cross-platform YUV hardware encoding photography method, characterized in that, include: In the Android camera application layer, when a user operation command is detected to trigger a photo capture, the raw YUV image data generated by the camera hardware abstraction layer is obtained; The application layer uses a pre-established IPC channel between the abstract namespace Unix domain socket and the camera hardware abstraction layer service process, and combines file descriptor memfd shared memory with file descriptor passing mechanism to send the YUV image data to the camera hardware abstraction layer service process with zero copy. The camera hardware abstraction layer service process calls the platform hardware JPEG encoder through its internally configured platform abstraction layer. The platform hardware JPEG encoder encodes and converts the YUV image data to generate JPEG image data, and then sends the JPEG image data back to the application layer via memfd shared memory.

2. The cross-platform YUV hardware encoding photography method according to claim 1, characterized in that, Before the step of obtaining the raw YUV image data generated by the camera HAL layer when a user operation command is detected to trigger taking a picture in the Android camera application layer, the following steps are included: An inter-process communication channel for IPC is pre-established between the Android camera application layer and the camera hardware abstraction layer service process through Unix domain sockets in the abstract namespace. This channel is used to achieve zero-copy cross-process transmission of YUV image data by using memfd file descriptor shared memory combined with SCM_RIGHTS file descriptor passing mechanism. In addition, a platform abstraction layer is set up in advance within the camera hardware abstraction layer service process. The platform abstraction layer defines a unified buffer management and encoding operation interface. Hardware JPEG encoders of different chip platforms complete platform adaptation by implementing the unified interface. The platform abstraction layer abstracts the operations of the hardware JPEG encoder into the following function interfaces: buffer allocation, buffer release, buffer attribute query, cache synchronization, format conversion, encoding capability query, and encoding execution.

3. The cross-platform YUV hardware encoding photography method according to claim 1, characterized in that, Before the step of sending the YUV image data to the camera hardware abstraction layer service process with zero copy, the method further includes: performing image processing algorithm operations on the YUV image data at the application layer, wherein the image processing algorithm includes at least one of beauty filter and image enhancement. The image processing algorithm is executed asynchronously in the background, and the camera application layer continuously responds to the user's shooting operation during the algorithm processing. The algorithm processing and shooting process are executed in parallel. The IPC channel uses Unix domain sockets with abstract namespaces and transmits image data between the application layer and the camera hardware abstraction layer service process through the memfd shared memory mechanism.

4. The cross-platform YUV hardware encoding photography method according to claim 1, characterized in that, The camera hardware abstraction layer service process caches the allocated hardware encoding buffer through the Linux kernel memory management component buffer pool. Before the first encoding, the Linux kernel memory management component buffer is pre-allocated and cached through a warm-up protocol, and subsequent encoding requests directly reuse the cached buffer.

5. The cross-platform YUV hardware encoding photography method according to claim 1, characterized in that, It also includes the following steps: When rotation is required, the MTK platform uses a two-stage encoding method: first, the YUV data is rotated by the mobile display processor (MDP) hardware, and then encoded by the hardware JPEG encoder; the Unisoc and Qualcomm platforms complete the rotation encoding in one step using the rotation parameters natively supported by the hardware JPEG encoder. The encoding monitoring module is loaded into the camera HAL service process as a shared library. The monitoring thread is automatically started through the shared library constructor, without requiring modification of the camera HAL service source code.

6. The cross-platform YUV hardware encoding photography method according to claim 1, characterized in that, The IPC channel supports multiple data transfer modes: transferring YUV byte array data via shared memory using the file descriptor memfd; The camera's dma-buf file descriptor is passed directly via the SCM_RIGHTS file descriptor; and the Linux kernel memory management component buffer pool is pre-allocated in warm-up mode without performing actual coding.

7. The cross-platform YUV hardware encoding photography method according to claim 1, characterized in that, It also includes the following steps: A time-slot-based circular buffer management structure is constructed to divide the buffer pool into a fixed number of consecutive time slots. Each time slot is pre-bound to a complete ION buffer, forming a logical circular queue. Each time slot independently manages the occupancy status of its buffer. Continuously captured image frames occupy different time slots in chronological order, physically isolating the buffer resources of adjacent frames. A high-water mark and low-water mark monitoring mechanism is set up in the circular buffer; the encoding request density per unit time is monitored in real time. When a high-frequency continuous shooting mode is detected and the number of idle time slots is lower than the low-water mark, the extension protocol is automatically triggered to request an additional buffer group from the system ION buffer memory and inject it into the preparation queue; when the shooting frequency decreases and the idle time slots exceed the high-water mark, the excess buffer is gradually released to the low-water mark level to dynamically balance memory usage and concurrent processing capacity. Establish buffer inheritance relationships between adjacent shooting frames to achieve zero-release reuse; after the previous frame completes hardware encoding, the ION buffer group it occupies directly enters a hot standby state, and keeps the memory mapping valid through a reference counting mechanism, giving priority to the encoding request of the next frame; only when the buffer size is mismatched or the shooting interval exceeds the threshold is the actual release and reallocation triggered. A hardware encoder is introduced to complete the signal-driven buffer state machine. When the hardware encoder completes JPEG compression and triggers an interrupt, the buffer state of the corresponding time slot is transferred from the encoder to a reusable state that can be occupied by subsequent frames. The operation of sending JPEG data back to the application layer uses an independent temporary mapping buffer or directly establishes a read-only mapping in the original buffer for asynchronous transmission. By using a time-slotted ring structure to isolate multi-frame resources in the spatial dimension, a dual-water level mechanism to dynamically adapt to traffic fluctuations in the temporal dimension, an inter-frame inheritance protocol to eliminate unnecessary memory allocation overhead, and asynchronous state transition to compress resource occupation cycle, the four elements work together to allocate ready buffer resources for each frame encoding request with constant low latency in continuous high-speed shooting scenarios.

8. A cross-platform YUV hardware encoding and imaging system, characterized in that, The system includes: The application layer encoding interface module is used to provide a Java interface for hardware JPEG encoding in the Android camera application layer. It loads the local encoding library through JNI and establishes an IPC connection with the camera hardware abstraction layer service process. The IPC communication module is used to transfer image data between the application layer and the camera hardware abstraction layer service process via Unix domain sockets and shared memory. The platform abstraction layer module is set in the camera hardware abstraction layer service process. It is used to define a unified hardware encoding operation interface and adapt to the hardware JPEG encoder implementation of different chip platforms. The hardware encoding execution module is used to call the platform's hardware JPEG encoder to complete the YUV to JPEG encoding conversion; The algorithm pipeline management module is used to manage image processing algorithm nodes at the application layer, supporting asynchronous execution and overlay processing of algorithms. The algorithm pipeline management module works with the application layer encoding interface module to trigger hardware encoding after the algorithm processing is completed. The cross-platform YUV hardware encoding photography system uses the cross-platform YUV hardware encoding photography method described in any one of claims 1-7 to perform photography processing.

9. A smart terminal, characterized in that, It includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, wherein the one or more programs include methods for performing any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-7.

Images