A camera data processing method and system based on a dual-stream parallel architecture

Abstract

This application discloses a camera data processing method and system based on a dual-stream parallel architecture. It utilizes the Android Camera2 API to construct a dual-stream parallel architecture containing both display and data streams, decoupling display and data processing. A producer-consumer model is used to perform non-blocking processing on the data stream. A YUV format image reader is configured, and stable data transmission is ensured through data acquisition, deep copying, release, and distribution processes. Simultaneously, asynchronous event notifications are used to achieve seamless snapshots, decoupling the photo request from the preview stream and eliminating screen freezes. This application enables full-frame-rate preview, seamless photo capture, and real-time data distribution even on low-configuration terminals, reducing development complexity, adapting to the needs of scenarios such as financial terminals, and improving business reliability and user experience.

Description

Technical Field

[0001] This application relates to the fields of digital image processing and embedded systems technology, specifically to a camera data processing method and system based on a dual-stream parallel architecture. Background Technology

[0002] With the trend towards mobile and intelligent financial services, portable smart terminals play a crucial role in scenarios such as bank outreach marketing and remote counter services. Their built-in cameras frequently need to perform functions such as QR code scanning, ID card OCR recognition, and facial recognition, which places stringent demands on the stability, smoothness, and response speed of the camera driver. The industry primarily uses the Android operating system's standard camera application programming interface (API), namely the Camera2 API, for development. However, on resource-constrained low-to-mid-range hardware platforms, such as the Rockchip (RK) and MediaTek (MTK) main controllers commonly used in portable financial terminals, existing technologies suffer from one or more of the following significant drawbacks:

[0003] (1) The real-time data stream channel is missing and unstable.

[0004] The Camera2 API does not provide a direct and stable raw preview frame callback interface. This requires developers to design complex data transfer and thread synchronization mechanisms, increasing the complexity and risk of software implementation. Improper design can easily lead to system-level instability issues such as camera data stream blockage or memory overflow.

[0005] (2) Common preview performance bottlenecks

[0006] Traditional camera architecture couples the "display" of an image with the potential "processing" tasks into a single data stream. On low-performance hardware, any additional data processing (such as format conversion or image analysis), or even an inappropriate threading model, will directly preempt the computing resources used for image rendering, causing frame drops, stuttering, or even freezing in the preview, severely impacting the success rate and user experience of all services that rely on real-time visual interaction (such as barcode scanning for autofocus).

[0007] (3) The dilemma of acquiring real-time data streams

[0008] When the business needs to obtain preview frame data, the above-mentioned performance bottleneck becomes particularly prominent, creating a technical dilemma:

[0009] JPEG performance pitfall: If the data Surface format is set to JPEG for the convenience of upper-layer use, the real-time compression of the hardware image signal processor (ISP) will generate a huge computational burden, resulting in severe frame drops and stuttering in the preview, which directly affects the success rate of scanning, recognition and other services and the user experience.

[0010] YUV Usability Pitfalls: If the format is set to the original format such as YUV_420_888 to ensure smooth preview, the upper-layer application cannot use it directly and must perform a format conversion. If this conversion is performed at the Android application layer, the CPU consumption will be huge, which will also cause lag; and improper data processing can easily block the camera data stream.

[0011] (4) Disjointed photo-taking experience

[0012] The system's standard API photo capture process requires interrupting continuous preview requests, initiating an independent still image capture, and then resuming preview after completion. This "interrupt-capture-resume" process causes the preview screen to freeze or go black briefly at the moment of taking the photo, disrupting the continuity of operation and resulting in a poor user experience.

[0013] Therefore, there is an urgent need in this field for an innovative technical solution that can systematically solve the common problems of preview performance bottlenecks, data acquisition difficulties, and photo capture interruptions from the architectural level within the native API framework, with low complexity and high stability.

[0014] Explanation of relevant English terminology:

[0015] JPEG stands for Joint Photographic Experts Group, an image compression standard commonly used as a still image format.

[0016] Surface, translated as "Surface" in Chinese, is the carrier used for image rendering and data transmission in the Android system.

[0017] YUV is a color encoding format widely used in video and image processing. Its core feature is that it stores luminance and chrominance information separately. Summary of the Invention

[0018] The purpose of this application is to provide a camera data processing method and system based on a dual-stream parallel architecture, which can completely decouple display from data processing, thereby improving preview smoothness and data processing stability.

[0019] In a first aspect, this application provides a camera data processing method based on a dual-stream parallel architecture, comprising:

[0020] Construct a dual-stream parallel data processing architecture: Call the Android Camera2 API's create capture session method to create a camera capture session containing at least two functionally independent output targets. The output targets include a display stream target S_display for image rendering and a data stream target S_data for data processing. The display stream target S_display is a surface associated with a UI component, constituting a hardware-accelerated display stream Stream_Display. The data stream target S_data is a surface managed by an image reader, constituting a data stream Stream_Data.

[0021] Perform non-blocking data processing: Process the data stream Stream_Data based on the producer-consumer model, including the configuration phase, production phase, and consumption phase;

[0022] Configuration phase: Create an image reader instance and configure it to YUV format (such as YUV_420_888 format), create a thread-safe bounded data buffer Q_data, start an independent background consumer thread T_process, set a listener for the image reader and schedule the listener callback to a dedicated camera data processing thread;

[0023] Production phase: The listener callback acts as producer P, performing image data acquisition, deep copying, and release according to the non-blocking operation protocol, and storing the image data copy into Q_data; among them, the release operation is to immediately and unconditionally return the hardware resources occupied by the image object to the internal circular queue of the image reader after the data deep copy is completed, so that it can receive the next frame of data.

[0024] Consumption phase: The background consumer thread T_process retrieves an image data packet from Q_data in a blocking manner; and distributes the corresponding image data to the upper-layer business through the callback interface.

[0025] Traditional solutions couple "image rendering" and "data processing" into a single data stream, causing data processing (such as format conversion and recognition analysis) to compete for rendering resources, resulting in preview stuttering and frame drops. The solution described in this application addresses this problem at its root through a dual-stream parallel architecture (separating the display stream Stream_Display and the data stream Stream_Data). The display stream S_display, associated with the UI component's Surface, is dedicated to image rendering using hardware acceleration and does not participate in any data processing tasks, ensuring that rendering resources are not occupied. The data stream S_data, managed by the image reader, independently handles data processing needs; all format conversions and distribution operations are completed within this stream, without interfering with the real-time performance of the display stream. Even on low-configuration financial terminals such as Rockchip (RK) and MediaTek (MTK), smooth preview at full frame rate can be achieved, completely resolving the preview performance bottleneck of traditional solutions and ensuring the success rate of services relying on real-time visual interaction, such as barcode scanning and facial recognition.

[0026] Traditional solutions rely on self-designed data transmission and thread synchronization mechanisms, which are prone to data flow blocking, memory overflow, or data loss. The solution described in this application achieves stable data transmission through a producer-consumer model and a strict non-blocking operation protocol. The listener callback acts as the producer, executing a fixed process of "data acquisition → deep copy → release," with the release operation being immediate and unconditional—hardware resources are returned immediately after the deep copy is completed, ensuring the image reader can continuously receive the next frame of data and avoiding data flow blocking. A thread-safe bounded data buffer, Q_data, balances production and consumption speeds, preventing memory overflow caused by data accumulation and avoiding invalid occupation when there is no data. The listener callback is scheduled to a dedicated camera data processing thread (such as HandlerThread) to achieve serial processing, eliminating the risks of resource contention, data corruption, or program crashes caused by multi-threaded concurrency. This provides a stable, frame-free real-time image data source for upper-layer applications (QR code scanning, OCR recognition, face recognition), solving the pain point of the Camera2 API lacking a direct and stable raw preview frame callback interface, and improving the reliability and functional scalability of financial services.

[0027] Traditional solutions face a dilemma: JPEG performance pitfalls and YUV usability pitfalls. While JPEG is convenient for upper-layer use, real-time compression by the hardware ISP causes preview stuttering; YUV, while smooth, suffers from time-consuming application-layer conversion. This application's solution configures a YUV format (e.g., YUV_420_888) image reader, directly circumventing these pitfalls. It eliminates the need for hardware-based real-time JPEG compression, reducing hardware computational burden and ensuring smooth preview. Subsequent data processing (such as format conversion) is completed through an independent data stream and background thread, without affecting the display stream. Simultaneously, the YUV format preserves original image details, laying the foundation for subsequent high-quality processing (such as ID card recognition). Thus, it balances preview smoothness and data integrity, resolving the technical dilemma of traditional solutions where "smoothness and usability are mutually exclusive."

[0028] Traditional solutions require the design of complex thread synchronization and data transfer mechanisms, and may rely on underlying technologies such as OpenGLES, resulting in high development barriers and poor compatibility. The solution described in this application is implemented based on the Android native Camera2 API, eliminating the need for additional underlying technologies. The standardized architecture design (dual-stream separation, producer-consumer model) reduces the potential risks of custom logic and facilitates subsequent maintenance. It is not dependent on a specific hardware platform and can be adapted to various Android financial terminals, providing a unified solution for devices with different configurations. This reduces development and maintenance costs, improves system compatibility and stability, and meets the deployment requirements of financial terminals for "low complexity and high reliability."

[0029] In one possible implementation, the data deep copy step in the production stage specifically includes: obtaining a data plane array through the getPlanes() method of the image object, traversing the array and obtaining the byte buffer, line stride and pixel stride parameters of each plane, and copying and splicing the valid image data of each plane to a new contiguous memory area according to the parameters to generate a complete and independent image data copy.

[0030] This scheme can generate an independent data copy decoupled from the original image object, avoiding data loss after the original image object is released, while eliminating memory fragmentation in the data plane and improving the reading efficiency of subsequent data processing.

[0031] In one possible implementation, the method further includes: performing a seamless snapshot based on asynchronous event notification: achieving uninterrupted snapshot taking through a producer-consumer model of photo request events, including event definition, event production, event consumption and data sampling, and background photo processing;

[0032] Event definition: Define a photo capture request event Event_capture, and create a thread-safe event channel C_event with a capacity of 1 to transmit the Event_capture;

[0033] Event production: Responds to the takePicture() API call of the upper-layer application. The API calling thread publishes the photo request event to C_event and returns immediately.

[0034] Event consumption and data sampling: After the data acquisition operation in the production stage of non-blocking data processing, check and attempt to consume the photo request event from C_event. If the consumption is successful, perform an additional deep copy on the current image data to generate a photo data copy. The photo data copy is then dispatched to the photo processing thread pool. Background photo processing: In the photo processing thread, the photo data copy is encoded into JPEG format with the highest quality configuration, the image is rotated according to the orientation of the device sensor, and finally saved as a photo file.

[0035] This solution decouples photo-taking requests from preview stream processing through an asynchronous event mechanism, preventing photo-taking operations from blocking the preview data stream. Simultaneously, the event channel with a capacity of 1 prevents multiple photo-taking requests from accumulating, reducing hardware resource usage and achieving a seamless snapshot effect with zero lag in the preview. The event channel C_event has a capacity of 1 to avoid resource waste or preview stream interference caused by multiple photo-taking requests accumulating, and its thread-safety feature ensures mutual exclusion of cross-thread operations through an internal locking mechanism.

[0036] In one possible implementation, the process of encoding the photographed data copy into JPEG format at the highest quality is implemented using the libjpeg-turbo library, with the encoding quality parameter set to 100%.

[0037] Compared to the native encoding library of the Android system, this encoding method can improve the encoding speed, while retaining 100% of the original image details with high quality parameters, meeting the high requirements for image clarity in scenarios such as ID card recognition and QR code scanning in financial terminals.

[0038] In one possible implementation, the dedicated camera data processing thread is a serial processing thread implemented based on the Android processor thread HandlerThread.

[0039] This thread can serialize the execution of the image reader's listener callback events, avoiding resource contention caused by multi-threaded concurrent processing of image data, eliminating the risk of data corruption and program crashes, and ensuring that image data is output in the order of acquisition, thus improving the orderliness of data processing.

[0040] In one possible implementation, the distribution of the corresponding image data to the upper-layer business via the callback interface can be achieved by first calling the Java Native Interface (JNI, used for interaction between Java and native C / C++ code) to efficiently convert the corresponding YUV format image data into a format that is easy for the upper layer to use, and further compressing it into JPEG format, and then distributing the JPEG format image data to the upper-layer business via the callback interface.

[0041] Low-level format conversion implemented through JNI can reduce data processing time, lower processing latency of consumer threads, and improve the response speed of upper-layer business logic compared to application-layer Java code conversion.

[0042] Secondly, this application provides a camera data processing system based on a dual-stream parallel architecture, including: a memory and a processor;

[0043] The memory is used to store computer programs;

[0044] The processor is used to invoke the computer program to execute the method described above.

[0045] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the method described above.

[0046] Fourthly, this application provides a computer program product, including a computer program that, when run on an electronic device, causes the electronic device to perform the method described above.

[0047] The specific implementation methods of the second to fourth aspects of this application can refer to the implementation methods of the first aspect, and will not be elaborated here.

[0048] Compared with the prior art, this application has the following significant advantages:

[0049] (1) Extreme preview smoothness: Through the dual-stream parallel architecture, the display and data processing are completely decoupled, and the YUV raw data format is used to avoid hardware coding bottlenecks, so that full frame rate smooth preview can be achieved even on low-configuration financial terminals.

[0050] (2) Seamless photo-taking experience: The original "asynchronous event-data sampling" seamless snapshot mechanism completely decouples the photo-taking operation from the preview stream, completely eliminating the problem of image freezing in traditional photo-taking and providing a smooth and seamless user experience.

[0051] (3) Stable and reliable real-time data channel: The producer-consumer model is designed with clear responsibilities and a strict resource management protocol is established, providing a stable, reliable, and frame-free real-time image data source for upper-layer applications (camera scanning, face recognition, OCR, etc.), which greatly improves the reliability and functional scalability of financial business.

[0052] (4) Significantly reduce development and maintenance complexity: The solution is implemented entirely within the Android native API framework, without the need to introduce more complex graphics pipelines such as OpenGLES at a lower level. The code logic is clearer, the development threshold is lower, and the system compatibility and stability are higher.

[0053] (5) Universality and high compatibility: This method does not depend on a specific hardware platform and can run universally on various Android financial terminals, providing a unified high-performance camera solution for devices of different forms and configurations. Attached Figure Description

[0054] Figure 1 A schematic diagram of a existing single-data-stream coupled processing architecture. Among them, Figure 1 (a) shows the photo-taking process. Figure 1 (b) shows the preview process. This diagram illustrates the single, linear data stream processing mode of traditional camera solutions. This architecture couples the "screen rendering" and "data processing" tasks together, which is the common root cause of the two core defects: preview performance bottleneck (processing tasks preempt rendering resources) and photo capture interruption (the capture task must interrupt the preview stream).

[0055] Figure 2 This application includes an architectural block diagram of a method according to one embodiment. Figure 1 In stark contrast, this demonstrates the core architecture of one embodiment of this application, which uses CameraCaptureSession to output data to two independent targets, S_display and S_data, in parallel, fundamentally solving the single-stream coupling problem.

[0056] Figure 3 This application presents a flowchart of a non-blocking data processing pipeline according to an embodiment of the present application. The diagram details an efficient processing method for the S_data data stream according to an embodiment of the present application, specifically demonstrating the interaction between the producer P (OnImageAvailableListener), the consumer C (T_process), and the buffer Q (Q_data), as well as the key steps of the Op_P operation sequence that ensures the continuity of the data stream.

[0057] Figure 4This application presents an embodiment of a non-disruptive snapshot flowchart. The diagram illustrates how an embodiment of this application achieves non-disruptive snapshot taking through an asynchronous event mechanism. Specifically, it depicts how the API call thread publishes a snapshot event, and how the camera data processing thread handles the snapshot without interrupting the main data stream (e.g., ...). Figure 3 In the case shown, the event is consumed and data is sampled. Detailed Implementation

[0058] To enable those skilled in the art to better understand the present application, the technical solution of the present application will be further described in detail below with reference to the embodiments and accompanying drawings.

[0059] Specific embodiments according to this application will now be described with reference to the accompanying drawings.

[0060] Example 1:

[0061] This application proposes a camera data processing method based on a dual-stream parallel architecture. Its core lies in constructing a dual-stream parallel processing architecture that decouples display and data functions, and then employing non-blocking data processing operations on top of this architecture. The method of this application is described in detail below.

[0062] When creating a camera capture session, construct a parallel data stream that contains at least two functionally independent output targets.

[0063] This session can be formally defined as:

[0064] Session_Dual=CreateCaptureSession({S_display, S_data}) (Formula 1);

[0065] in:

[0066] Session_Dual: Dual-stream camera capture session;

[0067] Createcapturesession(): API method for creating a capture session; Createcapturesession() is the standard method in the Android Camera2 API for creating a camera capture session. Its function is to establish a connection with the underlying camera hardware and specify the "data output target list" of the session. The raw image data subsequently captured by the camera will be automatically distributed to the corresponding output stream according to the target configured by this method.

[0068] The curly braces contain two functionally independent data output targets, which together form the core of the two-stream parallel architecture, as detailed below:

[0069] S_display: Display stream target. A Surface used for user interface (UI) components (such as TextureView) forms a hardware-accelerated display stream (Stream_Display), specifically responsible for smooth screen rendering.

[0070] S_data: Data stream target. A Surface managed by an ImageReader (the core class in Android used to obtain raw camera data) constitutes the data stream (Stream_Data), serving as the sole, raw data source for all subsequent data processing (including preview frame callbacks and taking photos).

[0071] This architecture fundamentally decouples "display" from "processing," laying the foundation for subsequent efficient algorithm implementation.

[0072] Non-blocking data processing operations.

[0073] The core idea of ​​non-blocking data processing is to establish a producer-consumer model, the logic of which can be abstracted as follows:

[0074] C←Q_data←P(Stream_Data) (Formula 2);

[0075] In this context, P is the producer, responsible for extracting raw data from the data stream (Stream_Data) and sending it to the buffer; Q_data is the buffer; and C is the consumer.

[0076] Specifically, the non-blocking data processing provided in this application, applied to the data stream (Stream_Data), includes the following steps:

[0077] Step S1, Configuration Phase:

[0078] Step S1.1: Create an ImageReader instance and configure its image format to YUV format, such as YUV_420_888 format, i.e., ImageFormat.YUV_420_888, to avoid the hardware JPEG encoding bottleneck.

[0079] Step S1.2: Create a thread-safe, bounded data buffer Q_data;

[0080] For example, a BlockingQueue instance. The queue capacity is set to 5-10 data packets; this capacity setting can ensure data caching capacity while avoiding memory overflow caused by excessive data accumulation in the buffer, achieving a dynamic balance between production speed and consumption speed, and further optimizing the smoothness of non-blocking data processing.

[0081] Step S1.3: Create and start a separate background consumer thread (T_process) to consume data in Q_data.

[0082] Step S1.4: Set up a listener for the image reader and schedule its callbacks to a dedicated camera data processing thread; for example, implement it through a processor thread (HandlerThread, a background thread class with a message queue in the Android system to ensure serial processing) to ensure that all callback events are processed serially.

[0083] Step S2, Production Stage:

[0084] In the OnImageAvailableListener callback, it acts as a producer (P) and strictly adheres to the Non-Blocking Operation Protocol.

[0085] The image listener (OnImageAvailableListener) is a callback interface for ImageReader used to detect new image data. This callback is scheduled to a dedicated background thread with a message queue, such as through Android's HandlerThread, to ensure the serialization of all callback events.

[0086] The non-blocking operation protocol can be defined as follows:

[0087] Op_P={Acquire,DeepCopy,Release} (Formula 3);

[0088] In the formula, Op_P represents the set of operations of producer P, Acquire within curly braces means acquisition, DeepCopy means deep copy, and Release means release. These three are the three core steps of the protocol, which together constitute a complete data extraction link.

[0089] In some embodiments, the specific operation is as follows:

[0090] S2.1 Image Data Acquisition Steps: Call the method in the image reader to obtain the next frame image (such as the reader.acquireNextImage() method, which is a method of ImageReader used to obtain the original image object) to obtain the image object.

[0091] S2.2, Deep Copy of Image Data: This step performs an efficient deep copy operation in memory for the preview callback. This step includes: obtaining an array of all data planes using a data plane acquisition method (such as `image.getPlanes()`, a method of the `Image` object used to obtain the component planes of the YUV data); iterating through this array, and for each plane, obtaining its ByteBuffer, line stride, and pixel stride; and precisely copying and stitching the valid image data from each plane into a new, contiguous memory region (such as a new ByteBuffer or ByteArray) based on these parameters, thereby generating a complete and independent copy of the image data.

[0092] Specifically, the row stride is obtained through the getRowStride() method, and the pixel stride is obtained through the getPixelStride() method.

[0093] S2.3, Image Data Release Step: After the deep copy of the data is completed, the image close method image.close() must be called immediately and unconditionally to return the hardware resources occupied by the image object to the internal circular queue of the image reader so that it can receive the next frame of data;

[0094] This operation is crucial to ensuring that the data stream is not blocked. By doing so, the hardware resources occupied by the image object are released in a timely manner, allowing the ImageReader to reuse the resources and continuously receive new image data, thus avoiding data stream blockage.

[0095] S2.4. Encapsulate the image data copy generated in step S2.2 and store it in the data buffer Q_data.

[0096] Step S3, Consumption Stage:

[0097] In the background consumer thread (C), the following steps are executed in a loop: a data packet is retrieved from the data buffer Q_data in a blocking manner; and the corresponding image data is distributed to the upper-layer business through the callback interface.

[0098] The method of distributing the corresponding image data to the upper-layer service through the callback interface can first call JNI to efficiently convert the corresponding YUV format image data into a format that is easy for the upper layer to use (such as NV21), and further compress it into JPEG format, and then distribute the JPEG format image data to the upper-layer service through the callback interface.

[0099] NV21 is a YUV series image data format.

[0100] The specific implementation method for retrieving an image data packet from the data buffer Q_data in a blocking manner is as follows:

[0101] Q_data is a blocking queue (BlockingQueue). Calling its blocking data retrieval method `take()` involves the following process: After thread T_process calls `take()`, if Q_data contains data, it directly retrieves the first data packet; if Q_data is empty, T_process enters a blocked state until P stores a new data packet, at which point the `take()` method returns, and the thread resumes execution. Therefore, the consumer thread does not waste CPU resources, and the producer thread is not blocked due to slow consumer processing, ensuring both smooth display and stable data flow.

[0102] In some embodiments, a seamless snapshot operation based on asynchronous event notification is performed.

[0103] This operation decouples taking a photo from a lightweight asynchronous event, and its logic can be abstractly expressed as follows:

[0104] Image_photo = Process(Sample(Stream_Data, Consume(Event_capture)))(Formula 4);

[0105] In the formula, "Image_photo" is the target photo data finally generated by this method; the outermost function Process() is the background processing function for photo data; the middle layer function sample() is the data stream sampling function, which receives two parameters: one parameter is Stream_Data, which is the data source for sampling, i.e., the data stream in the dual-stream architecture of this application, providing raw image data in YUV_420_888 format, which is the basis for sampling; the other parameter is Consume(Event_capture), which is the sampling trigger condition. Sample() only performs sampling when the photo-taking event is successfully consumed; otherwise, no additional operations are performed on Stream_Data to avoid invalid sampling consuming resources. The inner function Consume(Event_capture) is the photo-taking event consumption function, which refers to the operation of obtaining the photo-taking request event from the event channel; Event_capture is the photo-taking request event defined in this application.

[0106] This logic is implemented through a producer-consumer model based on a "photo request event".

[0107] Specifically, a seamless snapshot based on asynchronous event notification may include the following steps:

[0108] Step A1, Event Definition: Define a photo capture request event Event_capture, and create a thread-safe event channel C_event with a capacity of 1 to pass the Event_capture.

[0109] C_event's design parameters include a capacity of 1, thread safety to adapt to business requirements for seamless snapshots, and each feature corresponds to a specific technical pain point solution.

[0110] Setting the capacity to 1, meaning that a single photo request event is temporarily stored, avoids the chaos caused by the accumulation of multiple photo requests. In photo-taking scenarios of financial terminals (such as ID card OCR and facial recognition), users typically need to take a single, accurate photo. If multiple photo requests are triggered consecutively within a short period (such as accidental button presses), and if the capacity of C_event is unlimited, multiple photo request events will accumulate, leading to multiple data sampling and photo generation, resulting in wasted resources (such as generating multiple identical photos), and even interfering with the preview stream. By setting the capacity to 1, when there is already an unconsumed photo request event in C_event, subsequent new photo request events will be temporarily blocked or directly discarded (the specific logic can be designed as needed), ensuring that only one photo request is processed at a time, guaranteeing the uniqueness and accuracy of the photo-taking operation, and meeting the controllable requirements of financial business operations.

[0111] Thread safety, meaning safe communication across threads. This is because `C_event` involves operations from two different threads:

[0112] Producer thread: This is usually the UI thread, such as when a user clicks the camera button to trigger the camera application interface, i.e., takePicture() APl, which publishes a camera request event to C_event;

[0113] Consumer thread: The camera data processing thread, which consumes photo request events from C_event.

[0114] If thread safety is not guaranteed, if two threads operate on C_event simultaneously, such as when the producer writes while the consumer reads, it will lead to data races, causing event loss, duplicate consumption, or program crashes.

[0115] The thread-safe data structure uses a locking mechanism to ensure that enqueue (publishing events) and dequeue (consuming events) operations are mutually exclusive, thus avoiding cross-thread operation conflicts.

[0116] Step A2, Event Production (Publish): Responding to the upper-layer application's photo application interface, i.e., the takePicture() API call, the API calling thread publishes (Produces) only one photo request event to the event channel C_event, and then returns immediately without causing any blocking.

[0117] Step A3, Event Consumption and Data Sampling: After the image data acquisition operation in the production phase of non-blocking data processing (Step S2), check and attempt to consume (Consume) a photo request event from the event channel C_event. If the event is successfully consumed, perform an additional deep copy on the currently acquired image data to generate a photo data copy, and dispatch the photo data copy to an independent photo processing thread pool. Regardless of whether the photo event has been consumed, continue to perform subsequent deep copy and release operations to ensure the absolute continuity of the preview data stream.

[0118] In this application, a copy of the photo data is dispatched to a photo processing thread pool to perform subsequent encoding, storage, and uploading operations. This thread pool is implemented based on Android's ThreadPoolExecutor. In some embodiments, the configuration parameters and adaptation rules are as follows:

[0119] Basic fixed parameters: core thread count is set to 2, maximum thread count is set to 4, idle thread lifespan is set to 68 seconds, and the work queue adopts a linked list blocking queue (LinkedBlockingQueue) with a capacity of 10. This configuration is compatible with mainstream quad-core / octa-core low-power processors in financial terminals, ensuring parallelism in photo processing while avoiding excessive thread contention for CPU resources in the camera preview stream.

[0120] Dynamic adaptation rules: Obtain the remaining memory and CPU load of the terminal through the Android ActivityManager. If the remaining memory is detected to be less than 200MB or the CPU load is continuously higher than 80%, the maximum number of threads will be dynamically reduced to 2 and the work queue capacity will be reduced to 5 to prioritize the core business of the preview stream and data stream. When the resources are restored, the basic parameters will be automatically called back.

[0121] Rejection strategy: The DiscardoldestPollicy strategy is adopted, which means that when the work queue is full, the oldest photo processing task is discarded. This is adapted to the business requirement of prioritizing the latest transaction photos in financial terminals and avoids preview lag caused by task backlog.

[0122] The above configuration has the following advantages:

[0123] (1) Scenario adaptability: Financial terminals have low configuration and high stability as their core requirements. Appropriate thread pool parameters are set to prevent resource exhaustion;

[0124] (2) Robustness guarantee: Add dynamic adaptation rules to solve the resource difference problem of financial terminals with different configurations, which can be applied to low-configuration terminals;

[0125] (3) Business priority: The rejection strategy can be tailored to the timeliness requirements of photos in financial transaction scenarios.

[0126] Step A4, Background Photo Processing: In a dedicated photo processing thread, a separate, high-quality processing flow is executed on the photo data copy obtained in Step A3.

[0127] A4.1 JPEG Encoding: Encode this YUV data into JPEG format using the highest quality settings (e.g., quality parameter 100). This can be achieved using high-performance JNI libraries such as libjpeg-turbo. libjpeg-turbo is an open-source, high-performance JPEG image encoding / decoding library.

[0128] A4.2 Image Rotation: Based on the sensor orientation obtained during the photo capture, the image data is precisely rotated to ensure the correct orientation of the final photo.

[0129] A4.3 Save the processed JPEG data stream as a photo file. It is worth emphasizing that although the data used for taking the picture and the frame data used for video callback originate from the same image object at the same moment, they enter two separate processing pipelines, thus ensuring both low latency in the preview stream and the highest image quality in the final photo.

[0130] It should be understood that the above step numbers, such as S1~S3 and A1~A4, are only used to distinguish and facilitate the expression of different steps, and do not necessarily constitute a restriction on the execution order between the steps.

[0131] This application utilizes the Android Camera2 API to construct a dual-stream parallel architecture containing both display and data streams, decoupling display and data processing. Two core operations—non-blocking data processing and asynchronous event snapshots—systematically solve three major challenges: previewing, capturing images, and data callbacks. Specifically, a producer-consumer model is used to perform non-blocking processing on the data stream. A YUV format image reader is configured, and stable data transmission is ensured through data acquisition, deep copying, release, and distribution processes. Simultaneously, asynchronous event notifications enable seamless snapshots, decoupling the image capture request from the preview stream and eliminating screen freezes. This solution does not rely on complex underlying technologies and can be applied to Android-based smart terminals (especially financial portable terminals). Even on low-configuration terminals, it can achieve high-performance image preview, seamless image capture, and real-time data distribution, reducing development complexity, adapting to the needs of financial terminals and other scenarios, and improving business reliability and user experience.

[0132] Example 2:

[0133] This embodiment provides a camera data processing system based on a dual-stream parallel architecture, including: a memory and a processor;

[0134] The memory is used to store computer programs;

[0135] The processor is configured to invoke the computer program to execute the method as described in Embodiment 1.

[0136] The system is applied to electronic devices such as financial smart terminals.

[0137] In some embodiments, the system further includes a camera module.

[0138] In some embodiments, the computer program specifically includes the following functional modules:

[0139] Session Management Module: Responsible for calling the createCaptureSessionAPI to create a camera capture session containing a display stream target S_display and a data stream target S_data, thus implementing a dual-stream parallel architecture.

[0140] Data processing module: Responsible for implementing the producer-consumer model, executing functions including YUV format configuration, non-blocking data production protocols (Acquire, DeepCopy, Release), and background data consumption and transformation.

[0141] Event notification module: responsible for managing the event channel for asynchronous photo-taking events, and implementing the logic for event production and consumption, and seamlessly sampling photo data from the real-time data stream based on the event consumption results.

[0142] `createCaptureSession` is a core operation in the Android system's Camera2 API. Essentially, it establishes a connection with the underlying camera hardware and defines the output targets and processing rules for camera data. It serves as the "basic session entry point" for all camera functions (preview, photo capture, video recording). In the Camera2 API architecture, camera hardware resources (such as image sensors, ISP processors, and data buses) are exclusive and require unified scheduling—applications cannot directly manipulate the hardware and must use the camera capture session (CameraCaptureSession) as an intermediate layer to connect application requirements with hardware capabilities. The core function of `createCaptureSession` is to "create and initialize this intermediate layer session," explicitly telling the camera hardware: "Which targets (such as screen display, data processing) should the raw image data you subsequently acquire be output to, and what rules should be used for processing (such as format and resolution)?"

[0143] Example 3:

[0144] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is run on an electronic device, it causes the electronic device to perform the method described in Embodiment 1.

[0145] Example 4:

[0146] This embodiment provides a computer program product, including a computer program that, when run on an electronic device, causes the electronic device to perform the method described in Embodiment 1.

[0147] The specific implementation of the system, electronic device, computer-readable storage medium, and computer program product provided in this application can be referred to the specific embodiments of the above methods, and will not be repeated here.

[0148] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0149] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing camera data based on a dual-stream parallel architecture, the method comprising: include: Construct a dual-stream parallel data processing architecture: Call the Android Camera2 API's create capture session method to create a camera capture session containing at least two functionally independent output targets. The output targets include a display stream target S_display for image rendering and a data stream target S_data for data processing. The display stream target S_display is a surface associated with UI components, constituting a hardware-accelerated display stream Stream_Display. The data stream target S_data is a surface managed by the image reader, constituting a data stream Stream_Data. Perform non-blocking data processing: Process the data stream Stream_Data based on the producer-consumer model, including the configuration phase, production phase, and consumption phase; Configuration phase: Create an image reader instance and configure it to YUV format, create a thread-safe bounded data buffer Q_data, start an independent background consumer thread T_process, set a listener for the image reader and schedule the listener callback to a dedicated camera data processing thread; Production phase: The dedicated camera data processing thread serializes the listener callback events of the image reader; the listener callback, as producer P, performs image data acquisition, deep copy, release, and stores the image data copy in Q_data according to the non-blocking operation protocol; among them, the release operation is to immediately and unconditionally return the hardware resources occupied by the image object to the internal circular queue of the image reader after the data deep copy is completed, so that it can receive the next frame of data. Consumption phase: The background consumer thread T_process retrieves an image data packet from Q_data in a blocking manner; and distributes the corresponding image data to the upper-layer business through the callback interface.

2. The method of claim 1, wherein, The data deep copying step in the production stage specifically includes: obtaining a data plane array through the method provided by the image object, traversing the array and obtaining the byte buffer, line stride and pixel stride parameters of each plane, and copying and splicing the valid image data of each plane to a new contiguous memory area according to the parameters to generate a complete and independent image data copy.

3. The method of claim 1, wherein, The method also includes: performing seamless snapshots based on asynchronous event notifications: achieving uninterrupted snapshots through a producer-consumer model of photo request events, including event definition, event production, event consumption and data sampling, and background photo processing; Event definition: Define a photo capture request event Event_capture, and create a thread-safe event channel C_event with a capacity of 1 to transmit the Event_capture; Event production: Responds to the takePicture() API call of the upper layer application. The API calling thread publishes the photo request event to C_event and returns immediately. Event consumption and data sampling: After data acquisition operations in the production phase of non-blocking data processing, check and attempt to consume photo request events from C_event. If consumption is successful, perform an additional deep copy on the current image data to generate a photo data copy, and dispatch the photo data copy to the photo processing thread pool. Background photo processing: In the photo processing thread, the photo data copy is encoded into JPEG format at the highest quality, the image is rotated according to the device sensor orientation, and finally saved as a photo file.

4. The method of claim 3, wherein, The process of encoding the photographed data copy into JPEG format at the highest quality configuration is implemented using the libjpeg-turbo library, with the encoding quality parameter set to 100%.

5. The method according to any one of claims 1 to 4, characterized in that, The dedicated camera data processing thread is a serial processing thread implemented based on the Android processor thread HandlerThread.

6. The method of any one of claims 1-4, wherein, The process of distributing the corresponding image data to the upper-layer business via a callback interface first calls the Java Native Interface (JNI) to efficiently convert the YUV format image data into a format that is easy for the upper layer to use, and then further compresses it into JPEG format. Finally, the JPEG format image data is distributed to the upper-layer business via the callback interface.

7. A camera data processing system based on a dual-stream parallel architecture, characterized in that, include: Memory and processor; The memory is used to store computer programs; The processor is configured to invoke the computer program to perform the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1 to 6.

Images