Camera synchronization processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a camera synchronization processing method and device, electronic equipment and a storage medium, and the method comprises the steps: triggering at least two image sensors to synchronously collect original frames based on a synchronization pulse signal generated by a main control module; the original frame collected by each image sensor is stored in a corresponding independent buffer area, the independent buffer area comprises a plurality of memory blocks, and the plurality of memory blocks are dynamically managed through a linked list structure; and on the basis of the same timestamp, obtaining the original frames from the independent buffer area corresponding to each image sensor for synthesis to obtain synthesized frames, and forming a synthesized video by a plurality of continuous synthesized frames. The synchronization precision is improved by taking physical layer time sequence alignment as a starting point and through a hardware-level time synchronization mechanism. Through an independent buffer area and a shared memory mapping area, memory blocks are managed in combination with a linked list structure, the number of times of full-frame copy transmission is reduced, and a graphic processor directly accesses a shared memory to realize zero-copy data transmission.

Description

Technical Field

[0001] This application relates to the field of image acquisition and processing, and in particular to a method, apparatus, electronic device and storage medium for synchronous processing of cameras. Background Technology

[0002] In multi-camera systems for mobile devices, dual-camera synchronous video acquisition and synthesis technology has become one of the core capabilities of smart terminals, and is widely used in scenarios such as 4K / 8K video recording, multi-view live streaming, AR / VR real-time image fusion, and industrial inspection.

[0003] Existing technologies obtain the timestamps of camera frame data through application programming interfaces (APIs) and perform frame alignment based on system clock interrupts. This approach relies on operating system scheduling and is significantly affected by background tasks, with timing errors typically ranging from 10 to 50 milliseconds, leading to significant image misalignment issues. Furthermore, the video compositing process often employs fixed-resolution encoding and full-frame copy transmission, resulting in high memory bandwidth consumption; the static resource allocation strategy cannot adapt to dynamic scene changes, easily causing power consumption spikes and performance fluctuations. Summary of the Invention

[0004] This application provides a camera synchronization processing method, apparatus, electronic device, and storage medium to improve synchronization accuracy, increase synthesis efficiency, and solve the problem of rigid resource allocation.

[0005] In a first aspect, embodiments of this application provide a camera synchronization processing method, the method comprising:

[0006] Based on the synchronization pulse signal generated by the main control module, at least two image sensors are triggered to synchronously acquire raw frames.

[0007] Each raw frame acquired by an image sensor is stored in a corresponding independent buffer. The independent buffer consists of multiple memory blocks, which are dynamically managed through a linked list structure.

[0008] Based on the same timestamp, original frames are obtained from the corresponding independent buffer of each image sensor and synthesized to obtain composite frames. Multiple consecutive composite frames constitute a composite video.

[0009] In one possible implementation, based on a synchronization pulse signal generated by the main control module, at least two image sensors are triggered to synchronously acquire raw frames, including:

[0010] The main control module's synchronous pulse generator generates a synchronous pulse signal with a fixed pulse width.

[0011] A fixed-width pulse signal is transmitted to the clock calibration circuitry of at least two image sensors to correct the local clock skew of each image sensor.

[0012] Based on the corrected local clock, at least two image sensors are triggered to acquire raw frames.

[0013] In one possible implementation, storing the raw frames acquired by each image sensor into a corresponding independent buffer includes:

[0014] Determine the output frame rate and data volume for each image sensor;

[0015] Adjust the capacity of the independent buffer corresponding to each image sensor based on the output frame rate and data volume of each image sensor.

[0016] The raw frames acquired by each image sensor are stored in an independent buffer with adjusted capacity.

[0017] In one possible implementation, based on the same timestamp, original frames are obtained from the corresponding independent buffers of each image sensor and synthesized to obtain a synthesized frame, including:

[0018] By directly accessing the frame data in the independent buffer corresponding to each image sensor through the shared memory mapping area of ​​the graphics processor, at least two raw frames with the same timestamp can be obtained.

[0019] Based on the graphics processor's preset mixing weight matrix, pixel-level fusion is performed on at least two original frames with the same timestamp to obtain a composite frame.

[0020] In one possible implementation, the method further includes:

[0021] Real-time acquisition of the fill rate of the independent buffers corresponding to at least two image sensors;

[0022] If the fill rate is less than a preset threshold, adjust the resolution and / or bit rate of at least two image sensors.

[0023] In one possible implementation, at least two image sensors include a primary image sensor and a secondary image sensor; adjusting the resolution and / or bitrate of the at least two image sensors includes:

[0024] Identify scene types based on synthesized video;

[0025] When the scene type is static, the resolution of the secondary image sensor is reduced and the compression ratio of the synthesized video is increased. The scene type is determined based on historical frames.

[0026] When the scene type is a dynamic scene, the bitrate allocation of at least two image sensors is automatically balanced, and the synthesized video is transmitted based on the bitrate allocation.

[0027] In one possible implementation, the method further includes:

[0028] Upon detecting that the user has selected a recording mode, the multi-level temperature control unit is activated. The multi-level temperature control unit is used to dynamically adjust the operating frequency of the main control module and / or the graphics processor based on feedback from the temperature sensor.

[0029] Secondly, embodiments of this application provide a camera synchronization processing device, the device comprising:

[0030] The synchronization control module is used to trigger at least two image sensors to synchronously acquire raw frames based on the synchronization pulse signal generated by the main control module.

[0031] The storage module is used to store the raw frames acquired by each image sensor into a corresponding independent buffer. The independent buffer includes multiple memory blocks, which are dynamically managed through a linked list structure.

[0032] The compositing module is used to acquire raw frames from the corresponding independent buffer of each image sensor based on the same timestamp and compose them to obtain composite frames. Multiple consecutive composite frames constitute a composite video.

[0033] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0034] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0035] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0036] The camera synchronization processing method, apparatus, electronic device, and storage medium provided in this application include: triggering at least two image sensors to synchronously acquire raw frames based on a synchronization pulse signal generated by a main control module; storing the raw frames acquired by each image sensor into a corresponding independent buffer, the independent buffer including multiple memory blocks, the multiple memory blocks being dynamically managed through a linked list structure; and based on the same timestamp, obtaining raw frames from the corresponding independent buffer of each image sensor for synthesis to obtain a synthesized frame, with multiple consecutive synthesized frames constituting a synthesized video. Starting with physical layer timing alignment, a hardware-level time synchronization mechanism is used to improve synchronization accuracy. By using independent buffers and shared memory mapping areas, combined with a linked list structure to manage memory blocks, the number of full-frame copy transmissions is reduced, and zero-copy data transfer is achieved through direct access to shared memory by the graphics processor. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] Figure 1 Flowchart of the camera synchronization processing method provided in this application Figure 1 ;

[0039] Figure 2 Flowchart of the camera synchronization processing method provided in this application Figure 2 ;

[0040] Figure 3 Flowchart of the camera synchronization processing method provided in this application Figure 3 ;

[0041] Figure 4 A schematic diagram of the camera synchronization processing device provided in this application;

[0042] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0043] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0045] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the amount of data for the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0046] CPU: Central Processing Unit;

[0047] GPU: Graphics Processing Unit;

[0048] AR: Augmented Reality;

[0049] VR: Virtual Reality;

[0050] CMOS: Complementary Metal Oxide Semiconductor;

[0051] RTC: Real-Time Clock.

[0052] In multi-camera systems for mobile devices, dual-camera synchronous video capture and synthesis technology has become one of the core capabilities of smart terminals, widely used in scenarios such as 4K / 8K video recording, multi-view live streaming, AR / VR real-time image fusion, and industrial inspection. For example, in 4K multi-camera recording scenarios, users need to simultaneously capture images from both the main viewpoint and wide-angle viewpoint using dual cameras, and then achieve seamless panoramic video output in post-production synthesis; in multi-view live streaming scenarios, broadcasters need to switch between different angles using dual cameras, or display multi-dimensional content through real-time image fusion; in AR / VR applications, dual cameras need to simultaneously capture the user's surrounding environment and virtual objects, generating an immersive experience through pixel-level fusion.

[0053] Existing technologies mainly employ the following solutions to address the issues of dual-camera synchronization and video compositing:

[0054] Software time synchronization solution: This solution obtains the timestamps of camera frame data through the system application programming interface and performs frame alignment based on system clock interrupts. This solution relies on operating system scheduling and is significantly affected by background tasks, with timing errors typically ranging from 10-50ms, leading to significant image misalignment issues.

[0055] Dual-channel independent encoding scheme: The video streams from the two cameras are encoded independently and then synthesized by software. This scheme has a simple process, but it requires multiple memory copies, such as from the camera sensor to the buffer, from the buffer to the encoder, and from the encoder to the synthesizer, which increases the synthesis latency and has a high memory bandwidth usage.

[0056] Fixed resolution strategy: The secondary camera always operates at the same resolution as the main camera to ensure image consistency. This solution wastes computing power in static scenes and cannot adapt to the dynamic adjustment requirements of resolution and bitrate in dynamic scenes.

[0057] Global performance mode: This mode locks the CPU / GPU to run at high frequencies to ensure video processing performance, but it causes a significant increase in SoC (System on Chip) temperature and a shortened battery life when the device is recording continuously.

[0058] In summary, existing technologies have significant shortcomings in synchronization accuracy, synthesis efficiency, and dynamic adaptability, leading to problems such as image misalignment, high latency, and high power consumption, making it difficult to meet market demands for high resolution, low latency, and long battery life. Furthermore, with the widespread adoption of 5G networks and increasing user requirements for real-time interactive experiences, dual-camera systems need to achieve microsecond-level timing alignment at the physical layer, optimize memory bandwidth usage at the data processing layer, and intelligently allocate computing resources in dynamic scenarios. This places higher demands on system architecture design. Current technical solutions rely on system clock interrupts to align frame data at the software layer, resulting in uncontrollable timing errors; the video synthesis stage uses fixed resolution and full-frame copy transmission, leading to high memory bandwidth consumption; and the static resource allocation strategy cannot adapt to dynamic scene changes, resulting in wasted computing power and a surge in power consumption. Therefore, there is an urgent need for a dual-camera video acquisition and synthesis system capable of hardware-level synchronization, dynamic buffer management, and adaptive resource allocation to meet performance requirements in complex scenarios.

[0059] To address the aforementioned technical issues, this application provides a camera synchronization processing method applicable to the synchronous processing of multiple cameras. Examples include scenarios such as dual-camera synchronous video recording on smart terminal devices, multi-view live streaming, and real-time AR / VR image fusion. Specifically, in 4K / 8K multi-camera recording, the user needs to simultaneously capture both the main viewpoint and wide-angle viewpoint images using the main and secondary cameras, and eliminate timing discrepancies during the compositing stage; in multi-view live streaming, the broadcaster needs to switch between or fuse different angle images using dual cameras to generate seamless live content in real time; in AR / VR applications, dual cameras need to simultaneously capture the user's environment and virtual objects, generating an immersive experience through pixel-level fusion.

[0060] The camera synchronization processing method provided in this application starts with physical layer timing alignment and improves synchronization accuracy through a hardware-level time synchronization mechanism. It reduces the number of full-frame copy transfers by using independent buffers and shared memory mapping areas, combined with a linked list structure to manage memory blocks, and achieves zero-copy data transfer by allowing the graphics processor to directly access shared memory.

[0061] In one embodiment, such as Figure 1 As shown, the camera synchronization processing method includes:

[0062] Step 101: Based on the synchronization pulse signal generated by the main control module, trigger at least two image sensors to synchronously acquire raw frames;

[0063] Step 102: Store the raw frames acquired by each image sensor into a corresponding independent buffer. The independent buffer includes multiple memory blocks, which are dynamically managed through a linked list structure.

[0064] Step 103: Based on the same timestamp, the original frames are obtained from the corresponding independent buffer of each image sensor and synthesized to obtain the synthesized frames. Multiple consecutive synthesized frames constitute the synthesized video.

[0065] The main control module is the main control SoC chip of the camera system. As the core processing unit of the system, it is responsible for overall scheduling, data processing, instruction issuance, and resource allocation. In this embodiment, it is used to: generate synchronization pulse signals and send them to the clock calibration circuits of each camera via the I2C bus; run software algorithms such as video acquisition, encoding, and synthesis; manage and schedule resources such as CPU, GPU, memory, and temperature control; and implement intelligent strategies such as dynamic resolution allocation and temperature control adjustment.

[0066] Image sensors are responsible for converting optical signals to electrical signals and are the physical front end of image acquisition. Upon receiving a synchronization pulse signal, they precisely control the start and end times of exposure; output raw image data to the next processing unit; and support functions such as multi-frame output, exposure control, and gain adjustment. The camera system applicable to the embodiments of this application includes at least two image sensors to achieve multi-angle, multi-functional image acquisition. It should be noted that when multiple image sensors are used, the other components required for acquisition can be independent, i.e., a complete camera in terms of product form, or they can be shared. Taking independent sensors as an example, when there are two CMOS image sensors, each CMOS image sensor corresponds to one camera, and the system is defined as a dual-camera system.

[0067] A synchronization pulse signal is a timing signal used to trigger the synchronized operation of multiple image sensors. It has a fixed pulse width and period, such as a trigger signal with a 20ns pulse width.

[0068] During data acquisition, a synchronization pulse signal is sent by the main control module to trigger multiple image sensors to acquire data synchronously. Specifically, based on the synchronization pulse signal generated by the main control module, at least two image sensors are triggered to synchronously acquire raw frames, including:

[0069] The main control module's synchronous pulse generator generates a synchronous pulse signal with a fixed pulse width.

[0070] A fixed-width pulse signal is transmitted to the clock calibration circuitry of at least two image sensors to correct the local clock skew of each image sensor.

[0071] Based on the corrected local clock, at least two image sensors are triggered to acquire raw frames.

[0072] Synchronization pulse generator: A hardware module within the main control module used to generate precise time-triggered signals. For example, the timer module built into the main control chip.

[0073] Fixed pulse width refers to a pulse signal whose width remains constant along the time axis. For example, a trigger signal with a pulse width of 20ns.

[0074] Clock calibration circuitry refers to hardware units, such as RTC chips, used to correct the local clock offset of an image sensor.

[0075] The main control module generates a fixed-width pulse signal using its built-in synchronization pulse generator and transmits this signal to the clock calibration circuit via the I2C bus. The synchronization pulse generator ensures the timing accuracy of the pulse signal, while the I2C bus provides a stable data transmission channel, enabling the clock calibration circuit to receive and process the synchronization pulse signal in a timely manner. The clock calibration circuit in the image sensor adjusts its local clock according to the synchronization pulse signal to align it with the clock of the main control module, thereby ensuring clock alignment between at least two image sensors. Subsequently, at least two image sensors synchronously start frame acquisition according to the corrected local clock, ensuring that at least two image sensors achieve frame acquisition timing alignment at the physical layer. The entire process, through hardware-level signal transmission and calibration, avoids the impact of software scheduling on timing, forming a closed-loop synchronization mechanism.

[0076] In 4K recording scenarios, the main control chip sends a synchronization pulse signal every frame, and the clock calibration circuit of the dual-camera system adjusts its local clock so that the two CMOS sensors complete the exposure within ±1μs, eliminating image misalignment.

[0077] In one embodiment, the main control chip can also send a rasterization synchronization signal, such as a horizontal synchronization pulse HSYNC, to at least two image sensors to ensure that at least two sensors start exposure at the same stage of raster scanning, such as the scan line start point.

[0078] Rasterization synchronization aligns the raster scanning timing of the two CMOS sensors through hardware-level signal alignment, further eliminating microsecond-level timing deviations. For example, in real-time AR / VR fusion scenarios, rasterization synchronization ensures that the main camera and the secondary camera are exposed simultaneously at precise points in the raster scan, avoiding image misalignment caused by differences in scanning timing and improving the spatiotemporal consistency of the fused image.

[0079] An infrared synchronized flash can also be added between the two cameras, triggering exposure via a photoelectric sensor. This increases cost but improves synchronization accuracy to the nanosecond level. Furthermore,

[0080] The system uses a multi-channel buffer to store the raw frame data output by the image sensor. The amount of data in each channel of the multi-channel circular buffer corresponds to the amount of data in the image sensor; one channel corresponds to one image sensor, meaning one image sensor corresponds to an independent buffer. The buffer can be a circular buffer, a cyclic memory queue structure used for efficient management of frame data.

[0081] In the design of the multi-channel buffer, each camera corresponds to an independent circular buffer, which consists of multiple memory blocks. A linked list structure is used to dynamically manage the allocation and reclamation of memory blocks.

[0082] Workflow of independent buffers:

[0083] Writing phase: The raw frame data captured by the camera is directly written to the currently free memory block without additional copying.

[0084] Reading phase: The synthesis thread or encoding thread reads data directly from the ready memory block. After reading, the memory block is marked as free for use in the next frame.

[0085] Compared to the multi-level memory copying path of "acquisition → copy → encoding → re-copy" in traditional solutions, the multi-channel buffer design allows data to flow only in a single circular buffer, reducing CPU and memory bandwidth usage.

[0086] In one embodiment, such as Figure 2 As shown, based on the same timestamp, original frames are obtained from the corresponding independent buffers of each image sensor and synthesized to obtain a synthesized frame, including:

[0087] Step 201: Access the frame data in the independent buffer corresponding to each image sensor directly through the shared memory mapping area of ​​the graphics processor to obtain at least two original frames with the same timestamp.

[0088] Step 202: According to the preset mixing weight matrix of the graphics processor, perform pixel-level fusion on at least two original frames with the same timestamp to obtain a composite frame.

[0089] A shared memory map is a memory region that the GPU can directly access, enabling zero-copy data transfer. Examples include shared memory regions allocated via CMA (Contiguous Memory Allocator). This shared memory map serves as the compositing workspace, storing the raw frame data to be fused by the GPU. The GPU directly reads frames from both cameras through this region, performs pixel-level alpha blending, and generates the composite frame.

[0090] Specifically, each memory block in the circular buffer corresponds to a dmabuf handle, which can be passed to the GPU driver. When the compositing thread needs to process a frame, it imports the corresponding dmabuf handle into the GPU's shared memory mapping area, allowing the GPU to directly access that physical memory page.

[0091] Pointer passing instead of data copying: The synthesized thread only passes pointers (or handles) to memory blocks. The shared memory map actually maps the physical pages of the circular buffer to the GPU's address space through memory remapping, so the data exists in the same physical memory from beginning to end.

[0092] Dynamic buffer switching: The circular buffer is managed by a linked list. When a frame of data is filled into a memory block, the block is marked as "ready". When the synthesis thread consumes the data, it directly removes the block from the linked list, maps its physical address to the shared memory area, and then notifies the GPU to process it. After processing, the block is recycled back to the circular buffer for reuse by the acquisition end.

[0093] By managing dynamic buffers and sharing memory mapping with the GPU, memory copy latency during video compositing is further reduced. For example, in live streaming scenarios, after raw frame data from at least two image sensors is written to a dynamic circular buffer, the GPU directly reads the data through shared memory, avoiding full-frame copy transmission, significantly shortening compositing latency, and improving the real-time interactive experience.

[0094] In one embodiment, storing the raw frames acquired by each image sensor into a corresponding independent buffer includes:

[0095] Determine the output frame rate and data volume for each image sensor;

[0096] Adjust the capacity of the independent buffer corresponding to each image sensor based on the output frame rate and data volume of each image sensor.

[0097] The raw frames acquired by each image sensor are stored in an independent buffer with adjusted capacity.

[0098] In this embodiment, the capacity of the independent buffer is dynamically changed. The size of the memory pool is adjusted in real time based on the current output speed (frame rate) and output weight (resolution / data volume), and these memories are managed in a cyclical manner to avoid constantly requesting memory and copying data.

[0099] The system monitors the image sensor's status in real time. This includes whether the frame rate is 30fps or 60fps; the data volume can be determined based on resolution and data format (4K or 1080P); and the data format (RAW or YUV). Then, it calculates the required data volume: for example, if the secondary camera is currently capturing static text (low data volume) and the frame rate is stable at 30fps, and the system calculates that the secondary camera generates relatively little data per second, requiring no large buffer for temporary storage, it can notify the memory management unit to reduce the total capacity of the secondary camera's corresponding independent buffer from 20MB to 8MB. Conversely, if the scene suddenly becomes more complex (e.g., the secondary camera switches to capturing high-speed motion), the system will immediately expand the buffer to prevent data overflow and frame drops.

[0100] In one embodiment, the method further includes:

[0101] Real-time acquisition of the fill rate of the independent buffers corresponding to at least two image sensors;

[0102] If the fill rate is less than a preset threshold, adjust the resolution and / or bit rate of at least two image sensors.

[0103] The buffer fill rate is a key indicator of whether the system's production and consumption speeds are matched. In the multi-channel circular buffer design of this embodiment, each memory block has three states: empty (writable), full (readable / pending processing), and processing (occupied). Fill rate = (size of the buffer currently full of data) / (total buffer capacity). The preset threshold is a preset safety level. For example, a preset threshold of 70% means that the amount of data accumulated in the buffer is already below 70% of the total capacity.

[0104] The low fill rate is essentially due to the fact that the delivery speed of the "producer (camera)" is slower than the processing speed of the "consumer (encoding / compositing thread)". Specific reasons may include: (1) Reduced scene complexity. When the scene changes from a fast-moving scene (such as a ball game) to a static scene (such as a presentation PPT), the encoder processes static frames very quickly (because the changes between frames are small and the compression is fast). The consumer processes too quickly, and the data in the buffer is "emptied" instantly. (2) Camera acquisition delay. Although the solution has hardware synchronization, if the environment becomes dark, the camera sensor may reduce the frame rate (such as from 30fps to 25fps) in order to ensure correct exposure, resulting in a slower "production" speed. (3) Temporary release of system resources. The CPU / GPU resources that were originally occupied are suddenly released, causing the compositing / encoding thread to suddenly accelerate and quickly consume the backlog of frames.

[0105] When the fill rate is below the threshold, the compositing thread may idle due to a lack of data, causing stuttering or a drop in frame rate in the output video. This issue can be resolved by reducing the resolution / bitrate. Specifically, reducing the resolution decreases the time it takes for the sensor to read data, effectively increasing the frame rate and allowing the producer to run the video faster.

[0106] In one embodiment, such as Figure 3 As shown, at least two image sensors include a main image sensor and a secondary image sensor; adjusting the resolution and / or bitrate of at least two image sensors includes:

[0107] Step 301: Identify the scene type based on the synthesized video;

[0108] Step 302: When the scene type is a static scene, reduce the resolution of the secondary image sensor and increase the compression ratio of the synthesized video. The scene type is determined based on historical frames.

[0109] Step 303: When the scene type is a dynamic scene, automatically balance the bitrate allocation of at least two image sensors and transmit the synthesized video based on the bitrate allocation.

[0110] The identification of scene types can be achieved through a scene analysis module. This module specifically includes at least one of the following: a motion detection subunit based on optical flow; a complexity calculation subunit based on DCT coefficient analysis; and a lightweight AI model.

[0111] The motion detection subunit calculates the motion vector for each pixel between adjacent frames. If the motion vectors of most pixels are close to zero, the scene is considered static; if there are a large number of non-zero vectors with significant amplitude, the scene is considered dynamic. Furthermore, it can combine metrics such as the amount of data on moving objects and their speed to distinguish between "slight shaking" and "vigorous movement."

[0112] The complexity calculation subunit divides the image into small blocks and transforms it to the frequency domain. The amount of high-frequency coefficients (details, edges) reflects the texture complexity of the image. If the energy of the high-frequency coefficients is very low, it indicates that the image is flat and has few details, which is suitable for increasing the compression ratio; if the high-frequency coefficients are abundant, more details need to be preserved, and excessive compression is not advisable.

[0113] Motion information and texture complexity can be fused to obtain a scene complexity score, which is then categorized according to preset thresholds: Static scenes: low motion + low texture → reduce secondary camera resolution + increase compression ratio. Dynamic scenes: high motion or high texture → maintain high resolution + balance bitrate allocation.

[0114] A lightweight AI model is introduced into the scene analysis module. By analyzing the motion characteristics (such as optical flow detection of dynamic regions) and texture complexity (such as DCT coefficient analysis) of dual-camera images in real time, it predicts the scene complexity of the next frame. The AI ​​model outputs a dynamic resolution allocation strategy to the resolution controller, prioritizing high resolution in dynamic regions (such as moving objects) and reducing resolution in static regions (such as the background). By learning scene patterns from historical frame data, the AI ​​model can more accurately identify dynamic and static regions, avoiding the misjudgment of local motion by traditional optical flow methods. For example, in a meeting recording scenario, the AI ​​model can identify the speaker's gesture areas and maintain high resolution, while automatically reducing the resolution of background areas, thereby significantly reducing computational consumption while preserving image detail.

[0115] The aforementioned adaptive allocation strategy addresses the pain points of traditional fixed-resolution solutions: in static scenes, it avoids wasting computing power by having the secondary camera run at high resolution; in dynamic scenes, it ensures that both cameras capture sufficient detail to guarantee the quality of the synthesized video. Overall, it achieves a balance between power consumption and image quality, extends recording time, and improves the user experience. Through this mechanism, the system can intelligently sense environmental changes and dynamically adjust parameters, satisfying high-performance requirements while also considering energy saving and heat dissipation.

[0116] In one embodiment, the method further includes:

[0117] Upon detecting that the user has selected a recording mode, the multi-level temperature control unit is activated. The multi-level temperature control unit is used to dynamically adjust the operating frequency of the main control module and / or the graphics processor based on feedback from the temperature sensor.

[0118] The multi-level temperature control unit is a power management unit integrated into the main control module, containing multiple levels of power consumption threshold switches. For example, a three-level temperature control strategy based on a PMIC (Power Management IC) may be employed.

[0119] The multi-level temperature control unit monitors the device temperature in real time through temperature sensors and dynamically adjusts the CPU / GPU frequency according to preset power consumption thresholds (such as 60℃ / 70℃ / 80℃). For example, it reduces the GPU frequency to maintain voltage stability under high temperature conditions.

[0120] The system's energy efficiency is further optimized through a multi-level temperature control unit. For example, in continuous 4K recording scenarios, the temperature control unit dynamically shuts down unnecessary CPU cores based on temperature sensor feedback, maintaining the GPU voltage within a reasonable range, thereby extending the device's battery life and reducing surface temperature.

[0121] The following explanation uses the synchronous processing of a dual-camera system as an example. The core idea of ​​the dual-channel pipeline architecture is that acquisition, analysis, synthesis, and encoding are executed in parallel, with the circular buffer and shared memory mapping area playing crucial roles.

[0122] (a) Production line stage division

[0123] Phase 1: Dual-camera data collection

[0124] Two cameras capture data independently, each writing the raw frames to its own circular buffer. Each buffer is managed by a linked list, supporting dynamic expansion to ensure no frame loss at high frame rates.

[0125] Phase 2: Scenario Analysis and Decision Making

[0126] The scene analysis module (optical flow method + DCT) reads frame data (or metadata) in the circular buffer in real time, calculates the image complexity and motion level, and outputs the resolution allocation strategy to the controller.

[0127] Phase 3: Synthesis Preparation

[0128] The synthesis thread selects the corresponding frame (e.g., a frame with the same timestamp) from the two circular buffers according to the controller instructions, obtains its dmabuf handle, and maps its physical memory page to the shared memory region.

[0129] Phase 4: GPU Synthesis

[0130] The GPU directly accesses the raw data of the two frames through a shared memory region, performs pixel-level fusion according to the alpha mixing weight matrix, generates a composite frame, and stores it in the output buffer.

[0131] Phase 5: Encoding Output

[0132] The encoder reads synthesized frames from the output buffer, compresses them, and then stores or streams them.

[0133] (ii) Parallel pipeline mechanism

[0134] The multi-block structure of the circular buffer allows for simultaneous acquisition and consumption: the acquisition end continuously writes new frames to the free block, while the composition end simultaneously reads ready blocks without interference. Dynamic binding of the shared memory mapping area allows the GPU to process multiple pairs of frames continuously: while the GPU is processing the current frame, the composition thread can pre-map the physical memory of the next pair of frames to another set of virtual addresses in the shared area, achieving pipelined parallelism. The flexibility of linked list management allows the system to adjust the buffer depth according to scene changes: increasing the buffer capacity in dynamic scenes to prevent sudden data bursts, and reducing the capacity in static scenes to save memory.

[0135] (III) Coordination of Data Flow and Control

[0136] Synchronization control: Hardware synchronization pulses ensure dual-camera frame alignment, and the synthesis thread matches the corresponding frame pairs from the circular buffer based on the timestamp or frame sequence number.

[0137] Dynamic resolution allocation: The output of the scene analysis module directly affects the compositing stage: For example, in a static scene, the secondary camera's circular buffer may only store low-resolution frames, the shared memory mapping area will be mapped to a smaller memory region, and the GPU's compositing algorithm will also be adapted.

[0138] Temperature control intervention: When the temperature exceeds the threshold, PMIC restricts non-core threads through the task scheduler, but retains the priority of synthesis threads to ensure that the main pipeline path is not affected.

[0139] Zero copy: Data always resides in the same physical memory from the sensor to the GPU. The circular buffer and the shared memory mapping area are shared through dmabuf, completely eliminating CPU copying.

[0140] Low latency: The ring buffer decouples acquisition and processing, and the shared memory mapping area allows the GPU to access it directly, reducing the synthesis latency to less than 40ms.

[0141] High bandwidth utilization: Dynamic buffer management avoids memory waste, and shared memory mapping reduces memory bandwidth usage.

[0142] Strong real-time performance: Hardware synchronization ensures frame alignment, and pipeline parallelism ensures that the processing speed matches the acquisition speed.

[0143] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0144] Based on the same inventive concept, this application also provides a camera synchronization processing device. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more camera synchronization processing device embodiments provided below can be found in the limitations of the camera synchronization processing method above, and will not be repeated here.

[0145] In one embodiment, such as Figure 4 As shown, the camera synchronization processing device includes a synchronization control module 401, a storage module 402, and a synthesis module 403, wherein:

[0146] Synchronization control module 401 is used to trigger at least two image sensors to synchronously acquire raw frames based on the synchronization pulse signal generated by the main control module.

[0147] Storage module 402 is used to store the raw frames acquired by each image sensor into a corresponding independent buffer. The independent buffer includes multiple memory blocks, which are dynamically managed through a linked list structure.

[0148] The compositing module 403 is used to acquire original frames from the corresponding independent buffer of each image sensor based on the same timestamp and composite them to obtain composite frames. Multiple consecutive composite frames constitute a composite video.

[0149] In one possible implementation, the synchronization control module 401 is specifically used for:

[0150] The main control module's synchronous pulse generator generates a synchronous pulse signal with a fixed pulse width.

[0151] A fixed-width pulse signal is transmitted to the clock calibration circuitry of at least two image sensors to correct the local clock skew of each image sensor.

[0152] Based on the corrected local clock, at least two image sensors are triggered to acquire raw frames.

[0153] In one possible implementation, storage module 402 is specifically used for:

[0154] Determine the output frame rate and data volume for each image sensor;

[0155] Adjust the capacity of the independent buffer corresponding to each image sensor based on the output frame rate and data volume of each image sensor.

[0156] The raw frames acquired by each image sensor are stored in an independent buffer with adjusted capacity.

[0157] In one possible implementation, the synthesis module 403 is specifically used for:

[0158] By directly accessing the frame data in the independent buffer corresponding to each image sensor through the shared memory mapping area of ​​the graphics processor, at least two raw frames with the same timestamp can be obtained.

[0159] Based on the graphics processor's preset mixing weight matrix, pixel-level fusion is performed on at least two original frames with the same timestamp to obtain a composite frame.

[0160] In one possible implementation, the camera synchronization processing device further includes an adaptive allocation module for:

[0161] Real-time acquisition of the fill rate of the independent buffers corresponding to at least two image sensors;

[0162] If the fill rate is less than a preset threshold, adjust the resolution and / or bit rate of at least two image sensors.

[0163] In one possible implementation, at least two image sensors include a main image sensor and a secondary image sensor; the adaptive allocation module is specifically used for:

[0164] Identify scene types based on synthesized video;

[0165] When the scene type is static, the resolution of the secondary image sensor is reduced and the compression ratio of the synthesized video is increased. The scene type is determined based on historical frames.

[0166] When the scene type is a dynamic scene, the bitrate allocation of at least two image sensors is automatically balanced, and the synthesized video is transmitted based on the bitrate allocation.

[0167] In one possible implementation, the camera synchronization processing device further includes a temperature control module for:

[0168] Upon detecting that the user has selected a recording mode, the multi-level temperature control unit is activated. The multi-level temperature control unit is used to dynamically adjust the operating frequency of the main control module and / or the graphics processor based on feedback from the temperature sensor.

[0169] Each module in the aforementioned camera synchronization processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of each module.

[0170] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0171] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0172] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0173] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0174] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0175] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0176] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0177] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0178] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0179] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0180] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0183] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0184] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0185] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A camera synchronization processing method, characterized in that, The method includes: Based on the synchronization pulse signal generated by the main control module, at least two image sensors are triggered to synchronously acquire raw frames. Each raw frame acquired by an image sensor is stored in a corresponding independent buffer. The independent buffer includes multiple memory blocks, which are dynamically managed through a linked list structure. Based on the same timestamp, original frames are obtained from the corresponding independent buffer of each image sensor and synthesized to obtain composite frames. Multiple consecutive composite frames constitute a composite video.

2. The method according to claim 1, characterized in that, The synchronization pulse signal generated by the main control module triggers at least two image sensors to synchronously acquire raw frames, including: The main control module's synchronous pulse generator generates a synchronous pulse signal with a fixed pulse width. The fixed-width pulse signal is transmitted to the clock calibration circuit of the at least two image sensors to correct the local clock offset of each image sensor. The at least two image sensors are triggered to acquire raw frames based on the corrected local clock.

3. The method according to claim 1, characterized in that, The step of storing the raw frames acquired by each image sensor into a corresponding independent buffer includes: Determine the output frame rate and data volume for each image sensor; Adjust the capacity of the independent buffer corresponding to each image sensor based on the output frame rate and data volume of each image sensor. The raw frames acquired by each image sensor are stored in an independent buffer with adjusted capacity.

4. The method according to claim 1, characterized in that, The process of obtaining original frames from the corresponding independent buffers of each image sensor based on the same timestamp and synthesizing them to obtain synthesized frames includes: By directly accessing the frame data in the independent buffer corresponding to each image sensor through the shared memory mapping area of ​​the graphics processor, at least two raw frames with the same timestamp can be obtained. According to the preset mixing weight matrix of the graphics processor, pixel-level fusion is performed on at least two original frames with the same timestamp to obtain a composite frame.

5. The method according to claim 1, characterized in that, The method further includes: The fill rate of the corresponding independent buffers of the at least two image sensors is obtained in real time; If the fill rate is less than a preset threshold, adjust the resolution and / or bit rate of the at least two image sensors.

6. The method according to claim 5, characterized in that, The at least two image sensors include a main image sensor and a secondary image sensor; adjusting the resolution and / or bitrate of the at least two image sensors includes: Identify scene types based on synthesized video; When the scene type is a static scene, the resolution of the sub-image sensor is reduced and the compression ratio of the synthesized video is increased, wherein the scene type is determined based on historical frames; When the scene type is a dynamic scene, the bitrate allocation corresponding to the at least two image sensors is automatically balanced, and the synthesized video is transmitted based on the bitrate allocation.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Upon detecting that the user has selected a recording mode, a multi-level temperature control unit is activated. This multi-level temperature control unit is used to dynamically adjust the operating frequency of the main control module and / or the graphics processor based on feedback from the temperature sensor.

8. A camera synchronization processing device, characterized in that, The device includes: The synchronization control module is used to trigger at least two image sensors to synchronously acquire raw frames based on the synchronization pulse signal generated by the main control module. The storage module is used to store the raw frames acquired by each image sensor into a corresponding independent buffer. The independent buffer includes multiple memory blocks, which are dynamically managed through a linked list structure. The compositing module is used to acquire raw frames from the corresponding independent buffer of each image sensor based on the same timestamp and compose them to obtain composite frames. Multiple consecutive composite frames constitute a composite video.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images