Multi-camera synchronization control method, system and robotic work system

Abstract

This application provides a multi-camera synchronous control method, system, and robot working system. The method includes: generating trigger signals for unified control of multiple cameras based on a single clock source in the multi-camera synchronous control system, generating trigger events corresponding to the trigger signals, and adding the trigger events to a trigger event queue in the order of generation. Each trigger event includes a trigger timestamp, a trigger sequence number, and camera parameters for each camera. The clock source provides time accuracy at least at the microsecond level. The method also includes: receiving image frames acquired and fed back by each camera according to the trigger signals; and matching the image frames fed back by each camera with the corresponding trigger events in the trigger event queue to determine the target timestamp of each camera's image frame. This achieves unified management of the exposure start time and image timestamps of multiple cameras, ensuring the consistency and predictability of the data acquired by each camera in the time dimension.

Description

Technical Field

[0001] This application relates to the field of industrial camera technology, and more specifically, to a multi-camera synchronous control method, system, and robot working system. Background Technology

[0002] With the widespread application of intelligent robots in industrial automation, warehousing and logistics, inspection and collaborative operations, their ability to accurately perceive the environment increasingly relies on multi-camera vision systems. Especially in tasks such as high-speed movement, dynamic obstacle avoidance, hand-eye coordination or stereo mapping, multiple image frames must be strictly aligned with microsecond-level time accuracy; otherwise, it will lead to perception distortion, positioning drift or even control failure.

[0003] Currently, multi-camera synchronization solutions mainly rely on hardware triggering or timestamps generated by the operating system or the camera itself.

[0004] However, existing multi-camera synchronization schemes use independent clocks or system timestamps for each camera, resulting in fragmented time references. Furthermore, when determining the image acquisition time, the trigger time is directly used as the image acquisition time without considering the impact of other factors on image acquisition, leading to inaccuracies in the determined image acquisition time. In addition, in multi-robot collaborative operations, existing schemes cannot achieve global time alignment across robots, and network transmission jitter further exacerbates time drift, affecting the consistency of group perception. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a multi-camera synchronization control method, system, and robot working system to improve the accuracy of multi-camera synchronization.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a multi-camera synchronization control method, the method comprising: A single clock source in a multi-camera synchronization control system generates trigger signals for unified control of multiple cameras, generates trigger events corresponding to the trigger signals, and adds the trigger events to the trigger event queue in the order of generation. The trigger events include a trigger timestamp, a trigger sequence number, and camera parameters of each camera. The clock source provides time accuracy of at least microseconds. Each of the cameras receives image frames acquired and fed back by the trigger signal; The image frames fed back by each camera are matched with the corresponding trigger events in the trigger event queue to determine the target timestamp of the image frames of each camera.

[0007] Optionally, generating the trigger event corresponding to the trigger signal includes: Obtain the current system time and camera parameters of each camera, wherein the current system time is used to indicate the time preceding the actual output time of the trigger signal; The current system time is corrected based on the signal deviation to obtain the trigger timestamp in the trigger event corresponding to the trigger signal. The trigger sequence number of the trigger event is generated according to the sequence number of the previous trigger event of the trigger event. The trigger timestamp and the camera parameters of each camera are used as the trigger event.

[0008] Optionally, the step of matching the image frames fed back by each of the cameras with the corresponding trigger events in the trigger event queue to determine the target timestamp of the image frames of each camera includes: The trigger sequence number of each trigger event in the trigger event queue is traversed. For the current trigger sequence number, the current trigger sequence number is matched with the frame sequence number in the image frame. If the current trigger sequence number matches the frame sequence number, the trigger event corresponding to the current trigger sequence number is taken as the target trigger event that matches the image frame. Each frame sequence number of the image frame has a corresponding trigger sequence number. Based on the frame information of the image frame and the target trigger event, determine the target timestamp of the image frame; and / or, traverse the trigger sequence number of each corresponding trigger event in the trigger event queue. If the trigger sequence number in the trigger event queue is empty, mark the image frame as unmatched.

[0009] Optionally, determining the target timestamp of the image frame based on the frame information of the image frame and the target triggering event includes: The exposure center time of the image frame is determined based on the frame information of the image frame, and the exposure center time is corrected based on the trigger timestamp of the target trigger event and the trigger jitter parameters to obtain the target timestamp of the image frame.

[0010] Optionally, it also includes: Retrieve multiple event pairs, each of which includes two adjacent historical trigger events; Obtain the actual trigger interval and expected trigger interval for each event pair; The triggering error between the actual triggering interval and the expected triggering interval for each event pair is determined, and the triggering error for each event pair is obtained. The predicted triggering error is predicted based on the changing trend of the triggering error for each event pair. The expected triggering time of the next triggering signal is corrected based on the predicted triggering error, and the target expected triggering time of the next triggering signal is obtained.

[0011] Optionally, the process of the camera acquiring image frames and generating frame information includes: The trigger signal is parsed to obtain the signal number in the trigger signal; Image frames are acquired based on the signal sequence number, and a frame sequence number corresponding to the signal sequence number is generated, and the frame sequence number is used as the frame information.

[0012] Optionally, before receiving the image frames acquired and fed back by each of the cameras according to the trigger signal, the process includes: The trigger signal is sent to the slave control device corresponding to each camera; If the trigger signal is the first trigger signal, each slave control device calculates the time deviation based on the local reception time of each slave control device and the trigger timestamp in the trigger event corresponding to the trigger signal, and performs alignment correction on the local reception time based on the time deviation to obtain the global timestamp corresponding to the first trigger signal. Each slave control device sends the trigger signal to the camera connected to the slave control device based on the global timestamp. And / or, if the trigger signal is a trigger signal other than the first trigger signal, then the global timestamp corresponding to the trigger signal is determined according to the trigger sequence number, trigger timestamp and trigger period in the trigger event corresponding to the trigger signal, and each slave control device sends the trigger signal to the camera connected to the slave control device according to the global timestamp.

[0013] Optionally, it also includes: For each triggering event in the triggering event queue, perform event continuity detection, event loss detection, and event priority sorting; And / or, after generating a trigger event corresponding to the trigger signal, if no image frame corresponding to the trigger event is received within a preset time, a timeout exception is determined and the trigger event is discarded; If no matching trigger event is found for the image frame, the image frame is marked as having an invalid timestamp, and an exception log is recorded. If the triggering period of the trigger signal is abnormal, the parameter of the triggering period will be automatically adjusted. If there are consecutive missing events or discontinuous image frame numbers, the missing sequence number is recorded and a warning message is generated.

[0014] Secondly, embodiments of this application also provide a multi-camera synchronous control system, including: a trigger control unit, a timestamp management unit, an image frame binding unit, a debugging and monitoring unit, and a configuration and parameter management unit; A trigger control unit is used to generate a trigger signal for unified control of multiple cameras based on a single clock source in a multi-camera synchronization control system, generate trigger events corresponding to the trigger signals, and add the trigger events to a trigger event queue in the order of generation. The trigger events include a trigger timestamp, a trigger sequence number, and camera parameters of each camera. The clock source provides a time accuracy of at least microseconds. The timestamp management unit is used to receive image frames acquired and fed back by each of the cameras according to the trigger signal; The timestamp management unit is used to match the image frames fed back by each camera with the corresponding trigger events in the trigger event queue to determine the target timestamp of the image frames of each camera.

[0015] The multi-camera synchronous control system further includes: an image frame binding unit, a debugging and monitoring unit, and a configuration and parameter management unit; The image frame binding unit is used to receive image frames from each camera and bind the target timestamp information of each image frame to the image frame metadata according to the frame number or queue order of each image frame. The debugging and monitoring unit is used to record the system operating status, trigger time deviation, timestamp generation results, and abnormal events. The configuration and parameter management unit is used to manage the number of cameras, triggering cycle, exposure parameter update strategy, and synchronization mode.

[0016] Thirdly, embodiments of this application also provide a robot working system, including multiple cameras and the multi-camera synchronization control system described in the second aspect; The multiple cameras are all mounted on the robot, and the multi-camera synchronization control system is communicatively connected to the multiple cameras.

[0017] The beneficial effects of this application are: This application provides a multi-camera synchronous control method, system, and robot working system. Based on a single clock source in the multi-camera synchronous control system, it generates trigger signals for unified control of multiple cameras and generates trigger events corresponding to these signals. These trigger events are added to a trigger event queue in the order of generation. The image frames fed back by each camera are matched with the corresponding trigger events in the trigger event queue to determine the target timestamp of each camera's image frame. This achieves synchronous management of trigger control, timestamp generation, and image frame time calibration for multiple cameras. Each camera shares the same trigger signal and timestamp reference, achieving precise alignment of multiple cameras on the same timeline. Furthermore, through centralized time control and hierarchical modular design, it achieves unified management of the exposure start time and image timestamps of multiple cameras without relying on internal camera timestamps or proprietary implementations, ensuring the consistency and predictability of the data acquired by each camera in the time dimension. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This application provides a schematic diagram of the architecture of a robot working system. Figure 2 A flowchart illustrating a multi-camera synchronization control method provided in an embodiment of this application; Figure 3 A flowchart illustrating the second multi-camera synchronization control method provided in this application embodiment; Figure 4 A flowchart illustrating the third multi-camera synchronization control method provided in this application embodiment; Figure 5 A flowchart illustrating the fourth image timestamp determination method provided in this application embodiment; Figure 6 This is a flowchart illustrating the fifth multi-camera synchronization control method provided in this application embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0021] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0023] Optionally, the multi-camera synchronization control method provided in this application embodiment is applied to a multi-camera synchronization control system, which can be a system on an electronic device deployed on a robot or a server deployed in the cloud.

[0024] Figure 1 This is a schematic diagram of the architecture of a robot working system provided in an embodiment of this application, such as... Figure 1 As shown, the multi-camera synchronization control system may include a multi-camera synchronization control system and multiple cameras, all of which are mounted on the robot. The multi-camera synchronization control system may include a trigger control unit, a timestamp management unit, an image frame binding unit, a debugging and monitoring unit, and a configuration and parameter management unit. Furthermore, the multi-camera synchronization control system can communicate with all the cameras deployed on the robot.

[0025] Optionally, the various units in the multi-camera synchronization control system operate collaboratively within a CameraSync Daemon, uniformly scheduled by the main control SoC. The main control SoC periodically outputs trigger signals to the deserializer chip via GPIO ports. The deserializer distributes these trigger signals to each camera in TTL format, and each camera adjusts its exposure start time in real time based on these trigger signals. Since cameras cannot directly record high-precision timestamps internally, the system does not rely on the cameras' time synchronization capabilities. Instead, it employs a multi-camera synchronization trigger mechanism, using the high-precision system timestamp captured by the main control SoC before outputting the trigger signal as the unified time reference for all image frames, used for time calibration of subsequent image frames acquired by each camera. All timestamps originate from a single time source within the main control SoC, effectively avoiding the uncertainties introduced by multiple clock sources.

[0026] The trigger control unit is used to generate camera trigger signals in a periodic or event-driven manner, and to control the pulse width, frequency, and phase of the trigger signals. This trigger control unit ensures the stability of the trigger cycle through a timing mechanism and completes timestamp acquisition before the trigger signal is output.

[0027] The timestamp management unit records the time information and exposure parameters of the triggered event, and generates the target timestamp of the image frame according to a preset algorithm. This timestamp management unit can maintain the trigger record queue to ensure a one-to-one correspondence between the triggered event and the image frame.

[0028] The image frame binding unit is used to receive image frame data from each camera and bind the corresponding timestamp information to the image frame metadata according to the frame number or queue order to achieve time calibration.

[0029] The configuration and parameter management unit is used to manage configuration items such as the number of cameras, triggering period, exposure parameter update strategy, and synchronization mode. It supports runtime parameter updates and ensures the timing consistency between configuration changes and triggering behavior.

[0030] The debugging and monitoring unit is used to record system operating status, trigger time deviation, timestamp generation results and abnormal events, supporting subsequent diagnosis and system status monitoring.

[0031] The configuration and parameter management unit is used to manage the number of cameras, triggering cycle, exposure parameter update strategy, and synchronization mode.

[0032] Optionally, after the multi-camera synchronous control system starts as a daemon process, it sequentially completes configuration loading, hardware initialization, timestamp management initialization, and thread startup. First, it reads the configuration file to obtain information such as the number of cameras, trigger cycle, exposure parameters, and synchronization mode, and then sends these parameters to the trigger control unit, timestamp management unit, and image frame binding unit. Next, it initializes the trigger signal output interface and camera data interface, confirms that each camera is in a triggerable state, and establishes a trigger event queue and an image frame buffer queue. The timestamp management unit obtains the system master clock source, initializes the time counter, and loads the camera hardware delay, exposure parameters, and line scan parameters for calculating the target timestamp of subsequent image frames from each camera. After completing the above initialization, it starts the trigger control thread, timestamp management thread, image frame binding thread, and monitoring thread, and the multi-camera synchronous control system enters normal operation.

[0033] Figure 2 This is a flowchart illustrating a multi-camera synchronization control method provided in an embodiment of this application. The execution entity of this method is as described in the aforementioned multi-camera synchronization control system. Figure 2 As shown, the method includes: S101. Based on a single clock source in a multi-camera synchronous control system, generate trigger signals for unified control of multiple cameras, generate trigger events corresponding to the trigger signals, and add the trigger events to the trigger event queue in the order of generation.

[0034] The trigger event may include a trigger timestamp, a trigger sequence number, and camera parameters for each camera, with the clock source providing time accuracy at least at the microsecond level.

[0035] Optionally, this single clock source is a high-precision, low-drift global single clock source, ensuring timestamp continuity and global consistency. The trigger control unit uses this single clock source as the sole time reference for the entire synchronization system and constructs a logical event entity, i.e., the trigger event, at a deterministic moment before the trigger signal is output. This trigger event is not a passive recording of hardware signals, but a composite data structure integrating the trigger timestamp, trigger sequence number, and parameters of each camera. The generation of this trigger event is controlled by the single clock source and possesses time resolution capabilities at the microsecond level or higher.

[0036] Optionally, after a trigger event is generated, it is injected into the trigger event queue in an orderly manner according to the generation sequence. This trigger event queue not only has storage function, but also serves as the mapping index source for subsequent image frame time binding. After each image frame enters the multi-camera synchronization system, it needs to complete the confirmation and assignment of the image frame time identity through this trigger event queue, thereby establishing a causal closed loop and time traceability path for the entire link of triggering, acquisition, and binding.

[0037] S102: Receive the image frames acquired and fed back by each camera according to the trigger signal.

[0038] Optionally, after the trigger control unit generates a trigger signal, it sends the generated trigger signal to each camera. Specifically, such as... Figure 1 As shown, the trigger control unit can send a trigger signal to the deserializer chip, which then distributes the trigger signal to each camera in TTL format. Upon receiving the trigger signal generated by the single clock source, each camera performs image acquisition according to a preset imaging strategy and feeds back the acquired image frames and their frame identifiers to the timestamp management unit in the multi-camera synchronization control system. The frame identifier information for each image frame may include a frame sequence number, camera identifier, and exposure parameters. The frame sequence number indicates the image frame's position in the local acquisition sequence of its respective camera, and the camera identifier distinguishes between different cameras.

[0039] S103. Match the image frames fed back by each camera with the corresponding trigger events in the trigger event queue to determine the target timestamp of the image frames of each camera.

[0040] Optionally, after receiving image frames from each camera, the timestamp management unit in the multi-camera synchronization control system matches each image frame with the corresponding trigger event in the trigger event queue to obtain the target trigger event that matches each image frame. The timestamp management unit can then determine the target timestamp of each camera's image frame based on the target trigger event that matches each image frame and the frame identifier information of each image frame.

[0041] When calculating the target timestamp for each camera, the timestamp management unit does not rely on the absolute timestamps actively reported by each camera, nor does it assume network clock synchronization among the cameras. Instead, it treats the received image frames from each camera as time decoding objects to determine the target trigger events for each camera's image frames. The frame sequence number in the frame identifier information of each camera's image frame constitutes a unique key value logically aligned with the trigger sequence number in the trigger event queue. Furthermore, instead of directly reusing the trigger timestamp from the target trigger event, it uses the trigger timestamp from the target trigger event as the reference origin for time calibration, further integrating the frame identifier information of each camera's image frames and system errors to determine the target timestamp for each camera's image frames. This constructs a verifiable, auditable, and iteratively optimizeable time semantic transformation pipeline from trigger events to image frames and then to target timestamps.

[0042] Optionally, after obtaining the target timestamp of the image frames of each camera, the image frame binding unit can bind the target timestamp of the image frames of each camera to the image frames of each camera to complete the time calibration.

[0043] In this embodiment, a trigger signal for unified control of multiple cameras is generated based on a single clock source in the multi-camera synchronous control system, and a trigger event corresponding to the trigger signal is generated. These trigger events are added to the trigger event queue in the order of generation. The image frames fed back by each camera are matched with the corresponding trigger events in the trigger event queue to determine the target timestamp of each camera's image frame. This achieves synchronous management of trigger control, timestamp generation, and image frame time calibration for multiple cameras. Each camera shares the same trigger signal and timestamp reference, achieving precise alignment of multiple cameras on the same timeline. Furthermore, through centralized time control and hierarchical modular design, unified management of the exposure start time and image timestamps of multiple cameras is achieved without relying on internal camera timestamps or proprietary implementations, ensuring the consistency and predictability of data acquired by each camera in the time dimension.

[0044] Figure 3 This is a flowchart illustrating the second multi-camera synchronization control method provided in the embodiments of this application, as shown below. Figure 3 As shown, the generation of the trigger event corresponding to the trigger signal in S101 above may include: S201. Obtain the current system time and camera parameters for each camera.

[0045] The current system time refers to the time preceding the actual output time of the trigger signal. Specifically, it is the monotonic clock time acquired by the trigger control unit at the moment immediately preceding the actual output time of the trigger signal. Camera parameters for each camera may include, for example, the exposure parameters of each camera.

[0046] Optionally, the trigger signal is generated periodically, such as trigger signal 1 at time T0, trigger signal 2 at time T0+T, trigger signal 3 at time T0+2T, etc. The current system time refers to the moment before the actual output time of the trigger signal in the current period, where the trigger signal in the current period refers to the trigger signal of any given period.

[0047] S202. Correct the current system time based on the signal deviation to obtain the trigger timestamp in the trigger event corresponding to the trigger signal, generate the trigger sequence number of the trigger event according to the sequence number of the previous trigger event, and use the trigger timestamp and the camera parameters of each camera as the trigger event.

[0048] Optionally, to avoid the impact of system time jumps on the continuity of timestamps, the current system time can be corrected based on the signal deviation to obtain the trigger timestamp. The historical clock deviations between the current system time of the trigger signal for each historical period and the corresponding external reference time are calculated. Trend analysis and linear regression are performed on each historical clock deviation to obtain the signal deviation for the current period. Based on the signal deviation of the current period, the current system time for the current period is corrected, thus obtaining the trigger timestamp in the trigger event corresponding to the trigger signal of the current period. Here, the external reference time is the time signal output from a highly stable time source of the multi-camera synchronization control system, independent of the robot, which is externally connected to the multi-camera synchronization control system. The external reference time corresponding to each current system time refers to the external reference time acquired simultaneously when acquiring each current system time.

[0049] Optionally, a globally monotonically increasing and replay-resistant trigger numbering mechanism can be used to generate the trigger number of each trigger event. For example, the trigger number of the current trigger event can be obtained by incrementing the number of the previous trigger event by 1. After obtaining the trigger timestamp and trigger number of the trigger event, the trigger timestamp, trigger number, and camera parameters of each camera can be encapsulated into the trigger event.

[0050] In this embodiment, the current system time is corrected based on the signal deviation to make the trigger timestamps of each trigger event more accurate, avoid the impact of system time jumps on the continuity of timestamps, and generate trigger events corresponding to each trigger signal so that each subsequent camera can use the trigger event as a reference.

[0051] Optionally, the process in S103 above, which involves matching the image frames fed back by each camera with the corresponding trigger events in the trigger event queue to determine the target timestamp of the image frames from each camera, may include: The process iterates through the trigger event queue, traversing the trigger numbers of each corresponding trigger event. For the current trigger number, it matches it with the frame number in the image frame. If the current trigger number matches the frame number, the trigger event corresponding to the current trigger number is taken as the target trigger event for the image frame. Then, based on the frame information of the image frame and the target trigger event, the target timestamp of the image frame is determined. Note that each image frame's frame number has a corresponding trigger number; that is, there is a correspondence between the signal number of the trigger signal, the trigger number of the trigger event, and the frame number of the image frame.

[0052] Optionally, each camera outputs an image frame after being triggered. To ensure a one-to-one correspondence between the image frame and the trigger event in the trigger event queue, an independent first-in-first-out (FIFO) queue is introduced in each camera for buffering image frames, ensuring that image frames are not discarded or overwritten before timestamp binding. The trigger event queue can adopt a circular buffer structure to guarantee the order of the first-in-first-out (FIFO) queues of each camera. When matching image frames with trigger events, the FIFO order of each camera can also be referenced to ensure that image frames and corresponding trigger events are matched according to the generation order of the trigger events. Each camera's FIFO queue contains the trigger timestamp of each trigger event, the frame number of the image frame, and the camera parameters of each camera.

[0053] For example, for trigger signal 1, trigger event 1 corresponding to the trigger signal is generated. Camera 1 acquires image frame A based on trigger signal 1, camera 2 acquires image frame B based on trigger signal 1, and camera 3 acquires image frame C based on trigger signal 1. When the timestamp management unit receives image frames A, B, and C, it can match the frame numbers of image A, B, and C with the trigger numbers of each trigger event according to the correspondence between frame numbers and trigger event trigger numbers. If trigger event 1 is an event generated at time T0, and the frame numbers in image A, B, and C all correspond to trigger event 1 trigger number 1, then trigger event 1 can be determined as the target trigger event for images A, B, and C.

[0054] Optionally, the trigger sequence number of each trigger event in the trigger event queue is traversed. If the trigger sequence number in the trigger event queue is empty, the image frame is marked as unmatched. Here, an empty trigger sequence number in the trigger event queue means that there is no trigger event in the queue. Specifically, the debugging and monitoring unit can detect whether there are trigger events in the trigger event queue and feed the detection result back to the timestamp management unit, which then performs the matching. In this embodiment, after receiving each image frame, each image frame is matched with the trigger events in the trigger event queue to obtain the target trigger events that match each image frame. Subsequently, the target timestamp of each image frame can be calculated based on the trigger timestamp of the matched target trigger events, achieving alignment of multiple images on a unified timeline. Furthermore, marking the image frame as unmatched when the trigger sequence number in the trigger event queue is empty can eliminate image frames without events caused by communication failures or hardware anomalies, ensuring that each frame output by the system has a traceable physical trigger source.

[0055] Optionally, determining the target timestamp of the image frame based on the frame information and the target triggering event may include: Optionally, the exposure center time of the image frame can be determined based on the frame information, and then corrected according to the trigger timestamp of the target trigger event and the trigger jitter parameters to obtain the target timestamp of the image frame. The trigger jitter parameters can be predicted based on the trigger jitter errors of historical trigger events. The exposure center time of the image frame refers to the center time when the camera captures the exposure of that image frame, and the trigger jitter parameters are the predicted trigger errors calculated below.

[0056] Figure 4 A flowchart illustrating the third multi-camera synchronization control method provided in this application embodiment is shown below. Figure 4 As shown, determining the exposure center time of an image frame based on its frame information can include: S301. Construct the camera's photon accumulation rate curve based on the camera's exposure mode.

[0057] The camera's exposure modes may include, but are not limited to, global shutter exposure mode, rolling shutter exposure mode, and zone exposure mode. The horizontal axis of the photon accumulation rate curve can be time t, and the vertical axis can be the instantaneous effective photon flux, that is, the effective photon flux I(t) at time t.

[0058] Optionally, the camera's exposure process differs in different exposure modes, resulting in different shapes for the photon accumulation rate curves constructed under different exposure modes. For example, in global shutter exposure mode, all pixels start and end exposure simultaneously, and the photon accumulation rate curve in this mode can be a trapezoidal or S-shaped curve. In rolling shutter exposure mode, pixels are exposed sequentially by row or column, with a fixed inter-row delay in the exposure time of each row. Therefore, in this rolling shutter exposure mode, a photon accumulation rate curve needs to be constructed separately for each row. Ideally, the photon accumulation rate curve in rolling shutter exposure mode can approach an ideal rectangle. Partition exposure mode divides the total exposure time of a frame into multiple discontinuous time periods, each of which can have different gains, different charge accumulation directions, or different subsets of pixels being exposed. In partition exposure mode, the photon accumulation rate curve can be a superposition of multiple rectangular pulses or arbitrary waveforms.

[0059] S302. Determine the exposure center time based on the photon accumulation rate curve, the exposure start time and the exposure end time in the frame information of the image frame.

[0060] Optionally, the exposure center time t can be obtained by integrating the photon accumulation rate curve based on the exposure start time and exposure end time in the frame information of the image frame. center .

[0061] In this embodiment, by deeply coupling the semantics of the exposure mode with the time-series modeling of the photon response, sub-microsecond-level physical reconstruction of the time base of the image acquisition process is achieved.

[0062] Figure 5 A flowchart illustrating the fourth image timestamp determination method provided in this application embodiment is shown below. Figure 5 As shown, determining the exposure center time in S302 based on the photon accumulation rate curve, the exposure start time, and the exposure end time in the image frame information may include: S401. Determine the time-weighted integral of photon accumulation based on the exposure start time, exposure end time, and photon accumulation rate curve.

[0063] The exposure start time refers to the moment when the camera shutter begins to open and the pixels begin to accumulate charge, while the exposure end time refers to the moment when the camera shutter is completely closed and the pixels stop accumulating charge.

[0064] Alternatively, it can be done through a formula The time-weighted integral is calculated, where, The start time of the exposure. The end time of the exposure. This refers to the time offset of each moment in the exposure process, from the start time to the end time, relative to the start time of the exposure. Let be the effective photon flux at time t.

[0065] S402. Calculate the total number of photons accumulated from the start time of exposure to the end time of exposure.

[0066] Specifically, total photon accumulation = ,in, The start time of the exposure. This is the end time of the exposure. Total photon accumulation refers to the total number of photons accumulated during the entire exposure of this image frame on the camera.

[0067] S403. The sum of the time-weighted integral and the total cumulative photon amount, and the exposure start time, is used as the exposure center time.

[0068] Specifically, the exposure center time can be calculated using the following formula (1).

[0069] Formula (1) in, The start time of the exposure. The end time of the exposure. This refers to the time offset of each moment in the exposure process, from the start time to the end time, relative to the start time of the exposure. Let be the effective photon flux at time t. The time of exposure.

[0070] In this embodiment, the exposure center time of each image frame is obtained by integrating the exposure curve, which can accurately represent the center moment of actual photon collection. Furthermore, the method in this embodiment not only takes the exposure midpoint but also considers the nonlinear characteristics of the exposure response, ensuring that the determined exposure center time can accurately reflect the center of photon accumulation in rolling shutter or nonlinear exposure modes.

[0071] Optionally, the above-mentioned correction of the exposure center time based on the trigger timestamp of the target trigger event and the trigger jitter parameters to obtain the target timestamp of the image frame may include: Optionally, the target timestamp can be the sum of the trigger timestamp, exposure center time, trigger jitter parameter, environmental compensation parameter, and line scan delay compensation in the camera parameters. Specifically, the target timestamp can be calculated using the following formula (2).

[0072] t ts =t trigger +t center +t linedelay +t δjitter +γ teny Formula (2) Among them, t trigger For trigger timestamp, t center For the exposure center time, t δjitter To trigger the jitter parameter, γ teny For environmental compensation parameters, t linedelay For line scan delay compensation in camera parameters, t ts For the target timestamp.

[0073] In this embodiment, the target timestamp for each image frame is based on the trigger timestamp of the target trigger event of that image frame, and is obtained by compensating for factors such as optical exposure curve, trigger jitter, and system noise, which significantly improves the accuracy of the acquisition timestamps of each image frame. Figure 6 A flowchart illustrating the fifth multi-camera synchronization control method provided in this application embodiment is shown below. Figure 6 As shown, the method may further include: S501. Obtain multiple event pairs, the actual trigger interval for each event pair, and the expected trigger interval.

[0074] Each event pair includes two adjacent historical trigger events, which refer to all trigger events generated in the trigger event queue. The actual trigger interval for each event pair refers to the time difference between the trigger timestamps recorded by the two adjacent historical trigger events, and the expected trigger interval for each event pair refers to the expected trigger time difference between the trigger signals corresponding to the two adjacent historical trigger events. The expected trigger time of the trigger signal corresponding to each trigger event is generated recursively based on the initial time and is used to characterize the expected output time of each trigger signal. For example, the expected trigger time sequence of the trigger signal corresponding to each trigger event is (T0, T0+T, T0+2T, ..., T0+NT), and the sequence of the actual trigger time of the trigger signal corresponding to each trigger event is, for example, (T0+δ1, T0+T+δ2, T0+2T+δ3, ..., T0+NT+δi), where δ is the signal deviation. The actual trigger time of the trigger signal corresponding to each trigger event refers to the trigger timestamp recorded in the trigger event.

[0075] For example, if the event pair is an adjacent historical trigger event 1 and a historical trigger event 2, the expected trigger time of historical trigger event 1 is T0, and the actual trigger time is T0+δ1; the expected trigger time of historical trigger event 2 is T0+T, and the actual trigger time is T0+T+δ2. Then the actual trigger interval for this event pair is T0+T+δ2-(T0+δ1)=T+δ2-δ1, and the expected trigger interval is T0+T-(T0)=T.

[0076] S502. Determine the trigger error between the actual trigger interval and the preset trigger interval for each event pair, obtain the trigger error for each event pair, predict the predicted trigger error based on the changing trend of the trigger error for each event pair, and correct the expected trigger time of the next trigger signal for the current trigger signal based on the predicted trigger error, so as to obtain the target expected trigger time of the next trigger signal.

[0077] Optionally, the timestamp management unit can use the absolute value of the trigger error between the actual trigger interval and the preset trigger interval for each event pair as the trigger error for each event pair. For example, the trigger error between historical trigger event 1 and historical trigger event 2 is |T+δ2-δ1-T|=|δ2-δ1|. Predictions can be made based on these trigger errors to obtain the predicted trigger error. This predicted trigger error is then used to correct the expected trigger time of the next trigger signal following the current trigger signal, thereby obtaining the target expected trigger time for the next trigger signal. This ensures that the next trigger signal is triggered at this target trigger time, satisfying the requirements of microsecond-level time alignment and stable triggering.

[0078] In this embodiment, by predicting the trend of trigger error and correcting the expected trigger time of the next trigger signal based on the error trend, the target expected trigger time of the next trigger signal is made more accurate. This effectively suppresses the impact of instantaneous jitter on trigger accuracy, ensuring the microsecond-level stability and continuity of trigger time, and allowing the trigger event to be advanced or delayed to offset the deviation. This ensures that even under external interference or trigger interval fluctuations, the image time series of all cameras maintain high-precision synchronization and continuity, thus meeting the requirements of multi-camera synchronization control systems for microsecond-level time alignment and stable triggering.

[0079] Optionally, the process of the camera acquiring image frames and generating frame information may include: Optionally, the trigger signal can be parsed to obtain the signal sequence number, image frames can be acquired based on the signal sequence number, and a frame sequence number corresponding to the signal sequence number can be generated, which is then used as frame information. For example, the signal sequence number can be used as the frame sequence number of the image frame, or other methods can be used to establish the correspondence between each image frame and the signal sequence number.

[0080] In this embodiment, each camera generates a frame number corresponding to the trigger number when acquiring image frames, which facilitates the subsequent matching of the corresponding trigger events with the image frames of each camera from the trigger event queue.

[0081] Optionally, before receiving the image frames acquired and fed back by each camera according to the trigger signal in S102 above, the following may be included: The trigger signal is sent to the slave control device corresponding to each camera. If the trigger signal is the first trigger signal, each slave control device calculates the time deviation based on its local reception time and the trigger timestamp in the trigger event corresponding to the trigger signal, and performs alignment correction on the local reception time based on the time deviation to obtain the global timestamp corresponding to the first trigger signal. Each slave control device sends the trigger signal to the camera connected to the slave control device based on the global timestamp.

[0082] And / or, if the trigger signal is a trigger signal other than the first trigger signal, then the global timestamp corresponding to the trigger signal is determined according to the trigger sequence number, trigger timestamp and trigger period in the trigger event corresponding to the trigger signal, and each slave control device sends the trigger signal to the camera connected to the slave control device according to the global timestamp.

[0083] Optionally, after receiving the trigger event corresponding to the trigger signal, each slave control device writes the trigger event corresponding to the trigger signal into its local trigger event queue in sequence, and uses the trigger event as a unified reference for local reception time calculation and image binding.

[0084] In this embodiment, the local receiving time of each slave device is aligned and corrected based on the time deviation, so as to achieve accurate alignment of cross-device image frames on a unified time axis and meet the needs of multi-robot collaborative perception.

[0085] Optionally, the debugging and monitoring unit can also perform event continuity detection, event loss detection, and event priority sorting on each trigger event in the trigger event queue.

[0086] Optionally, the method may further include: After generating a trigger event corresponding to the trigger signal, if no image frame corresponding to the trigger event is received within a preset time, a timeout exception is determined, and the trigger event is discarded. Specifically, the debugging and monitoring unit can monitor the image frames. If no image frame corresponding to the trigger event is received within the preset time, a timeout exception is determined, meaning that no image frame is detected during the trigger. The trigger event is then deleted from the trigger event queue. Simultaneously, the exception of the trigger event can be recorded in the log, including the trigger sequence number, timestamp, and timeout duration.

[0087] If no matching trigger event is found for the image frame, the image frame is marked as having an invalid timestamp. This allows the image frame to be isolated, and an exception log is recorded, which includes the image frame sequence number and the image frame's reception time.

[0088] If the trigger period of the trigger signal is abnormal, the trigger period parameter will be automatically adjusted. A sliding window method can be used to detect the trigger period; if the trigger interval within the sliding window deviates from the preset period, the trigger period parameter will be automatically adjusted.

[0089] If there are consecutive missing events or discontinuous image frame numbers, the missing sequence number is recorded and a warning message is generated.

[0090] It can also perform jitter detection on the trigger timestamp of the triggered event. If the trigger timestamp deviation continues to increase within the window, it indicates that the trigger timestamp jitter is abnormal. Clock calibration or jitter compensation can be started, and a jitter abnormality log can be recorded. The jitter amplitude and jitter trend information can be added to the jitter abnormality log.

[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0092] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0093] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A multi-camera synchronization control method, characterized in that, The method includes: A single clock source in a multi-camera synchronization control system generates trigger signals for unified control of multiple cameras, generates trigger events corresponding to the trigger signals, and adds the trigger events to the trigger event queue in the order of generation. The trigger events include a trigger timestamp, a trigger sequence number, and camera parameters of each camera. The clock source provides time accuracy of at least microseconds. Each of the cameras receives image frames acquired and fed back by the trigger signal; The image frames fed back by each camera are matched with the corresponding trigger events in the trigger event queue to determine the target timestamp of the image frames of each camera.

2. The multi-camera synchronization control method according to claim 1, characterized in that, The generation of the trigger event corresponding to the trigger signal includes: Obtain the current system time and camera parameters of each camera, wherein the current system time is used to indicate the time preceding the actual output time of the trigger signal; The current system time is corrected based on the signal deviation to obtain the trigger timestamp in the trigger event corresponding to the trigger signal. The trigger sequence number of the trigger event is generated according to the sequence number of the previous trigger event of the trigger event. The trigger timestamp and the camera parameters of each camera are used as the trigger event.

3. The multi-camera synchronization control method according to claim 1, characterized in that, The step of matching the image frames fed back by each of the cameras with the corresponding trigger events in the trigger event queue to determine the target timestamp of the image frames of each camera includes: The trigger sequence number of each trigger event in the trigger event queue is traversed. For the current trigger sequence number, the current trigger sequence number is matched with the frame sequence number in the image frame. If the current trigger sequence number matches the frame sequence number, the trigger event corresponding to the current trigger sequence number is taken as the target trigger event that matches the image frame. Each frame sequence number of the image frame has a corresponding trigger sequence number. Based on the frame information of the image frame and the target trigger event, determine the target timestamp of the image frame; And / or, Traverse the trigger sequence number of each corresponding trigger event in the trigger event queue. If the trigger sequence number in the trigger event queue is empty, mark the image frame as unmatched.

4. The multi-camera synchronization control method according to claim 3, characterized in that, Determining the target timestamp of the image frame based on the frame information of the image frame and the target trigger event includes: The exposure center time of the image frame is determined based on the frame information of the image frame, and the exposure center time is corrected based on the trigger timestamp of the target trigger event and the trigger jitter parameters to obtain the target timestamp of the image frame.

5. The multi-camera synchronization control method according to claim 3 or 4, characterized in that, Also includes: Retrieve multiple event pairs, each of which includes two adjacent historical trigger events; Obtain the actual trigger interval and expected trigger interval for each event pair; The triggering error between the actual triggering interval and the expected triggering interval for each event pair is determined, and the triggering error for each event pair is obtained. The predicted triggering error is predicted based on the changing trend of the triggering error for each event pair. The expected triggering time of the next triggering signal is corrected based on the predicted triggering error, and the target expected triggering time of the next triggering signal is obtained.

6. The multi-camera synchronization control method according to any one of claims 1-4, characterized in that, The process of the camera acquiring image frames and generating frame information includes: The trigger signal is parsed to obtain the signal number in the trigger signal; Image frames are acquired based on the signal sequence number, and a frame sequence number corresponding to the signal sequence number is generated, and the frame sequence number is used as the frame information.

7. The multi-camera synchronization control method according to any one of claims 1-4, characterized in that, Before receiving the image frames acquired and fed back by each of the cameras according to the trigger signal, the process includes: The trigger signal is sent to the slave control device corresponding to each camera; If the trigger signal is the first trigger signal, each slave control device calculates the time deviation based on the local reception time of each slave control device and the trigger timestamp in the trigger event corresponding to the trigger signal, and performs alignment correction on the local reception time based on the time deviation to obtain the global timestamp corresponding to the first trigger signal. Each slave control device sends the trigger signal to the camera connected to the slave control device based on the global timestamp. And / or, if the trigger signal is a trigger signal other than the first trigger signal, then the global timestamp corresponding to the trigger signal is determined according to the trigger sequence number, trigger timestamp and trigger period in the trigger event corresponding to the trigger signal, and each slave control device sends the trigger signal to the camera connected to the slave control device according to the global timestamp.

8. The multi-camera synchronization control method according to any one of claims 1-4, characterized in that, Also includes: For each triggering event in the triggering event queue, perform event continuity detection, event loss detection, and event priority sorting; And / or, After generating a trigger event corresponding to the trigger signal, if no image frame corresponding to the trigger event is received within a preset time, a timeout exception is determined and the trigger event is discarded; If no matching trigger event is found for the image frame, the image frame is marked as having an invalid timestamp, and an exception log is recorded. If the triggering period of the trigger signal is abnormal, the parameter of the triggering period will be automatically adjusted. If there are consecutive missing events or discontinuous image frame numbers, the missing sequence number is recorded and a warning message is generated.

9. A multi-camera synchronous control system, characterized in that, include: Trigger control unit and timestamp management unit; A trigger control unit is used to generate a trigger signal for unified control of multiple cameras based on a single clock source in a multi-camera synchronization control system, generate trigger events corresponding to the trigger signals, and add the trigger events to a trigger event queue in the order of generation. The trigger events include a trigger timestamp, a trigger sequence number, and camera parameters of each camera. The clock source provides a time accuracy of at least microseconds. The timestamp management unit is used to receive image frames acquired and fed back by each of the cameras according to the trigger signal; The timestamp management unit is used to match the image frames fed back by each camera with the corresponding trigger events in the trigger event queue to determine the target timestamp of the image frames of each camera.

10. The multi-camera synchronization control system according to claim 9, characterized in that, The multi-camera synchronous control system further includes: an image frame binding unit, a debugging and monitoring unit, and a configuration and parameter management unit; The image frame binding unit is used to receive image frames from each camera and bind the target timestamp information of each image frame to the image frame metadata according to the frame number or queue order of each image frame. The debugging and monitoring unit is used to record the system operating status, trigger time deviation, timestamp generation results, and abnormal events. The configuration and parameter management unit is used to manage the number of cameras, triggering cycle, exposure parameter update strategy, and synchronization mode.

11. A robot working system, characterized in that, It includes multiple cameras and the multi-camera synchronization control system as described in any one of claims 9-10; The multiple cameras are all mounted on the robot, and the multi-camera synchronization control system is communicatively connected to the multiple cameras.

Images