A multi-sensor time alignment method and apparatus

Abstract

The application provides a multi-sensor time alignment method and device. The method comprises the following steps: obtaining a reference time of a first type sensor; performing time alignment processing on each sensor of the first type sensor according to the reference time; matching a bidirectional nearest time parameter of a second type sensor according to the reference time; and obtaining a second time difference value between the bidirectional nearest time parameter and the reference time, so as to perform time alignment processing on the first type sensor and the second type sensor. The application effectively solves the defects of time mismatching of various different types of sensors caused by factors such as inherent sampling time deviation and non-deterministic delay of a transmission link, and effectively improves the data synchronization monitoring and data quality of the multi-sensor.

Description

Technical Field

[0001] This application belongs to the field of environmental sensing technology, specifically relating to a multi-sensor time alignment method and device. Background Technology

[0002] In the data acquisition process, time alignment is a fundamental and crucial technology for ensuring the accuracy of sensor data fusion and the reliability of subsequent decision-making and control. Data acquisition typically employs various heterogeneous sensors such as cameras, LiDAR, and inertial measurement units. Each sensor has its own independent data acquisition clock, different data transmission frequencies, and distinct data processing and transmission links. This heterogeneity in hardware and protocols leads to inherent deviations in the data at the moment of generation, making it impossible to naturally achieve strictly "simultaneous" acquisition.

[0003] Sensor data must pass through multiple stages, including the driver layer, middleware, and network transmission, before reaching the processing module. Each stage can introduce uncertain and varying delays. These delays include not only fixed transmission times but are also affected by factors such as system scheduling, network congestion, and fluctuations in computational load, causing the data's arrival time at the processing terminal to deviate significantly from its true physical sampling time. Common synchronization methods rely on timestamping each data frame; however, if each sensor uses an independent local clock, even if initially synchronized, cumulative errors will occur due to crystal oscillator drift. Timestamps may be generated at different stages, such as the data acquisition end, driver end, or application end, each representing a different physical time. Timestamps generated in the operating system's user space are affected by system scheduling, introducing millisecond-level uncertainties.

[0004] In summary, existing multi-sensor time alignment methods struggle to effectively overcome the challenges posed by hardware heterogeneity, nondeterministic latency, and clock skew in resource-constrained and dynamically changing embedded environments, and cannot reliably provide accurate time-correlated data for upper-layer fusion and decision-making algorithms. Summary of the Invention

[0005] To address the aforementioned technical problems, this application proposes a multi-sensor time alignment method and device.

[0006] Specifically, this application proposes a multi-sensor time alignment method, comprising: obtaining a reference time of a first type of sensor; performing time alignment processing on each sensor in the first type of sensor according to the reference time; matching the bidirectional nearest neighbor time parameter of a second type of sensor according to the reference time; and obtaining a second time difference between the bidirectional nearest neighbor time parameter and the reference time to perform time alignment processing on the first type of sensor and the second type of sensor.

[0007] In the above technical solution, by using the reference time of the first type of sensor as a basis and combining it with a bidirectional nearest neighbor matching mechanism, the nearest samples are searched on both sides of the reference time for alignment determination. Compared with unidirectional matching or arbitrary sample selection, this method can more accurately estimate the true time deviation between the data of each sensor and more accurately reflect the true state of the second type of sensor near the reference time. This provides accurate time-correlated heterogeneous sensor data for subsequent data fusion, significantly improving alignment accuracy. The use of a bidirectional nearest neighbor matching mechanism, rather than relying on a fixed time window or assuming uniform data arrival, effectively avoids arrival time jitter or delay caused by transmission link uncertainties, thus possessing adaptability to dynamic delays and data jitter, and improving the robustness of alignment. Time alignment processing of the first and second types of sensors using time differences can effectively reduce random errors caused by factors such as different sensor sampling frequencies and phase deviations, making the aligned data closer to the true physical synchronization state, thereby improving the accuracy of upper-layer sensing and positioning algorithms.

[0008] As one implementation, obtaining the reference time of the first type of sensor includes: using the exposure time of a preset sensor in the first type of sensor as the reference time.

[0009] By pre-designating a specific sensor from the first type of sensors as the reference source, inconsistencies in alignment results due to an uncertain reference source are avoided, thus improving the determinism and reproducibility of the alignment behavior. Furthermore, by using the preset exposure time of the sensor as the reference time, software scheduling delays and transmission uncertainties introduced by timestamping in the driver layer and middleware are avoided.

[0010] Furthermore, the step of performing time alignment processing on each sensor in the first type of sensor according to the reference time includes: obtaining a first time difference between the reference time and the exposure time of each other sensor in the first type of sensor; if the first time difference is less than a first preset time threshold, then it is determined that the exposure time of each other sensor is normally aligned with the reference time; if the first time difference is greater than or equal to the first preset time threshold and less than a second preset time threshold, then it is determined that the exposure time of each other sensor is incorrectly aligned with the reference time, and a time alignment detection error is reported.

[0011] The system uses a first preset time threshold and a second preset time threshold to determine whether an alignment error is present. When the first time difference is between the first preset time threshold and the second preset time threshold, it is determined to be an alignment error and actively reported. This allows for the effective and timely detection of time synchronization faults between similar sensors. The use of dual thresholds effectively tolerates minor time deviations caused by factors such as transmission jitter and timestamp accuracy limitations, avoiding frequent false alarms due to overly strict threshold settings, thereby reducing unnecessary interference from abnormal reporting to the normal operation of the system.

[0012] Furthermore, the step of performing time alignment processing on each sensor in the first type of sensor according to the reference time also includes: if the first time difference is greater than or equal to the second preset time threshold, then the sensing data of the other sensors at the exposure time is written into the cache queue as the sensing data of the next frame.

[0013] When the deviation between the exposure time of other sensors and the reference time reaches or exceeds a second preset time threshold, it indicates that the data frame no longer belongs to the same acquisition cycle as the current reference frame. Directly discarding it would result in the loss of useful information. By writing the data frame into the buffer queue and marking it as the next frame of sensing data, alignment can be re-determined when the subsequent reference frame arrives, thereby maximizing the retention of effective sensor data and avoiding data waste caused by time deviations exceeding the threshold. This allows the system to adapt to fluctuations in the arrival order of sensor data or delays in the arrival of individual frames, effectively handling abnormal conditions such as data disorder and occasional delays, and significantly enhancing the robustness of time alignment.

[0014] Furthermore, the step of matching the bidirectional nearest neighbor time parameter in the second type of sensor according to the reference time includes: obtaining a first time parameter in the sensing data of the second type of sensor that is earlier than the reference time and has the smallest time difference, and a second time parameter that is later than the reference time and has the smallest time difference; obtaining the time parameter with the smallest time difference between the first time parameter and the second time parameter as the bidirectional nearest neighbor time parameter.

[0015] By acquiring a first time parameter that is earlier than the reference time and has the smallest time difference, and a second time parameter that is later than the reference time and has the smallest time difference, the sampling points of the second type of sensor on both sides of the reference time can be completely covered on the time axis. This avoids the matching deviation caused by relying on data from only one direction, making the characterization of the true state of the sensor near the reference time more accurate and comprehensive. Selecting the time parameter with the smallest time difference as the bidirectional nearest neighbor time parameter ensures that the matched data samples are closest to the reference time in the time dimension, thereby minimizing the time alignment error introduced by differences in sensor sampling frequency and phase shift.

[0016] Furthermore, the step of obtaining the second time difference between the bidirectional nearest neighbor time parameter and the reference time to perform time alignment processing on the first type of sensor and the second type of sensor includes: obtaining the second time difference between the bidirectional nearest neighbor time parameter and the reference time; if the second time difference is less than or equal to the second preset time threshold, then the sensing data of the bidirectional nearest neighbor time parameter of the second type of sensor is used as the second type of sensing data aligned with the reference time; otherwise, the sensing data of the bidirectional nearest neighbor time parameter of the second type of sensor is determined to be the sensing data of the next frame and written into the buffer queue.

[0017] By comparing the bidirectional nearest neighbor time parameter of the second type of sensor with the second time difference of the reference time and the second preset time threshold in a unified manner, the alignment between sensors of the same type and the alignment between sensors of different types adopt the same time deviation standard, thus ensuring the coordination and consistency of the time alignment judgment logic.

[0018] Furthermore, after performing time alignment processing on the first type of sensor and the second type of sensor, the method further includes: obtaining the time interval between any two frames of data in the time-aligned second type of sensor; and determining that the time alignment quality is abnormal when the time interval is greater than a preset time interval threshold.

[0019] After completing the time alignment process, continuous monitoring of the time interval between aligned data frames from the second type of sensor allows for judgment not only during the data matching stage but also after the alignment results are output. This enables the timely detection of potential anomalies that were not identified during the alignment process. When the time interval between any two frames exceeds a preset threshold, it indicates that there may be frame loss, sampling interruption, or incorrect data matching in the aligned data stream, thus effectively capturing such anomalies.

[0020] Furthermore, after performing time alignment processing on the first type of sensor and the second type of sensor, the method further includes: updating the reference time when the exposure time of the preset sensor in the first type of sensor changes; and performing time alignment processing on the data frames in the buffer queue using the updated reference time.

[0021] By updating the reference time, the latest sensor sampling time can be used as a time reference, ensuring that the time alignment process can continuously follow the actual acquisition rhythm of sensor data, thus guaranteeing the real-time and continuous nature of the alignment process.

[0022] Furthermore, after performing time alignment processing on the first type of sensor and the second type of sensor, the method further includes: obtaining the reference difference between the reference time and the updated reference time; if the reference difference is greater than a preset reference difference threshold, determining that the first type of sensor has reference frame loss and clearing the cache queue; otherwise, determining that the first type of sensor does not have reference frame loss.

[0023] By calculating the difference between two adjacent reference times and comparing it with a preset reference difference threshold, the system can proactively identify whether the first type of sensor has dropped or skipped frames during reference updates. When a sensor fails to collect data on time due to hardware failure, transmission interruption, or software scheduling delay, the reference time interval will abnormally increase. This system can promptly detect such abnormal states and avoid continuing invalid alignment operations when the reference is missing.

[0024] Based on the same inventive concept, this application also proposes an electronic device, including a processor and a memory; the memory is coupled to the processor, and the memory is used to store computer program code, the computer program code including computer instructions; when the processor reads the computer instructions from the memory, the electronic device executes the multi-sensor time alignment method.

[0025] Compared with the prior art, this application has at least the following beneficial effects: This application uses the reference time of the first type of sensor as a basis, combined with a bidirectional nearest neighbor matching mechanism, to find the nearest samples on both sides of the reference time for alignment determination. Compared with unidirectional matching or arbitrary sample selection, it can more accurately estimate the true time deviation between the data of each sensor, and more accurately reflect the true state of the second type of sensor near the reference time. This provides accurate time-correlated heterogeneous sensor data for subsequent data fusion, significantly improving alignment accuracy. Employing a bidirectional nearest neighbor matching mechanism instead of relying on a fixed time window or assuming uniform data arrival effectively avoids arrival time jitter or delay caused by transmission link uncertainties, thus possessing adaptability to dynamic delays and data jitter, and improving alignment robustness. Time alignment processing of the first and second type of sensors using time differences can effectively reduce random errors caused by factors such as different sensor sampling frequencies and phase deviations, making the aligned data closer to the true physical synchronization state, thereby improving the accuracy of upper-layer sensing and positioning algorithms. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a multi-sensor time alignment method according to an embodiment of this application.

[0027] Figure 2This is a flowchart illustrating the time alignment process for thread wake-up in an embodiment of this application.

[0028] Figure 3 This is a flowchart illustrating camera time alignment in an embodiment of this application.

[0029] Figure 4 This is a flowchart illustrating lidar time alignment in an embodiment of this application.

[0030] Figure 5 This is a flowchart illustrating the time alignment of the IMU and the camera as shown in an embodiment of this application.

[0031] Figure 6 This is a flowchart illustrating the time alignment of the IMU and the lidar as shown in an embodiment of this application.

[0032] Figure 7 This is a flowchart illustrating the time alignment of cameras across rounds, as shown in an embodiment of this application.

[0033] Figure 8 This is a flowchart illustrating the time alignment of a lidar across rounds, as shown in an embodiment of this application.

[0034] Figure 9 This is a flowchart illustrating cross-cycle time alignment of the camera / LiDAR and IMU as shown in an embodiment of this application.

[0035] Figure 10 This is a schematic diagram of an electronic device shown in an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0037] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. Example 1:

[0038] Please refer to Figure 1 The multi-sensor time alignment method mainly includes steps S100 to S400.

[0039] Step S100 includes: acquiring a reference time for a first type of sensor. This first type of sensor includes, but is not limited to, cameras and lidar. It can be applied in areas such as intelligent driving, obstacle detection, and active parking of vehicles, and the exposure time of a selected camera or lidar can be used as the reference time.

[0040] Preferably, obtaining the reference time of the first type of sensor includes: using the exposure time of a preset sensor in the first type of sensor as the reference time.

[0041] When applied in a vehicle, the surround-view rear camera in the vehicle's camera array can be used as the preset sensor, and the exposure time of the surround-view rear camera can be used as the reference time; or the front lidar in the vehicle's lidar array can be used as the preset sensor, and the exposure time of the front lidar can be used as the reference time.

[0042] Step S200 includes: performing time alignment processing on each sensor in the first type of sensor according to the reference time.

[0043] Preferably, the step of performing time alignment processing on each sensor in the first type of sensor according to the reference time includes: obtaining a first time difference between the reference time and the exposure time of each other sensor in the first type of sensor; if the first time difference is less than a first preset time threshold, then determining that the exposure time of each other sensor is normally aligned with the reference time; if the first time difference is greater than or equal to the first preset time threshold and less than a second preset time threshold, then determining that the exposure time of each other sensor is incorrectly aligned with the reference time, and reporting a time alignment detection error.

[0044] The first preset time threshold can be, but is not limited to, 10ms, and the second preset time threshold can be, but is not limited to, 50ms. When the first type of sensor is a camera, the exposure time of the surround-view rear camera is used as the reference time. A first time difference value, timeDiff = |camera exposure time - reference time|, is obtained between the exposure time of other cameras in the camera set and the reference time. If this first time difference value is less than 10ms, it is determined that the exposure time of the camera and the exposure time of the surround-view rear camera are in the same frame, meaning that the data frame of the camera is properly aligned with the data frame of the reference time. If the first time difference value is greater than or equal to 10ms and less than 50ms, it is determined that the data frame of the camera has an alignment error with the data frame of the reference time.

[0045] When the first type of sensor is a lidar, the exposure time of the previous lidar is used as the reference time. A first time difference value, timeDiff = |exposed time of other lidars – lidar reference time|, is obtained between the reference time and the exposure time of other lidars. If this first time difference is less than 10ms, it is determined that the exposure time of the lidar and the reference time are in the same frame, meaning the data frame of the lidar is correctly aligned with the data frame of the reference time. If the first time difference is greater than or equal to 10ms and less than 50ms, it is determined that the data frame of the lidar has an alignment error with the data frame of the reference time.

[0046] Preferably, the step of performing time alignment processing on each sensor in the first type of sensor according to the reference time further includes: if the first time difference is greater than or equal to the second preset time threshold, then the sensing data of the other sensors at the exposure time is written into the cache queue as the sensing data of the next frame.

[0047] For example, when the first type of sensor is a camera, if the first time difference is greater than or equal to 50ms, it is determined that the data frame of the camera is not in the same frame as the data frame of the reference time, and the data frame of the camera is used as the next data frame and written into the buffer queue. When the first type of sensor is a LiDAR, if the first time difference is greater than or equal to 50ms, it is determined that the data frame of the LiDAR is not in the same frame as the data frame of the reference time, and the data frame of the LiDAR is used as the next data frame and written into the buffer queue.

[0048] Step S300 includes: matching the bidirectional nearest neighbor time parameters of a second type of sensor according to the reference time. The second type of sensor includes, but is not limited to, an IMU (Inertial Measurement Unit).

[0049] Preferably, the step of matching the bidirectional nearest neighbor time parameter in the second type of sensor according to the reference time includes: obtaining a first time parameter that is earlier than the reference time and has the smallest time difference, and a second time parameter that is later than the reference time and has the smallest time difference in the sensing data of the second type of sensor; obtaining the time parameter with the smallest time difference between the first time parameter and the second time parameter as the bidirectional nearest neighbor time parameter.

[0050] For example, when the first type of sensor is a camera, the camera's reference time is obtained, and this reference time is matched with the sensing data from the second type of sensor. The time parameter of the nearest data frame in the second type of sensor whose time is earlier than the camera's reference time is obtained as the first time parameter, and the time parameter of the nearest data frame in the second type of sensor whose time is later than the camera's reference time is obtained as the second time parameter. The one of the first time parameter and the second time parameter that is closest to the reference time is used as the bidirectional nearest neighbor time parameter.

[0051] And step S400 includes: obtaining a second time difference between the bidirectional nearest neighbor time parameter and the reference time, so as to perform time alignment processing on the first type of sensor and the second type of sensor.

[0052] Preferably, obtaining the second time difference between the bidirectional nearest neighbor time parameter and the reference time to perform time alignment processing on the first type of sensor and the second type of sensor includes: obtaining the second time difference between the bidirectional nearest neighbor time parameter and the reference time; if the second time difference is less than or equal to the second preset time threshold, then using the sensing data of the bidirectional nearest neighbor time parameter of the second type of sensor as the second type of sensing data aligned with the reference time; otherwise, determining that the sensing data of the bidirectional nearest neighbor time parameter of the second type of sensor is the sensing data for the next frame and writing it into the buffer queue.

[0053] For example, if the second time difference is less than or equal to 50ms, the sensing data of the bidirectional nearest neighbor time parameter of the second type of sensor is taken as the second type of sensing data aligned with the reference time. That is, the sensing data of the bidirectional nearest neighbor time parameter is in the same frame as the data frame of the reference time. Otherwise, if the second time difference is greater than 50ms, it is determined that the sensing data of the bidirectional nearest neighbor time parameter is in the next frame of data to be checked and written into the buffer queue.

[0054] Preferably, after performing time alignment processing on the first type of sensor and the second type of sensor, the method further includes: acquiring the time interval between any two frames of data in the second type of sensor after time alignment processing; and determining that the time alignment quality is abnormal when the time interval is greater than a preset time interval threshold.

[0055] The preset time interval threshold can be 10ms, but it is not limited to this. For example, if the time interval between any two frames of data in the aligned IMU is greater than 10ms, the IMU alignment quality is considered abnormal.

[0056] Preferably, after performing time alignment processing on the first type of sensor and the second type of sensor, the method further includes: updating the reference time when the exposure time of the preset sensor in the first type of sensor changes; and performing time alignment processing on the data frames in the buffer queue using the updated reference time.

[0057] For example, when the first type of sensor is a camera or LiDAR, if the exposure time of the surround-view rear camera or front LiDAR changes, i.e., the reference time changes, the updated exposure time is used as the new reference time. Alignment detection is then performed on the data frames written to the buffer queue using the new reference time.

[0058] Preferably, after performing time alignment processing on the first type of sensor and the second type of sensor, the method further includes: obtaining the reference difference between the reference time and the updated reference time; if the reference difference is greater than a preset reference difference threshold, determining that the first type of sensor has reference frame loss and clearing the cache queue; otherwise, determining that the first type of sensor does not have reference frame loss.

[0059] The preset reference difference threshold can be 150ms, but it is not limited to this. That is, when the reference difference between the new reference time and the old reference time is greater than 150ms, it is determined that there is reference frame loss, and the data to be checked stored in the buffer queue is cleared. For example, when the reference difference between the old reference time and the new reference time of the camera is greater than 150ms, it is determined that the camera has reference frame loss, and the camera data to be checked stored in the buffer queue is cleared.

[0060] In the specific implementation process, please refer to Figure 2 When the IPC (Inter-Process Communication) channel receives sensor data, it triggers a time alignment processing task, writes the received sensor data into a buffer queue, and sends a signal to the condition variable m_threadCond to notify the waiting processing thread. After the processing thread is awakened, it checks whether the buffer queue is empty. If the buffer queue is empty, it means that it is a false wake-up, and the processing thread releases the mutex lock and returns to the waiting state to sleep. If the buffer queue is not empty, it retrieves the sensor data from the buffer queue and enters the time alignment processing flow.

[0061] When the sensing data for this buffer queue originates from a first-type sensor, please refer to... Figure 3 When the sensing data comes from the camera, Figure 4 When the sensing data originates from LiDAR, it determines whether a reference time needs to be found, i.e., whether the initial value of m_cameraBaseTime is 0. If a reference time needs to be found, it iterates through all sensing data in the cache queue to find the preset reference sensor. Figure 3 Central Rear View Cameras and Figure 4 The front-mounted LiDAR uses the exposure time of the preset reference sensor's sensing data to set the reference time m_cameraBaseTime or m_LidarBaseTime.

[0062] If a reference time is not required, then iterate through all the sensed data in the cache queue and determine whether the read sensed data originates from the preset reference sensor. If the currently read sensed data does not originate from the preset reference sensor, but from another sensor in the first type of sensor, Figure 3 The data is derived from other cameras, such as the front camera and the reversing camera. The exposure time of the current perceived data, measuretime, is calculated, and the time difference between the current exposure time and the base time m_cameraBaseTime is calculated. Figure 4 The data is derived from other LiDAR sensors. The exposure time (measuretime) of the current sensing data is calculated, and the time difference between it and the reference time (m_LidarBaseTime) is determined. If the time difference is within a first preset time threshold, the current sensing data is considered to be properly aligned with the reference time. If the time difference is greater than or equal to the first preset time threshold and less than 50ms, the current sensing data is considered to be incorrectly aligned with the reference time. If the time difference is greater than or equal to 50ms, the current sensing data is considered to be from a different frame than the reference time, and the current sensing data is written into the buffer queue as the next frame to be detected.

[0063] Please refer to Figure 5 and Figure 6 The second type of sensor is time-synchronized using the reference time of the first type of sensor; that is, Figure 5 Camera and IMU time synchronization detection and Figure 6The LiDAR and IMU are synchronized for time detection. By iterating through the sampling times of the second type of sensor (i.e., the sampling times of the IMU) to its nearest neighbors that are earlier or later than the reference time, the `MostLowProximate` variable is updated to store the nearest neighbor with the smallest time difference from the reference time. Similarly, for nearest neighbors that are later than the reference time but with the smallest time difference from the reference time, the `MostUpProximate` variable is updated to store the nearest neighbor with the smallest time difference from the reference time. If the difference between the nearest neighbor and the reference time is greater than 50ms, the nearest neighbor is written to the cache queue `m_lastRoundlmuCamData` or `m_lastRoundlmuLidarData`. The two nearest points that are earlier and later than the reference time are compared. If the time difference between the two nearest points is greater than 10ms, the time alignment between the first type sensor and the second type sensor is determined to be abnormal. If the time difference between the two nearest points is less than or equal to 10ms, the time alignment between the first type sensor and the second type sensor is determined to be qualified.

[0064] Figure 3 and Figure 4 If no reference time is needed, then all sensing data in the cache queue are traversed to determine whether the reading sensing data comes from the preset reference sensor. When the current reading sensing data comes from the preset reference sensor, that is, from the surround view rear camera or the front LiDAR, the reference time of the preset reference sensor is checked to see if it has changed. If it has changed, the reference time is updated.

[0065] The updated baseline time is used to perform cross-round time alignment detection on the data to be detected in the cache queue, such as... Figure 7 The diagram shows the process for camera cross-round time alignment detection. After the camera's base time is updated, the process iterates through `m_lastRoundlmuCamData` and calculates the time difference (`timeDiff`) between the time of the current sensed data read from `m_lastRoundlmuCamData` and the updated base time. If this time difference (`timeDiff`) is still >= 50ms, the cross-round count is incremented. If the cross-round count is still less than the threshold, `m_lastRoundlmuCamData` is written again; otherwise, an alarm is issued and the current sensed data is discarded. Please refer to [link / reference]. Figure 8The flowchart for LiDAR cross-round time alignment detection is as follows: After the LiDAR's reference time is updated, the time difference (timeDiff) between the time of the current sensing data read from m_lastRoundlmuLidarData and the updated reference time is calculated by iterating through m_lastRoundlmuLidarData. If the time difference (timeDiff) is still >= 50ms, the cross-round count is incremented. If the cross-round count is still less than the threshold, m_lastRoundlmuLidarData is written again; otherwise, an alarm is issued and the current sensing data is discarded. Please refer to Figure 9 This is a flowchart illustrating the cross-round time alignment detection process between the camera / LiDAR and the IMU. It iterates through the next frame of data to be inspected in the buffer queue (m_lastRoundlmuCamData or m_lastRoundlmuLidarData), determining whether the data to be inspected and the updated reference time are in the same frame and whether the time difference is less than or equal to 50ms. If so, it updates MostLowProximate and MostUpProximate, then compares the two nearest neighbors earlier and later than the reference time. If the time difference between the two nearest neighbors is greater than 10ms, the time alignment between the first type of sensor and the second type of sensor is considered abnormal; if the time difference is less than or equal to 10ms, the time alignment quality between the first type of sensor and the second type of sensor is considered acceptable. When the data to be inspected and the updated reference time are not in the same frame and the time difference is greater than 50ms, the data to be inspected is written to the next frame buffer queue, and the cross-round count is recorded. When the cross-round count exceeds the threshold, a warning is issued and the frame data is discarded. The number of counts can be set according to actual needs and is not limited. For example, if the number of counts is set to 3, a warning will be issued and the frame data will be discarded when the number of counts across rounds exceeds 3.

[0066] When detected Figure 3 The surround-view rear camera, whose reference time is set in the center, is not properly aligned with the time of other cameras. Figure 4 The time alignment between the front-mounted LiDAR with its reference time set and other LiDARs is abnormal. Figure 5 The timing alignment between the IMU and the camera is abnormal. Figure 6 When the time alignment between the IMU and the lidar is abnormal, Figure 7 , Figure 8 as well as Figure 9 When time alignment is detected as abnormal during the mid-to-span round time alignment detection, the number of consecutive time alignment abnormalities is recorded. When the number of consecutive time alignment abnormalities reaches 3, a time alignment abnormality detection report is submitted. Example 2:

[0067] Please refer to Figure 10 This application also proposes an electronic device including a processor and a memory; the memory is coupled to the processor and is used to store computer program code, the computer program code including computer instructions; when the processor reads the computer instructions from the memory, the electronic device executes the multi-sensor time alignment method as described in Embodiment 1.

[0068] The processor can be any suitable processing device or set of processing devices, such as, but not limited to, a microprocessor, a microcontroller-based platform, an integrated circuit, one or more field-programmable gate arrays (FPGAs) and / or one or more application-specific integrated circuits (ASICs).

[0069] Memory can be volatile memory (e.g., RAM including non-volatile RAM, magnetic RAM, ferroelectric RAM, etc.), non-volatile memory (e.g., disk storage, flash memory, EPROM, EEPROM, memristor-based non-volatile solid-state memory, etc.), immutable memory (e.g., EPROM), read-only memory, and / or high-capacity storage devices (e.g., hard disk drives, solid-state drives, etc.). In some examples, memory includes multiple types of memory, particularly volatile and non-volatile memory. Memory is a computer-readable medium on which one or more computer programs (such as software for operating the methods of this disclosure) can be embedded. The computer program can embody one or more of the methods or logic described herein. For example, the computer program resides wholly or at least partially within any one or more of memory, computer-readable media, and / or within a processor during execution.

[0070] The electronic device may also include a camera, LiDAR, IMU, and a communication module. The camera can be an RGB camera based on visible light imaging, an infrared camera based on infrared imaging, an event-driven camera, a depth camera, or any combination of the above types. In some examples, the exposure time of the surround-view rear-path camera in the camera array can be used as a reference time for the first type of sensor to drive the data from other cameras in the camera set and for time alignment processing with the data from the second type of sensor.

[0071] LiDAR can be pulsed lidar, frequency-modulated continuous wave coherent lidar, triangulation-based structured light lidar, or any combination of these types. In some examples, lidar includes mechanically rotating lidar, hybrid solid-state lidar, microelectromechanical system (MEMS) micromirror lidar, and optical phased array lidar. LiDAR is used to acquire three-dimensional point cloud data of the surrounding environment. Each frame of point cloud data it generates is associated with an exposure timestamp or scan start timestamp, indicating the moment when the lidar completes a full scan or acquires a frame of data. In this application, the exposure time of the vehicle's front lidar can be used as the reference time for the first type of sensor, driving the data from other lidars and time-aligned with the data from the second type of sensor.

[0072] An inertial measurement unit (IMU) can be an IMU based on a fiber optic gyroscope, an IMU based on a ring laser gyroscope, an IMU based on a quartz accelerometer, or any combination thereof. In some examples, the IMU may have different performance parameters such as sampling frequency, measurement range, bias instability, noise density, and bandwidth. In some examples, the IMU is used to continuously acquire the body motion state data of a robot or vehicle, and each frame of inertial data it generates is associated with a sampling timestamp. In this application, the IMU, as a second type of sensor, is time-aligned with the reference time of the first type of sensor through a bidirectional nearest neighbor matching strategy. The inertial data acquired by the IMU is transmitted to the processing module via a serial interface or bus. Because its sampling frequency is usually much higher than that of cameras and lidar, there may be cases where multiple frames of IMU data correspond to one frame of image or point cloud data during time alignment processing.

[0073] The communication module can be an interface module based on a wired communication protocol, a transceiver module based on a wireless communication protocol, or any combination of the above types. In some examples, the communication module includes wired communication components such as Ethernet interface modules, controller area network bus interface modules, serial peripheral interface bus modules, universal asynchronous transceiver transmitter interface modules, universal serial bus interface modules, and low-voltage differential signal interface modules. The communication module is used to establish data transmission links between various sensors, processing units, and other subsystems of the robot or vehicle, transmitting the sensed data collected by each sensor and related timestamp information from the data source to the processing terminal. In this application, the transmission delay and uncertainty of the communication module are important factors causing the sensor data arrival time to deviate from its true physical sampling time. The time alignment method of this application effectively overcomes the impact of the nondeterministic delay introduced by the communication module on the accuracy of time alignment through cross-round buffering and reference time update mechanisms.

[0074] In summary, this application uses the reference time of the first type of sensor as a basis and combines it with a bidirectional nearest neighbor matching mechanism to find the nearest samples on both sides of the reference time for alignment determination. Compared with unidirectional matching or arbitrary sample selection, this method can more accurately estimate the true time deviation between the data of each sensor and more accurately reflect the true state of the second type of sensor near the reference time. This provides accurate time-correlated heterogeneous sensor data for subsequent data fusion, significantly improving alignment accuracy. The use of a bidirectional nearest neighbor matching mechanism, rather than relying on a fixed time window or assuming uniform data arrival, effectively avoids arrival time jitter or delay caused by transmission link uncertainties, thus possessing adaptability to dynamic delays and data jitter, and improving the robustness of alignment. Time alignment processing of the first and second type of sensors using time differences can effectively reduce random errors caused by factors such as different sensor sampling frequencies and phase deviations, making the aligned data closer to the true physical synchronization state, thereby improving the accuracy of upper-layer sensing and positioning algorithms.

[0075] In the several embodiments provided in this application, it will be understood that each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0076] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application for those skilled in the art.

Claims

1. A multi-sensor time alignment method, characterized in that, include: Obtain the reference time for the first type of sensor; Time alignment processing is performed on each sensor in the first type of sensor according to the reference time; Match the bidirectional nearest neighbor time parameters of the second type of sensor according to the reference time; In addition, a second time difference between the bidirectional nearest neighbor time parameter and the reference time is obtained to perform time alignment processing on the first type of sensor and the second type of sensor.

2. The multi-sensor time alignment method according to claim 1, characterized in that, The acquisition of the reference time for the first type of sensor includes: The exposure time of the preset sensor in the first type of sensor is used as the reference time.

3. The multi-sensor time alignment method according to claim 1, characterized in that, The time alignment process for each sensor in the first type of sensor according to the reference time includes: Obtain the first time difference between the reference time and the exposure time of each of the other sensors in the first type of sensor; If the first time difference is less than the first preset time threshold, it is determined that the exposure time of the other sensors is normally aligned with the reference time. If the first time difference is greater than or equal to the first preset time threshold and less than the second preset time threshold, then it is determined that the exposure time of the other sensors is misaligned with the reference time, and a time alignment detection error is reported.

4. The multi-sensor time alignment method according to claim 3, characterized in that, The step of performing time alignment processing on each sensor in the first type of sensor according to the reference time further includes: If the first time difference is greater than or equal to the second preset time threshold, then the sensing data of the other sensors at the exposure time are written into the cache queue as the sensing data of the next frame.

5. The multi-sensor time alignment method according to claim 1, characterized in that, The step of matching the bidirectional nearest neighbor time parameters of the second type of sensor according to the reference time includes: Acquire a first time parameter that is earlier than the reference time and has the smallest time difference from the sensing data of the second type of sensor, and a second time parameter that is later than the reference time and has the smallest time difference. The time parameter with the smallest time difference between the first time parameter and the second time parameter is obtained as the bidirectional nearest neighbor time parameter.

6. The multi-sensor time alignment method according to claim 3, characterized in that, The step of obtaining the second time difference between the bidirectional nearest neighbor time parameter and the reference time to perform time alignment processing on the first type of sensor and the second type of sensor includes: Obtain the second time difference between the bidirectional nearest neighbor time parameter and the reference time; If the second time difference is less than or equal to the second preset time threshold, then the sensing data of the bidirectional nearest neighbor time parameter of the second type of sensor is used as the second type of sensing data aligned with the reference time. Otherwise, the sensing data of the bidirectional nearest neighbor time parameter of the second type of sensor is determined as the sensing data of the next frame and written into the buffer queue.

7. The multi-sensor time alignment method according to claim 6, characterized in that, After performing time alignment processing on the first type of sensor and the second type of sensor, the process further includes: Acquire the time interval between any two frames of data from the second type of sensor after time alignment processing; When the time interval is greater than a preset time interval threshold, the time alignment quality is determined to be abnormal.

8. The multi-sensor time alignment method according to claim 1, characterized in that, After performing time alignment processing on the first type of sensor and the second type of sensor, the process further includes: When the exposure time of the preset sensor in the first type of sensor changes, the reference time is updated; Time alignment is performed on the data frames in the cache queue using the updated base time.

9. The multi-sensor time alignment method according to claim 8, characterized in that, After performing time alignment processing on the first type of sensor and the second type of sensor, the process further includes: Obtain the reference difference between the reference time and the updated reference time; If the reference difference is greater than the preset reference difference threshold, it is determined that the first type of sensor has reference frame loss, and the buffer queue is cleared; otherwise, it is determined that the first type of sensor does not have reference frame loss.

10. An electronic device, characterized in that, It includes a processor and a memory; the memory is coupled to the processor, and the memory is used to store computer program code, the computer program code including computer instructions; when the processor reads the computer instructions from the memory, the electronic device performs the multi-sensor time alignment method as described in any one of claims 1-9.

Images