Time synchronization method and system for multi-sensor fusion

Abstract

The application discloses a time synchronization method and system for multi-sensor fusion, and the method comprises the following steps: a positioning sensor sends a correction time to a computing platform, so that the computing platform corrects its system time according to the correction time; after the time correction, the computing platform generates PWM signals of multiple frequencies, respectively synchronously drives a laser radar, an inertial navigation and a camera to collect data, so as to realize time synchronization in multi-sensor fusion. In the application, the sensors synchronously collect data based on the computing platform, and collect data under a unified time reference, so that the multi-sensor data can be accurately synchronized. In addition, the time synchronization method does not need interpolation operation, can reduce the calculation amount and the occupation of the computing resources, and can meet the autonomous mobile task with high real-time performance.

Description

Technical Field

[0001] This invention relates to the field of time synchronization technology, and in particular to a time synchronization method and system for multi-sensor fusion. Background Technology

[0002] Currently, autonomous mobility technology continues to develop and mature, primarily applied in fields such as drones, autonomous driving, and robotic vacuum cleaners. As the market for autonomous mobility technology expands, perception, a crucial component, faces increasingly stringent requirements for stability and accuracy. Multi-source sensor fusion is an indispensable means to achieve robust and high-precision perception; for example, common combinations of LiDAR and visual sensors can provide rich and stable perception data at a relatively low cost. Time synchronization is a common and critical technical challenge when fusing sensor data. Different sensors typically have their own independent time sources with varying sampling frequencies and delays. This makes it impossible for subsequent algorithms to process data on a unified time base or match data from different sensors at the same moment. If data cannot be synchronized effectively, mismatches can occur during the fusion process, potentially leading to errors and impacting the overall performance of the algorithm system.

[0003] Current mainstream autonomous mobile devices typically achieve sensor time synchronization through software, that is, by using algorithms and protocols to correct and align the timestamps of data from different sensors: using the time of one of the sensors as a reference, and employing... Figure 1 The red matching block in the middle interpolates and matches the data frames of the nearest neighbors of other sensors to the current sensor data frame, thereby synchronizing the time of all sensors to the same time base.

[0004] However, existing time soft synchronization schemes have the following problems: (1) Although existing time soft synchronization schemes unify sensor data under the same time reference at the software level, the sensors collect data in their own time systems. Rough matching will force data collected at different times to be aligned to the same time, which will lead to non-negligible system errors; (2) Existing time soft synchronization schemes perform interpolation calculations between data at different times, which will significantly increase the amount of computation and occupy computing resources. This cannot meet the requirements of some autonomous mobile tasks with high real-time requirements, and it also cannot guarantee high precision; (3) Existing time soft synchronization schemes lack additional hardware support, and sensor data cannot be collected simultaneously. Summary of the Invention

[0005] This invention aims to at least partially solve one of the technical problems in related technologies. Therefore, the first objective of this invention is to provide a time synchronization method for multi-sensor fusion, in which each sensor synchronously collects data based on a computing platform and performs data collection under a unified time reference, achieving precise synchronization of multi-sensor data. Furthermore, this time synchronization method eliminates the need for interpolation calculations, reducing computational load and resource consumption, thereby meeting the requirements of high-real-time autonomous mobile tasks.

[0006] The second objective of this invention is to provide a time synchronization system.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0008] A time synchronization method for multi-sensor fusion is applied to a time synchronization system, the time synchronization system including a computing platform and multiple sensors, the multiple sensors including a positioning sensor, a lidar, an inertial navigation system, and a camera, each sensor being connected to the computing platform, the method comprising:

[0009] The positioning sensor sends a calibration time to the computing platform so that the computing platform can correct its own system time according to the calibration time;

[0010] After time correction, the computing platform generates PWM signals of various frequencies to synchronously drive the lidar, inertial navigation system, and camera to acquire data, thereby achieving time synchronization during multi-sensor fusion.

[0011] Preferably, the method further includes: the positioning sensor sending a PPS signal to the computing platform, so that the computing platform performs time correction based on the PPS signal at preset intervals according to the correction time.

[0012] Preferably, the system time of the computing platform includes the absolute time provided by the independent real-time clock and the relative time provided by the CPU clock. The computing platform corrects its own system time according to the correction time, including: after receiving the PPS signal, the computing platform clears the relative time provided by the CPU clock to zero and corrects the absolute time provided by the independent real-time clock to the correction time, so as to achieve time correction.

[0013] Preferably, the positioning sensor includes an RTK module, and the positioning sensor sends GPRMC messages to the computing platform through the RTK module to achieve the time-corrected transmission.

[0014] Preferably, the computing platform includes a PWM generator and a counter, and the computing platform generates PWM signals of multiple frequencies, including:

[0015] The computing platform generates a PWM signal of a preset frequency through the PWM generator;

[0016] The computing platform counts the PWM signal of a preset frequency using the counter, and then generates PWM signals of different frequencies based on the counting results.

[0017] Preferably, the built-in sampling frequency of the lidar is the same as the PWM signal drive frequency of the camera.

[0018] Preferably, the method further includes: the computing platform also sending timestamps to the lidar and the inertial navigation system.

[0019] Preferably, the method further includes: using the timestamp of the lidar as a reference, updating the timestamp of the camera itself using the timestamp of the lidar, so as to achieve timestamp alignment between the lidar and the camera.

[0020] Preferably, the specific steps for aligning the timestamps of the lidar and the camera include:

[0021] The preset number of data frames collected by the lidar are sequentially placed into a preset shared memory according to the acquisition time order;

[0022] The timestamp of the first data frame acquired by the camera is determined, and the closest matching timestamp value is found in the shared memory to update its own timestamp. The timestamp of the first data frame to enter the shared memory is removed. At the same time, the data of the most recently acquired data frame by the LiDAR is put into the shared memory in the order of acquisition time. The timestamp alignment steps are repeated to align the timestamps of the next data frame from the camera until the timestamps of all data frames from the camera are aligned with the timestamps of the data frames acquired by the LiDAR.

[0023] To achieve the above objectives, a second aspect of the present invention provides a time synchronization system, including a computing platform and multiple sensors, wherein the multiple sensors include a positioning sensor, a lidar, an inertial navigation system, and a camera, and each sensor is connected to the computing platform;

[0024] The positioning sensor is used to send a correction time to the computing platform so that the computing platform can correct its own system time according to the correction time.

[0025] The computing platform is used to generate PWM signals of multiple frequencies after time correction, which synchronously drive the lidar, inertial navigation and camera to acquire data, so as to achieve time synchronization during multi-sensor fusion.

[0026] This invention has at least the following technical effects:

[0027] 1. This invention uses a high-precision time source sent by the RTK module in the positioning sensor to perform time correction on the computing platform. Based on this, all sensors on the computing platform collect data under a unified time reference, and this unified time reference is a more accurate time reference after correction. Therefore, it can be ensured that each sensor collects data under a more accurate unified time reference. Furthermore, this invention also uses the PPS signal to enable the computing platform to update its own system time every preset time, thereby ensuring the accuracy of the unified time reference.

[0028] 2. This invention transmits PWM signals of multiple frequencies through hardware connection, enabling each sensor to trigger data acquisition simultaneously, thereby ensuring data acquisition synchronization. Furthermore, this invention ensures that the built-in sampling frequency of the lidar is the same as the PWM signal driving frequency of the camera, and that the PWM signal driving frequency is implemented through a PWM generator and counter during construction, which can guarantee precise synchronization of lidar and camera data.

[0029] 3. This invention performs more refined timestamp alignment on the data at the software level, thereby eliminating the slight data timestamp inconsistencies caused by time differences between program processes, thus achieving zero data time error. Furthermore, this invention does not require interpolation calculations, which reduces the amount of computation and the consumption of computing resources, enabling this time synchronization method to meet the requirements of highly real-time autonomous movement tasks.

[0030] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of an existing time soft synchronization scheme.

[0032] Figure 2 This is a schematic diagram of the time synchronization system structure according to an embodiment of the present invention.

[0033] Figure 3 This is a flowchart of a time synchronization method for multi-sensor fusion according to an embodiment of the present invention.

[0034] Figure 4 This is a schematic diagram of the hard synchronization principle in an embodiment of the present invention.

[0035] Figure 5 This is a schematic diagram illustrating the generation of PWM signals at different frequencies according to an embodiment of the present invention.

[0036] Figure 6 This is a schematic diagram illustrating the data synchronization between the lidar and the camera in an embodiment of the present invention.

[0037] Figure 7This is a schematic diagram of timestamp alignment according to an embodiment of the present invention. Detailed Implementation

[0038] The following describes this embodiment in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0039] The time synchronization method and system for multi-sensor fusion of this embodiment are described below with reference to the accompanying drawings.

[0040] The time synchronization method for multi-sensor fusion in this embodiment is applied to a time synchronization system, such as... Figure 2 As shown, it includes a computing platform and multiple sensors, including a positioning sensor, a lidar, an inertial navigation system, and a camera, all of which are connected to the computing platform.

[0041] Figure 3 This is a flowchart of a time synchronization method for multi-sensor fusion according to an embodiment of the present invention. Figure 3 As shown, the method includes:

[0042] Step S1: The positioning sensor sends the calibration time to the computing platform so that the computing platform can calibrate its own system time according to the calibration time.

[0043] In this embodiment, the positioning sensor includes an RTK (Real-Time Kinematic) module. The positioning sensor sends a GPRMC (Protocol Message) message to the computing platform through the RTK module. The GPRMC message includes a correction time. After receiving the correction time, the computing platform can correct its own system time according to the correction time.

[0044] In one embodiment of the present invention, the positioning sensor also sends a PPS (pulses per second) signal to the computing platform so that the computing platform can perform time correction based on the PPS signal at preset intervals according to the correction time.

[0045] The system time of the computing platform includes the absolute time provided by the independent real-time clock and the relative time provided by the CPU (Central Processing Unit) clock. The computing platform corrects its own system time according to the correction time, including: after receiving the PPS signal, the computing platform clears the relative time provided by the CPU clock to zero and corrects the absolute time provided by the independent real-time clock to the correction time, so as to achieve time correction.

[0046] Figure 4 This is a schematic diagram illustrating the time hard synchronization principle of an embodiment of the present invention. (See also...) Figure 4This embodiment discloses a sensor time hard synchronization scheme, mainly targeting the most common sensor combination of "GPS (Global Positioning System, including RTK module) + inertial navigation + LiDAR + camera". The scheme uses the RTK module as the clock source, ensuring global uniformity under this time reference. Simultaneously, the NVIDIA NX development board combined with the HYAI-311UAV switch serves as the computing platform, acting as the central hub for synchronizing the time of each sensor in this scheme.

[0047] In this embodiment, the RTK module transmits a 1Hz PPS signal through the switch's GPIO (General Purpose Input / Output Port) and a GPRMC message through the serial port. The NVIDIA Orin NX (computer), through an interrupt mechanism, once receives the PPS signal (detecting the rising edge of the PPS signal), immediately parses the serial port message, reads the GPRMC message, obtains the UTC (Coordinated Universal Time) time (i.e., the corrected time), and modifies the computer time to achieve 1Hz time correction.

[0048] It's important to note that computer time consists of two parts: the absolute time provided by the RTC CLOCK (Real-Time Clock, which has its own power supply) and the relative time provided by the CPU clock. The RTC CLOCK has lower precision but an independent power supply, and generally provides absolute time, unaffected by power outages. The CPU time has higher precision but lacks an independent power supply and stops counting when the computer is shut down, so it typically provides relative time. When the computing platform receives the PPS signal, the CPU time is reset to zero, and the absolute time provided by the RTCCLOCK is corrected to the UTC time in the GPRMC message, i.e., the corrected time.

[0049] Specifically, after receiving the PPS signal, the computing platform corrects the absolute time provided by the RTC CLOCK to UTC time. Since the normal PPS signal is 1Hz, the computing platform corrects the time every second. Because the UTC time in the GPRMC message comes from the satellite positioning system, its time accuracy is higher. Therefore, in this embodiment, the first layer is to use the computing platform as a time reference for data acquisition from other sensors; the second layer is to use the computing platform's system time to correct itself based on the UTC time in the GPRMC message for greater accuracy; and the third layer is to use the PPS signal to achieve automatic correction at 1-second intervals, which can prevent the cumulative time error from increasing due to the influence of the computing platform's own system time progress. Therefore, the combined effect of these three layers of technology can ensure the high accuracy of the unified time reference for all sensors.

[0050] Step S2: After time correction, the computing platform generates PWM signals of various frequencies to synchronously drive the lidar, inertial navigation and camera to acquire data, so as to achieve time synchronization during multi-sensor fusion.

[0051] In one embodiment of the present invention, the computing platform includes a PWM (Pulse Width Modulation) generator and a counter. The computing platform generates PWM signals of multiple frequencies by: generating a PWM signal of a preset frequency through the PWM generator; counting the PWM signal of the preset frequency through the counter, and generating PWM signals of different frequencies based on the counting results.

[0052] Back Figure 2 The lidar, inertial navigation system, and camera are all mounted on a computing platform and receive a 1Hz PPS signal simulated by a PWM signal to trigger data acquisition. The specific implementation process is as follows: Figure 5 As shown, the computing platform, whose time has been calibrated by the RTK module, uses a high-frequency clock (the high-frequency part of the CPU) to generate a higher frequency, i.e., a preset frequency, PWM signal (such as 1kHz). Therefore, 1ms corresponds to one PWM cycle. A counter is used to count the PWM signal to generate a PWM signal of a specific frequency. For example, to simulate a 1Hz PPS signal, one PPS pulse signal is generated when the counter reaches 1000. To simulate a 10Hz PPS signal (the normal PPS signal is 1Hz, and this 10Hz signal is simulated by the PWM signal), one PPS signal is generated when the counter reaches 100. Therefore, 10 PPS signals are generated in 1 second, completing the 10Hz control.

[0053] Specifically, a PWM generator first generates a PWM signal with a preset frequency, such as 1kHz, meaning one PWM signal occurs every 1ms. To simulate a 1Hz PPS signal, a counter is used. When the count reaches 1000 PWM signals, a PPS signal is sent once, i.e., one PPS signal is sent every 1 second to trigger the LiDAR to acquire data. Therefore, the simulated PPS signal is 1Hz, and the 1Hz PWM signal sent to the LiDAR is the simulated 1Hz PPS signal. When a 10Hz simulated PPS signal needs to be sent to the camera, a counter is used again. When the count reaches 100 PWM signals, a PPS signal is sent once, i.e., one PPS signal is sent every 0.1 seconds. Therefore, the simulated PPS signal is 10Hz, and the 10Hz PWM signal sent to the LiDAR is the simulated 10Hz PPS signal, which triggers the camera to acquire data.

[0054] Figure 5 The hardware-level signal simulation control shown ensures the possibility of different sensors acquiring data at the same time. For example, if the camera and LiDAR are designed to acquire data at the same frequency of 10Hz, and the acquisition is triggered simultaneously by a PWM signal, then each frame of the radar point cloud will have a completely corresponding image frame in terms of timestamp, which greatly improves the accuracy of subsequent point cloud processing.

[0055] In this embodiment, to achieve time synchronization during data fusion, the computing platform can simultaneously send PWM signals of different frequencies. The reason for the different frequencies is that, in order to ensure time synchronization, the built-in sampling frequency of the LiDAR and the PWM signal driving frequency of the camera need to be the same. For example, both the camera and the LiDAR collect data at the same frequency of 10Hz, but the LiDAR's frequency is the radar's built-in sampling frequency, while the camera's sampling frequency is the PWM signal frequency. Therefore, the aforementioned PWM generator and counter need to work together to generate the signal.

[0056] As an example, such as Figure 6 As shown, the computing platform sends a 1Hz PWM signal to the LiDAR to simulate a 1Hz PPS signal, triggering the LiDAR to acquire data. Therefore, the LiDAR receives this signal every 1 second and samples at its own sampling frequency of 10Hz, thus acquiring data at 0.1s, 0.2s, and so on, resulting in 0.1s, 0.2s, and so on, data frames of 1.0s. Simultaneously, the computing platform sends a 10Hz PWM signal to the camera to simulate a 10Hz PPS signal, triggering the camera to acquire data. Since the camera shutter triggers acquisition upon receiving a signal, the camera's PWM signal frequency is 10Hz. This means the computing platform sends a PPS signal to the camera every 0.1s. Upon receiving the signal, the camera acquires data, also acquiring data at 0.1s, 0.2s, and so on, resulting in 0.1s, 0.2s, and so on, data frames of 1.0s. This achieves time synchronization between the LiDAR and camera data frames.

[0057] In one embodiment of the present invention, the method further includes: the computing platform also sends timestamps to the lidar and the inertial navigation system, and updates the timestamps on the camera itself based on the timestamps of the lidar, so as to achieve timestamp alignment between the lidar and the camera.

[0058] In this embodiment, the specific steps for aligning the timestamps of the LiDAR and the camera include: placing a preset number of data frames collected by the LiDAR into a preset shared memory in the order of their collection time; determining the timestamp of the first data frame collected by the camera, and finding the timestamp with the closest matching timestamp value from the shared memory to update its own timestamp, and removing the timestamp that first entered the shared memory; simultaneously placing the most recently collected data frame data from the LiDAR into the shared memory in the order of its collection time, and repeating the timestamp alignment steps to align the timestamps of the next data frame from the camera, until the timestamps of all data frames from the camera are aligned with the timestamps of the data frames collected by the LiDAR.

[0059] The lidar and camera are triggered by a simulated PPS signal to simultaneously acquire data. After receiving the simulated 1Hz PPS signal, the lidar drives the process to automatically acquire point cloud frames at 10Hz. At the same time as the simulated 1Hz PPS signal is emitted, a 10Hz PWM signal is emitted. Each PWM signal triggers the camera to take a picture once, finally enabling the lidar and camera to complete synchronous trigger acquisition at 10Hz.

[0060] After each frame of point cloud and image data is synchronously acquired, the time accuracy during timestamp marking by the driving process is affected by CPU speed. Different processes will have different random delays (but they are of the same order of magnitude, generally around tens of microseconds during normal CPU operation). This results in slight differences in the timestamps of sensor data acquired at the same time. For example, when the LiDAR and camera transmit data to the computer, the CPU speed can cause the final timestamps of the two sensors to be misaligned. Furthermore, the camera itself adds its own timestamp, but this timestamp is not based on the computing platform. Therefore, the timestamps of point cloud frames and image frames need to be processed. Since the timestamps of LiDAR and inertial navigation data are both based on the computing platform, the timestamps of the camera data can be updated using the timestamps of the point cloud frames as a reference to achieve timestamp alignment between the LiDAR and camera. For example, 10 frames of point cloud data can be cached to form a queue. Each image frame performs nearest neighbor matching on its timestamp within the point cloud frame queue to achieve complete synchronization of the timestamps of each frame of LiDAR and camera data.

[0061] Figure 7 This is a diagram illustrating timestamp alignment. Figure 7 In this context, "lidar" refers to a lidar radar, "camera" refers to a camera, "newlidar frame" refers to a new lidar data frame (i.e., a point cloud frame), "lidar frame i" refers to the i-th point cloud frame, and "cameraframe" refers to an image frame acquired by the camera. For example... Figure 7 As shown, the 10 point cloud frames acquired by the LiDAR are first placed into a preset shared memory in the order of acquisition time. After the 10 point cloud frames (such as data frames corresponding to timestamps of 0.1006, 0.2007, etc.) are placed, the first data frame (image frame) acquired by the camera is taken. If its corresponding timestamp is 0.1007s, the shared memory is searched for the timestamp with the closest matching value. If 0.1006s is found to be the closest, then 0.1006s is used as the timestamp of the image frame to achieve timestamp nearest neighbor matching. Then, LiDAR frame 1, i.e., the point cloud frame corresponding to 0.1006s, is moved out of the shared memory, and LiDAR frame 11 is put into the shared memory to form a queue of 10 frames. Timestamp nearest neighbor matching is performed again until the timestamps of all data frames from the camera are aligned with the timestamps of the data frames acquired by the LiDAR.

[0062] Therefore, this embodiment performs more refined timestamp alignment on the data at the software level, thereby eliminating the slight data timestamp inconsistencies caused by time differences between program processes, thus achieving zero data time error. Furthermore, this invention does not require interpolation operations, which reduces the amount of computation and the consumption of computing resources, enabling this time synchronization method to meet the requirements of autonomous mobile tasks with high real-time requirements.

[0063] Furthermore, the present invention also provides a time synchronization system. For example... Figure 2 As shown, the time synchronization system includes a computing platform and multiple sensors, including a positioning sensor, a lidar, an inertial navigation system, and a camera, all of which are connected to the computing platform.

[0064] The positioning sensor sends the calibration time to the computing platform so that the computing platform can correct its own system time according to the calibration time. After time correction, the computing platform generates PWM signals of various frequencies to synchronously drive the lidar, inertial navigation and camera to collect data, so as to achieve time synchronization when multiple sensors are fused.

[0065] In one embodiment of the present invention, the positioning sensor is also used to send a PPS signal to the computing platform so that the computing platform can perform time correction based on the PPS signal at preset intervals according to the correction time.

[0066] In one embodiment of the present invention, the system time of the computing platform includes the absolute time provided by the independent real-time clock and the relative time provided by the CPU clock. When the computing platform corrects its own system time according to the correction time, it specifically performs the following: after receiving the PPS signal, it clears the relative time provided by the CPU clock to zero and corrects the absolute time provided by the independent real-time clock to the correction time, so as to achieve time correction.

[0067] In one embodiment of the present invention, the positioning sensor includes an RTK module, and the positioning sensor is also used to send GPRMC messages to the computing platform through the RTK module in order to achieve time-corrected transmission.

[0068] In one embodiment of the present invention, the computing platform includes a PWM generator and a counter. When the computing platform generates PWM signals of multiple frequencies, it is specifically used for: generating a PWM signal of a preset frequency through the PWM generator; counting the PWM signal of the preset frequency through the counter, and generating PWM signals of different frequencies based on the counting results.

[0069] In one embodiment of the present invention, the built-in sampling frequency of the lidar is the same as the PWM signal driving frequency of the camera.

[0070] In one embodiment of the invention, the computing platform is also used to send timestamps to the lidar and inertial navigation systems.

[0071] In one embodiment of the present invention, the computing platform is further configured to update the timestamp of the camera itself based on the timestamp of the lidar, so as to achieve timestamp alignment between the lidar and the camera.

[0072] In one embodiment of the present invention, when the computing platform realizes the timestamp alignment between the LiDAR and the camera, it specifically performs the following steps: It sequentially places a preset number of data frames collected by the LiDAR into a preset shared memory according to the collection time order; it determines the timestamp of the first data frame collected by the camera, searches the shared memory for the timestamp with the closest matching timestamp value, updates its own timestamp, removes the timestamp that first entered the shared memory, and simultaneously places the most recently collected data frame data from the LiDAR into the shared memory according to the collection time order, repeating the timestamp alignment steps to realize the timestamp alignment of the next data frame from the camera, until the timestamps of all data frames from the camera are aligned with the timestamps of the data frames collected by the LiDAR.

[0073] It should be noted that the specific implementation of the time synchronization system in this embodiment of the invention can be found in the specific implementation of the time synchronization method for multi-sensor fusion described above. To avoid redundancy, it will not be repeated here.

[0074] In summary, this invention uses a high-precision time source sent by the RTK module in the positioning sensor to perform time correction on the computing platform. Based on this, all sensors on the computing platform collect data under a unified time reference, and this unified time reference is a more accurate time reference after correction. Therefore, it can ensure that each sensor collects data under a more accurate unified time reference. Furthermore, this invention also uses the PPS signal to enable the computing platform to update its own system time at preset intervals, thereby ensuring the accuracy of the unified time reference. This invention transmits multiple frequencies of PWM signals through hardware connection, enabling all sensors to trigger data acquisition simultaneously. This invention ensures synchronized data acquisition and makes the built-in sampling frequency of the LiDAR the same as the PWM signal driving frequency of the camera. The PWM signal driving frequency is implemented through a PWM generator and a counter during construction, which guarantees precise synchronization of LiDAR and camera data. This invention performs further fine-grained timestamp alignment on the data at the software level, thereby eliminating the slight data timestamp discrepancies caused by time differences between program processes, thus achieving zero data time error. Furthermore, this invention does not require interpolation calculations, which reduces the amount of computation and the consumption of computing resources, enabling this time synchronization method to meet the requirements of high-real-time autonomous mobile tasks.

[0075] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A time synchronization method for multi-sensor fusion, characterized in that, An application is made in a time synchronization system, the time synchronization system including a computing platform and multiple sensors, the multiple sensors including a positioning sensor, a lidar, an inertial navigation system, and a camera, each sensor being connected to the computing platform, the method comprising: The positioning sensor sends a calibration time to the computing platform so that the computing platform can correct its own system time according to the calibration time; After performing time correction, the computing platform sends timestamps to the lidar and the inertial navigation system. Using the lidar timestamp as a reference, the platform updates the timestamp on the camera itself to align the timestamps of the lidar and the camera. It also generates PWM signals of various frequencies to synchronously drive the lidar, inertial navigation system, and camera to acquire data, thereby achieving time synchronization during multi-sensor fusion.

2. The time synchronization method for multi-sensor fusion as described in claim 1, characterized in that, The method further includes: The positioning sensor sends a PPS signal to the computing platform, so that the computing platform can perform time correction based on the PPS signal at preset intervals according to the correction time.

3. The time synchronization method for multi-sensor fusion as described in claim 2, characterized in that, The system time of the computing platform includes the absolute time provided by an independent real-time clock and the relative time provided by the CPU clock. The computing platform corrects its own system time according to the correction time, including: After receiving the PPS signal, the computing platform clears the relative time provided by the CPU clock to zero and corrects the absolute time provided by the independent real-time clock to the corrected time, thereby achieving time correction.

4. The time synchronization method for multi-sensor fusion as described in claim 1, characterized in that, The positioning sensor includes an RTK module, and the positioning sensor sends GPRMC messages to the computing platform through the RTK module to achieve the time-corrected transmission.

5. The time synchronization method for multi-sensor fusion as described in claim 1, characterized in that, The computing platform includes a PWM generator and a counter, and generates PWM signals of various frequencies, including: The computing platform generates a PWM signal of a preset frequency through the PWM generator; The computing platform counts the PWM signal of a preset frequency using the counter, and then generates PWM signals of different frequencies based on the counting results.

6. The time synchronization method for multi-sensor fusion as described in claim 1, characterized in that, The built-in sampling frequency of the lidar is the same as the PWM signal drive frequency of the camera.

7. The time synchronization method for multi-sensor fusion as described in claim 1, characterized in that, The specific steps for aligning the timestamps of the lidar and the camera include: The preset number of data frames collected by the lidar are sequentially placed into a preset shared memory according to the acquisition time order; The timestamp of the first data frame acquired by the camera is determined, and the closest matching timestamp value is found in the shared memory to update its own timestamp. The timestamp of the first data frame to enter the shared memory is removed. At the same time, the data of the most recently acquired data frame by the LiDAR is put into the shared memory in the order of acquisition time. The timestamp alignment steps are repeated to align the timestamps of the next data frame from the camera until the timestamps of all data frames from the camera are aligned with the timestamps of the data frames acquired by the LiDAR.

8. A time synchronization system, characterized in that, It includes a computing platform and multiple sensors, including a positioning sensor, a lidar, an inertial navigation system, and a camera, each of which is connected to the computing platform; The positioning sensor is used to send a correction time to the computing platform so that the computing platform can correct its own system time according to the correction time. The computing platform is used to send timestamps to the lidar and the inertial navigation system after time correction. Using the timestamp of the lidar as a reference, the platform updates the timestamp of the camera itself to achieve timestamp alignment between the lidar and the camera. It also generates PWM signals of various frequencies to synchronously drive the lidar, inertial navigation system and camera to acquire data, so as to achieve time synchronization during multi-sensor fusion.

Images