Multi-sensor synchronous triggering method and system

Abstract

The multi-sensor synchronous triggering method comprises the following steps: initializing an FPGA (Field Programmable Gate Array) and equipment; fPGA pulse triggering configuration is carried out; pTP time synchronization of the radar is carried out; measuring and calibrating the starting delay of the FPGA; and issuing a trigger command to realize synchronization. The invention further provides a multi-sensor synchronous triggering system. The traditional method mostly depends on PPS signals of a GPS or pure PTP time synchronization, while the method combines the FPGA and the PTP, and also considers calculation of time pairs of different sensors and triggering time of the FPGA, so that the synchronization precision is improved, the problem of indoor failure of the GPS signals is solved, and a high-precision synchronization solution independent of the GPS is realized.

Description

Technical Field

[0001] The invention relates to the field of time synchronization technology, in particular to multi-sensor synchronization triggering technology. Background Art

[0002] With the continuous development of science and technology, sensor technology has been widely used in various fields, providing a rich data source for multi-source information fusion. In multi-sensor systems, time synchronization technology is a key link to achieve accurate information fusion. However, due to the differences in sensor types and technical characteristics, achieving time synchronization of multiple sensors has become a challenging task.

[0003] Specifically, some sensors, such as IMU (Inertial Measurement Unit) and cameras, support active triggering mechanisms, which can precisely control their data acquisition process through external signals or pulses. This active triggering method can ensure data acquisition at a specific moment, providing a reliable time reference for subsequent data fusion. However, not all sensors have the ability to be actively triggered. For example, radar sensors usually rely on PTP (Precision Time Protocol) for time synchronization. PTP is a protocol for network clock synchronization that provides precise time tags for radar data by ensuring the accuracy and consistency of timestamps between different devices.

[0004] At present, the common multi-sensor time synchronization methods in the field of SLAM (Simultaneous Localization and Mapping) are: 1) multiple sensors use PTP time synchronization to achieve data fusion; 2) through soft synchronization method; 3) use GPS to achieve time synchronization.

[0005] For example: The Chinese invention patent with publication number CN117176285A discloses a PTP time synchronization method and device based on FPGA, which includes exchanging PTP messages with a processor and recording the timestamp information in the PTP message; calculating the current offset of the local clock system according to the timestamp information; adjusting the clock frequency of the local clock system according to a preset standard based on the proportion of the offset; wherein exchanging PTP messages with the processor and recording the timestamp information in the PTP message is implemented based on FPGA. It should be noted that in this application, since FPGA has faster computing power, it can reduce the delay generated in the software processing process and further improve the synchronization accuracy. The Chinese invention patent with publication number CN114413885A discloses a time synchronization method and system based on multi-sensor fusion positioning. The method recursively derives the first positioning information of the vehicle at multiple times according to the acceleration and angular velocity output by the inertial measurement unit, and generates a recursive queue; at the reception time of each second positioning information, the corresponding second positioning information is aligned with the first positioning information in the recursive queue according to its sending time, so as to obtain the first positioning information and the second positioning information at the same time; the Kalman filter algorithm is used to predict and solve the first positioning information and the second positioning information at each time between the set historical time and the current time, and the Jacobian matrix is ​​obtained; according to the Jacobian matrix, the Kalman filter algorithm is used to measure and update the first positioning information at the current time; the present invention can effectively solve the problem of asynchronous input time of multiple sensors during fusion positioning, and improve the accuracy of fusion positioning. The Chinese invention patent with publication number CN103616710A discloses a time synchronization method, system and device for multi-sensor fusion, which uses GPRMC (Global Positioning System Recommended Minimum data) messages and PPS (Pulse Per Second) signals to achieve time synchronization between lidar and UTC (Coodinated Universal Time), uses a unified trigger signal to achieve time synchronization between various optical devices, uses a compensated trigger signal to achieve time synchronization between optical devices and lidar, and uses a nearby timestamp alignment method to achieve time synchronization between IMU and lidar.

[0006] The current multi-sensor time synchronization methods are mainly divided into two categories, hardware-based time synchronization and software-based time synchronization. The hardware-based time synchronization method is more accurate, and trigger synchronization is performed on sensors that support hardware triggering, that is, using hardware triggers to directly control the sensor's framing frequency through physical signals. Common hardware triggering methods include timing pulse triggering and mechanical triggering. In simple terms, hard synchronization is to provide the same reference time to each sensor through a unique clock source. Each sensor calibrates its own clock time according to the provided reference time, and achieves time synchronization from the hardware. At present, many hardware-based time synchronization methods use the PPS method for synchronization, but this method requires GPS signals. This method will fail in places where satellite signals are blocked, such as indoors or in tunnels. Moreover, many sensors do not support hardware triggering.

[0007] In summary, some sensors currently do not support active triggering and cannot be triggered using FPGA pulses; soft synchronization is only a method of finding similar timestamps to determine whether the sensors are collecting data at the same time, which actually results in a large error; in certain scenarios, GPS signals may not be available, making it impossible to achieve time synchronization. Most importantly, none of the above methods trigger the sensors at the same time, so the collected data will always be biased.

[0008] Therefore, in a multi-sensor system, how to achieve accurate time synchronization between sensors that support active triggering and sensors that do not support active triggering has become an urgent problem to be solved. Summary of the invention

[0009] The technical problem to be solved by the present invention is how to support accurate time synchronization of sensors that actively trigger or do not actively trigger.

[0010] The present invention solves the above technical problems through the following technical means: a multi-sensor synchronous triggering method, comprising the following steps:

[0011] Step 1: FPGA and device initialization

[0012] First, the FPGA and the device that supports active triggering and the radar that does not support active triggering but supports PTP time synchronization are initialized, including initializing the clock source and pulse generation module of the FPGA, so that the device that supports active triggering and the device that does not support active triggering are ready to receive the trigger signal and start data acquisition;

[0013] Step 2: FPGA Pulse Trigger Configuration

[0014] Configure the FPGA to generate pulse signals of different frequencies to trigger devices that support active triggering, set the trigger frequency of devices that support active triggering, and ensure that the FPGA sends trigger signals simultaneously in the same clock cycle;

[0015] Step 3: PTP time synchronization of radar

[0016] The radar is time synchronized through PTP, and the transmission path delay and time offset between the processor and the radar are calculated. The transmission path delay is:

[0017] Delay=[(t4-t1)-(t3-t2)] / 2

[0018] Among them, t1 is the timestamp when the processor sends a synchronization message to the radar, t2 is the timestamp when the radar receives the synchronization message from the processor, t3 is the timestamp when the radar replies with a synchronization confirmation message to the processor, and t4 is the timestamp when the processor receives the synchronization confirmation message replied by the radar;

[0019] Time offset:

[0020] T offset =(t2-t1)-Delay=[(t2-t1)+(t3-t4)] / 2

[0021] T offset It is the time offset between the processor and the radar, which is used to compensate the processor clock;

[0022] Record the time from radar triggering, data collection to data processing completion, that is, the point cloud collection and processing time T l , and the data transfer time T for data to be transferred to the processor t ;

[0023] Step 4: Measure and Calibrate FPGA Startup Delay

[0024] Measure the time T from when the processor sends the instruction to control the FPGA start pulse to when the FPGA actually receives and starts executing the instruction f ;

[0025] Step 5: Send trigger commands to achieve synchronization, including

[0026] 5.1 Assume that the fixed frame rate of the laser radar is f frames / second, and the time interval T between each frame is expressed as

[0027] 5.2 Based on the frame rate and the timestamp T of the nth frame n , predict the time T of the next frame arrival n+1 :

[0028] T n+1 =T n +T+Delay+T offset ;

[0029] 5.3 Considering the point cloud acquisition and processing time Tl And the data transmission time T t , then the trigger time of FPGA is:

[0030] T tri =T n+1 -T l -T t -T f

[0031] In T tri When the processor issues a command to trigger the FPGA, all sensors are triggered at the same time.

[0032] As a further optimized technical solution, in step 1, the devices supporting active triggering include sensors IMU and cameras that support active triggering, and the radars that do not support active triggering but support PTP time synchronization include lidar and millimeter wave radar.

[0033] As a further optimized technical solution, in step 2, the trigger frequency of the IMU is 200 Hz and the trigger frequency of the camera is 25 Hz.

[0034] The present invention also provides a trigger system using the above-mentioned multi-sensor synchronous triggering method, comprising a processor, an FPGA and two types of triggering sensors, one type is a device that supports active triggering, and the other type is a radar that does not support active triggering but supports PTP time synchronization. The device that supports active triggering is connected to the FPGA, and the radar that does not support active triggering is connected to the processor, and the processor and the FPGA are connected.

[0035] As a further optimized technical solution, the device supporting active triggering includes sensors IMU and cameras that support active triggering, and the radar that does not support active triggering but supports PTP time synchronization includes lidar and millimeter wave radar.

[0036] As a further optimized technical solution, the trigger frequency of the IMU is configured to be 200 Hz, and the trigger frequency of the camera is configured to be 25 Hz.

[0037] The advantages of the present invention are: the trigger time of the IMU and the camera is accurately controlled by FPGA, and the radar is synchronized with the PTP protocol at the same time. After the radar is started, a suitable trigger time is selected, and the processor sends a trigger command to the FPGA to realize the FPGA triggering of the IMU and the camera, thereby realizing the precise synchronization of multiple sensor data. This method makes full use of the high-speed processing capability of the FPGA and the time synchronization accuracy of the PTP protocol, and effectively solves the problems of low synchronization accuracy and poor stability in traditional methods. Compared with traditional methods, this method can achieve physical simultaneous triggering, and is not affected by the environment, and high-precision time synchronization can be achieved indoors and outdoors. The present invention can be used in autonomous driving or other scenarios that require high-precision time synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of hardware synchronization in an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of synchronous triggering in an embodiment of the present invention;

[0040] Figure 3 It is a schematic diagram of the PTP time synchronization principle in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The multi-sensor synchronous triggering system of the present invention mainly includes a processor, an FPGA and two types of triggering sensors, one of which is an IMU and a camera that supports active triggering. Figure 1 The IMU, visual camera, and infrared camera on the right side of the middle. The other type is the radar that does not support active triggering but supports PTP time synchronization, such as Figure 1 The lidar and millimeter-wave radar in the upper middle. The IMU and camera that support active triggering are connected to the FPGA, while the radar that does not support active triggering is connected to the processor, and the processor is connected to the FPGA.

[0043] The IMU, camera and radar data synchronization method based on FPGA and PTP time synchronization can be divided into the following key steps:

[0044] Step 1: FPGA and device initialization

[0045] First, initialize the FPGA and related IMU, camera, and radar devices. This includes initializing the FPGA's clock source and pulse generation modules to ensure they are in the correct working state. At the same time, the IMU, camera, and radar devices also need to be initialized so that they are ready to receive trigger signals and start data acquisition.

[0046] Step 2: FPGA Pulse Trigger Configuration

[0047] Configure the FPGA to generate pulse signals of different frequencies to trigger the IMU and camera. Set the trigger frequency of the IMU to 200Hz and the trigger frequency of the Camera to 25Hz. Ensure that the FPGA sends these trigger signals at the same clock cycle to achieve simultaneous triggering of the IMU and camera.

[0048] Step 3: PTP time synchronization of radar

[0049] The radar is time synchronized through the Precision Time Protocol (PTP). PTP ensures that the radar data packets have accurate timestamps when transmitted over the network. Figure 3 , calculate the transmission path delay and time offset between the processor and the radar.

[0050] Transmission path delay:

[0051] Delay=[(t4-t1)-(t3-t2)] / 2

[0052] Among them, t1 is the timestamp when the processor sends a synchronization message to the radar, t2 is the timestamp when the radar receives the synchronization message from the processor, t3 is the timestamp when the radar replies with a synchronization confirmation message to the processor, and t4 is the timestamp when the processor receives the synchronization confirmation message replied by the radar.

[0053] Time offset:

[0054] T offset =(t2-t1)-Delay=[(t2-t1)+(t3-t4)] / 2

[0055] T offset It is the time offset between the processor and the radar, used to compensate the processor clock.

[0056] Record the time from radar triggering, data collection to data processing completion, that is, the point cloud collection and processing time T l , and the data transfer time T for data to be transferred to the processor t . This information will be used for subsequent synchronization calculations.

[0057] Step 4: Measure and Calibrate FPGA Startup Delay

[0058] Measure the time T from when the processor sends the instruction to control the FPGA start pulse to when the FPGA actually receives and starts executing the instruction f This delay time needs to be accurately measured through experiments and serves as an important parameter for subsequent synchronization calibration.

[0059] Step 5: Send a trigger command to achieve synchronization

[0060] 5.1 Assume that the fixed frame rate of the laser radar is f frames / second, and the time interval T between each frame is expressed as

[0061] 5.2 Based on the frame rate and the timestamp T of the nth frame n , predict the time T of the next frame arrival n+1 :

[0062] T n+1 =T n +T+Delay+T offset

[0063] 5.3 Considering the point cloud acquisition and processing time T l And the data transmission time T t , then the trigger time of FPGA is:

[0064] T tri =T n+1 -T l -T t -T f

[0065] In T tri When the processor issues a command to trigger the FPGA, all sensors can be triggered at the same time.

[0066] The time stamp error of the time synchronization method implemented by this method is approximately at the sub-millisecond level.

[0067] PPS synchronization pulses are usually generated by GPS signals, but they have two disadvantages: (1) If the GPS fails, the synchronization pulse may have errors. Normally, the GPS module receives the GPS signal before it can normally output the PPS signal and time signal. When using it, it is necessary to ensure that the GPS signal is stable. (2) The frequency of the PPS signal is 1 Hz. For some high-frequency sensors, the PPS signal must be converted into a high-frequency signal through FPGA to trigger the sensor.

[0068] The present invention does not use GPS signals, but uses PTP time synchronization radar, thus avoiding the above problems. The present invention mainly sets different trigger pulses through different pins of FPGA to directly trigger sensors that support active triggering. Assuming that the trigger frequencies of the camera and IMU are multiples, triggering these two sensors at the same initial moment can ensure that the specific frames of these two sensors are triggered at the same moment, which is more convenient and accurate when used. Figure 2 , the moment corresponding to the dotted line, when using the data at this moment, can be regarded as data generated at the same time; the synchronization of these two sensors is simple to implement, and how to synchronize the radar with these two sensors is the main innovation of the present invention. There is a time difference (generally fixed) between the triggering of the radar and the completion of data collection, which is sub-millisecond level. First start the PTP radar time synchronization, wait until the determined moment before the radar data collection is completed at a certain moment (this moment is calculated in the text), send a command to start the FPGA, at this time, all sensors are synchronized, and the data at the moment corresponding to the dotted line can be regarded as data collected at the same moment.

[0069] FPGA itself has a high-precision clock source, and the accuracy of PTP time synchronization is also very high. The present invention links these two synchronizations:

[0070] FPGA triggering: Accurate pulse triggering of sensors that support active triggering (such as IMU, camera) to ensure that these sensors can collect data at the same time. FPGA has a high-precision clock source and can flexibly generate pulse signals of different frequencies to meet the triggering requirements of different sensors.

[0071] PTP time synchronization: PTP protocol is used to synchronize the time of sensors that do not support active triggering (such as lidar and millimeter-wave radar). PTP's high-precision timestamp ensures the synchronization of these sensor data.

[0072] System-level integration: Combining FPGA pulse triggering and PTP time synchronization to achieve physical simultaneous triggering, thereby ensuring high-precision time synchronization of multi-sensor systems in different scenarios (such as indoors, tunnels, and other scenarios without GPS signals). This system overcomes the problems of traditional methods in GPS dependence, insufficient synchronization accuracy, and asynchronous data collection from multiple sensors.

[0073] The present invention takes into account the time alignment of different sensors: FPGA triggering of IMU and camera is relatively direct, but the time synchronization of radar depends on PTP, and it is necessary to consider the delay from radar data acquisition to processing completion, as well as the network transmission delay between the processor and radar. These delays must be calculated and compensated so that the trigger command is fully synchronized with the actual data acquisition process.

[0074] The present invention takes into account the calculation of the trigger time of FPGA: since radar data collection and processing require time, and FPGA triggering of IMU and camera requires precise timing control, it is necessary to predict the arrival time of the next frame based on the radar timestamp and accurately calculate the trigger time of FPGA to ensure that the data of IMU, camera and radar can be synchronized and aligned.

[0075] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0076] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensor synchronous triggering method, characterized in that: The steps include: Step 1: FPGA and device initialization First, the FPGA and the device that supports active triggering and the radar that does not support active triggering but supports PTP time synchronization are initialized, including initializing the clock source and pulse generation module of the FPGA, so that the device that supports active triggering and the device that does not support active triggering are ready to receive the trigger signal and start data acquisition; Step 2: FPGA Pulse Trigger Configuration Configure the FPGA to generate pulse signals of different frequencies to trigger devices that support active triggering, set the trigger frequency of devices that support active triggering, and ensure that the FPGA sends trigger signals simultaneously in the same clock cycle; Step 3: PTP time synchronization of radar The radar is time synchronized through PTP, and the transmission path delay and time offset between the processor and the radar are calculated. The transmission path delay is: Delay=[(t4-t1)-(t3-t2)] / 2 Among them, t1 is the timestamp when the processor sends a synchronization message to the radar, t2 is the timestamp when the radar receives the synchronization message from the processor, t3 is the timestamp when the radar replies with a synchronization confirmation message to the processor, and t4 is the timestamp when the processor receives the synchronization confirmation message replied by the radar; Time offset: T offset =(t2-t1)-Delay=[(t2-t1)+(t3-t4)] / 2 T offset It is the time offset between the processor and the radar, which is used to compensate the processor clock; Record the time from radar triggering, data collection to data processing completion, that is, the point cloud collection and processing time T l , and the data transfer time T for data to be transferred to the processor t ; Step 4: Measure and Calibrate FPGA Startup Delay Measure the time T from when the processor sends the instruction to control the FPGA start pulse to when the FPGA actually receives and starts executing the instruction f ; Step 5: Send trigger commands to achieve synchronization, including Step 5.1 Assume that the fixed frame rate of the lidar is f frames / second, and the time interval T between each frame is expressed as Step 5.2 Based on the frame rate and the timestamp T of the nth frame n , predict the time T of the next frame arrival n+1 : T n+1 =T n +T+Delay+T offset ; Step 5.3 Considering the point cloud acquisition processing time T l And the data transmission time T t , then the trigger time of FPGA is: T tri =T n+1 -T l -T t -T f In T tri When the processor issues a command to trigger the FPGA, all sensors are triggered at the same time.

2. A multi-sensor synchronous triggering method as claimed in claim 1, characterized in that: In step 1, the devices supporting active triggering include sensors IMU and cameras that support active triggering, and the radars that do not support active triggering but support PTP time synchronization include lidar and millimeter wave radar.

3. A multi-sensor synchronous triggering method as claimed in claim 2, characterized in that: In step 2, the trigger frequency of the IMU is 200 Hz, and the trigger frequency of the camera is 25 Hz.

4. A trigger system using the multi-sensor synchronous triggering method according to claim 1, characterized in that: The invention comprises a processor, an FPGA and two types of triggering sensors, one of which is a device supporting active triggering, and the other is a radar that does not support active triggering but supports PTP time synchronization. The device supporting active triggering is connected to the FPGA, and the radar that does not support active triggering is connected to the processor, and the processor and the FPGA are connected.

5. A multi-sensor synchronous triggering system as claimed in claim 4, characterized in that: The devices supporting active triggering include sensors IMU and cameras supporting active triggering, and the radars that do not support active triggering but support PTP time synchronization include laser radar and millimeter wave radar.

6. A multi-sensor synchronous triggering system as claimed in claim 5, characterized in that: The trigger frequency of the IMU is configured to be 200 Hz, and the trigger frequency of the camera is configured to be 25 Hz.

Images