Vehicle sensor data multimode fusion time synchronization method

Abstract

The invention discloses a vehicle sensor data multi-mode fusion time synchronization method, and the main design concept of the method is as follows: calculating the timestamp difference of data collected by different sensors according to preset sensor characteristic parameters; according to the vehicle running speed and the timestamp difference, the dislocation distance is calculated; determining a fault-tolerant interval according to the dislocation interval, and solving a non-fixed time stamp threshold according to the fault-tolerant interval and the maximum driving speed of the vehicle; and then continuously acquiring sensor data, performing multimode data fusion in combination with the solved timestamp threshold value and the sensor characteristic parameters, and performing alignment fusion on the acquired data meeting the timestamp threshold value limiting condition according to the sampling frequency of the sensor. According to the method, a fixed timestamp threshold value is not set only depending on experience any more, and the problems that the multimode fusion soft synchronization timestamp threshold value cannot be effectively set, multimode fusion sampling data are seriously staggered and the like can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a vehicle sensor data multi-mode fusion time synchronization method. Background Art

[0002] In autonomous driving systems, multiple types of sensors are usually used to fully perceive the physical state information of the surrounding environment, so that the system can run more stably and in line with expectations. This multi-sensor fusion perception method is also commonly called multi-mode fusion.

[0003] However, the physical characteristics of different sensors vary greatly, and the operating frequencies of sensors are usually different. This results in a certain misalignment between the timestamps of different sensors and the sampling data of the surrounding environment when data fusion is performed.

[0004] To address this issue, engineers usually ignore the sampling data misalignment caused by sensor characteristics when performing multi-mode fusion. For timestamp misalignment, they usually set a default fixed threshold based on experience. This results in poor adaptability to different operating scenarios and makes it impossible to fundamentally achieve the optimal compatibility mode of multiple sensor fusion.

[0005] That is, currently, the time soft synchronization of multi-mode fusion is performed based on the empirical threshold set by experience, which may lead to the following problems:

[0006] (1) When a fixed empirical threshold is used for vehicles with different maximum speeds, the sampling data of different sensors will be too misaligned when the vehicle is traveling too fast, which will bring driving risks.

[0007] (2) If the empirical threshold is too small, the sensors will not be able to be fused, or frames will be lost before fusion can be performed (the current frame cannot reach the expected time threshold and needs to wait for the next frame). Summary of the invention

[0008] In view of the above, the present invention aims to provide a vehicle sensor data multi-mode fusion time synchronization method to solve the problems that the multi-mode fusion soft synchronization timestamp threshold cannot be effectively set and the multi-mode fusion sampling data is seriously misaligned.

[0009] The technical solution adopted by the present invention is as follows:

[0010] The present invention provides a vehicle sensor data multi-mode fusion time synchronization method, which includes:

[0011] Presetting sensor characteristics, wherein the sensor characteristics include a sampling frequency;

[0012] Calculating the timestamp difference of data collected by different sensors according to the sensor characteristics;

[0013] Calculate the misalignment distance based on the vehicle running speed and the time stamp difference of different sensors;

[0014] Determine the tolerance distance according to the misalignment distance, wherein the tolerance distance is the maximum allowable misalignment distance;

[0015] Calculate the timestamp threshold based on the fault tolerance interval and the maximum speed of the vehicle;

[0016] Continuously acquire sensor data, and perform multi-mode data fusion in combination with the timestamp threshold and the sensor characteristics: fuse the data that meets the timestamp threshold according to the sampling frequency of the sensor.

[0017] In at least one possible implementation manner, fusing the data satisfying the timestamp threshold according to the sampling frequency of the sensor includes:

[0018] If the sampling frequencies of the sensors are different, and the timestamp of the sensor data with a higher sampling frequency meets the timestamp threshold of data alignment, data alignment is performed;

[0019] If the sampling frequencies of the sensors are the same and the timestamps of the sensor data meet the timestamp threshold for data alignment, data alignment is performed.

[0020] In at least one possible implementation, the multimodal data fusion further includes:

[0021] If the sampling frequencies of the sensors are different, and the timestamp of the sensor data with a higher sampling frequency does not meet the timestamp threshold of data alignment, the sensor data with a higher sampling frequency is discarded; and when the total time of data frames continuously discarded by the sensor data with a higher sampling frequency is greater than the sampling time interval of the sensor data with a lower sampling frequency, the current data of the sensor with a lower sampling frequency is discarded;

[0022] If the sampling frequencies of sensors are the same and the timestamps of sensor data do not meet the timestamp threshold for data alignment, the sensor data with the smaller timestamp is discarded.

[0023] In at least one possible implementation manner, the offset distance=vehicle running speed×time stamp difference.

[0024] In at least one possible implementation manner, the timestamp threshold = error tolerance interval ÷ maximum vehicle speed.

[0025] In at least one possible implementation manner, obtaining the timestamp threshold includes: adding a time correction value preset based on sensor characteristics to the calculated timestamp threshold to obtain a final timestamp threshold.

[0026] In at least one possible implementation manner, the fault tolerance interval is constrained based on sensor types, sensor hardware parameters, the number of sensors, and preset sensor data usage conditions.

[0027] In at least one possible implementation, the timestamp difference is calculated by:

[0028] Based on the first start timing moment, obtaining a timestamp of the Nth frame of data of the first sensor;

[0029] Based on the second start timing moment, obtaining the timestamp of the Mth frame data of the second sensor; wherein the start timing moment does not exceed the time interval of one frame data of each sensor;

[0030] The absolute value of the difference between the two data timestamps is used as the timestamp difference.

[0031] Compared with the prior art, the main design concept of the present invention is to calculate the timestamp difference of the data collected by different sensors according to the pre-set sensor characteristic parameters; calculate the misalignment spacing according to the vehicle running speed and the timestamp difference; determine the fault tolerance spacing by the misalignment spacing, and obtain the non-solidified timestamp threshold according to the fault tolerance spacing and the maximum vehicle speed; then continuously obtain sensor data, and perform multi-mode data fusion in combination with the obtained timestamp threshold and sensor characteristic parameters, and align and fuse the collected data that meets the timestamp threshold limit conditions according to the sampling frequency of the sensor. The present invention no longer relies solely on experience to set a fixed timestamp threshold, and can effectively solve the problems of the inability to effectively set the timestamp threshold of multi-mode fusion soft synchronization and the serious misalignment of multi-mode fusion sampling data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described below with reference to the accompanying drawings, in which:

[0033] Figure 1 A schematic diagram of a vehicle sensor data multi-mode fusion time synchronization method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0035] The present invention proposes an embodiment of a vehicle sensor data multi-mode fusion time synchronization method. Specifically, Figure 1 shown, including:

[0036] Step S1, presetting sensor characteristics, wherein the sensor characteristics include a sampling frequency;

[0037] To elaborate, you can first set the characteristic data of the sensor according to the hardware parameters of the sensor, and you can set it according to the hardware parameters of various sensor devices. For example, the sampling frequency of the lidar is 10HZ and the resolution is 1cm; the sampling frequency of the camera is 30HZ; the sampling frequency of the IMU is 200HZ, etc.

[0038] Step S2, calculating the timestamp difference of data collected by different sensors according to the sensor characteristics;

[0039] The specific calculation method can be referred to as follows (in milliseconds). It should be noted that the following are only examples of two types of sensing devices:

[0040] Assume that the start time of camera data collection is T C , the camera's Nth frame data timestamp is:

[0041]

[0042] Where N is the current data frame number; 1000 / 30, i.e. 1000ms / 30Hz, represents the time interval of one frame of data collected by the camera;

[0043] Assume that the start time of radar data collection is T L , the timestamp of the Mth frame data of the radar is:

[0044]

[0045] Where M is the current data frame number; 1000 / 10, i.e. 1000ms / 10Hz, represents the time interval of one frame of data collected by the radar;

[0046] The start timing mentioned above must not exceed the time interval of one frame of data of each sensor device, that is, T C and T L The following constraints are met:

[0047]

[0048] From the above, the timestamp difference between the data collected by the radar and the camera can be calculated:

[0049]

[0050] Continuing from the above, step S3, calculating the misalignment distance according to the vehicle running speed and the time stamp difference of different sensors;

[0051] It should be understood that the sampling data of different sensor devices have a misaligned spacing. For example, when performing multi-mode fusion between visual data (camera) and radar data, the surrounding environment states sampled by the two sensors will be misaligned due to the moving speed of the vehicle and the actual distance deviation. That is, the misaligned spacing between the data collected by the two = vehicle running speed × timestamp difference T.

[0052] Step S4, determining a fault tolerance spacing for all sensors based on the misalignment spacing, wherein the fault tolerance spacing is a maximum allowable misalignment spacing;

[0053] Specifically, the fault tolerance distance can be the maximum allowable misalignment distance constrained by sensor categories, sensor hardware parameters, number of sensors and preset sensor data usage conditions. Among them, sensor categories include lidar, millimeter wave, and vehicle-mounted cameras; sensor hardware parameters include radar frequency, resolution, camera frequency, internal parameters, IMU frequency, accuracy, etc.; number of sensors include 1 or more radars, 1 or more cameras; sensor data usage conditions include voxelization of radar data, construction of occupancy grids, etc. The aforementioned maximum allowable misalignment distance cannot exceed the value of the maximum constraint conditions of each sensor (taking the camera as an example, its target detection box generally allows a deviation of 3 pixel values), and the final value of the fault tolerance distance is the minimum value among the maximum constraints of multiple sensors.

[0054] Step S5, obtaining a timestamp threshold according to the fault tolerance interval and the maximum speed of the vehicle;

[0055] In order to ensure maximum adaptation, the fault tolerance interval is the maximum vehicle speed × timestamp threshold. The optimal timestamp threshold of the sensor can be calculated by this formula. Here, in combination with actual operations, a time correction value preset based on sensor characteristics can be added to the calculated timestamp threshold to obtain the final timestamp threshold. In this preferred embodiment, the purpose of adding the time correction value is to ensure that the fault tolerance function is provided on the basis of the optimal timestamp threshold, thereby avoiding the difficulty of timestamp alignment due to the volatility of system performance.

[0056] Step S6, continuously acquiring sensor data, and combining the timestamp threshold and the sensor characteristics to perform multi-mode data fusion: according to the sampling frequency of the sensor, the data that meets the timestamp threshold is fused.

[0057] The sensor sampling process can use hard synchronization to implement data sampling, or it can implement data sampling through non-hard synchronization, as long as it ensures that there is a timestamp of the sampling moment or the end of sampling. The present invention does not limit or elaborate on this, but can specifically explain:

[0058] The fusing of data satisfying the timestamp threshold according to the sampling frequency of the sensor includes:

[0059] If the sampling frequencies of the sensors are different, and the timestamp of the sensor data with a higher sampling frequency meets the timestamp threshold of data alignment, data alignment is performed;

[0060] If the sampling frequencies of the sensors are the same and the timestamps of the sensor data meet the timestamp threshold for data alignment, data alignment is performed.

[0061] It can also be added that the multi-mode data fusion further includes:

[0062] If the sampling frequencies of the sensors are different, and the timestamp of the sensor data with a higher sampling frequency does not meet the timestamp threshold of data alignment, the sensor data with a higher sampling frequency is discarded; and when the total time of data frames continuously discarded by the sensor data with a higher sampling frequency is greater than the sampling time interval of the sensor data with a lower sampling frequency, the current data of the sensor with a lower sampling frequency is discarded;

[0063] If the sampling frequencies of sensors are the same and the timestamps of sensor data do not meet the timestamp threshold for data alignment, the sensor data with the smaller timestamp is discarded.

[0064] In combination with the above embodiments, the method for implementing the soft synchronization of the fusion time of the radar and the camera can be referred to as follows:

[0065] The sampling frequency of the camera is set to 30 Hz, and the sampling frequency of the radar data is set to 10 Hz. It can be estimated that the time stamp difference between the camera and the radar is between 0 and 16.666 milliseconds, and the time deviation of adjacent data frames of the camera is between 33.333 milliseconds and 50 milliseconds (the time deviation of adjacent frames of the radar data is greater than 50 ms, which is not considered in this example). Therefore, the timestamp threshold for time synchronization is between 16.666 milliseconds and 33.333 milliseconds. Considering the fluctuation of system performance, the time correction threshold is increased based on the optimal timestamp threshold of 16.666 milliseconds. In this example, the final timestamp threshold can be 20 milliseconds.

[0066] Continuously obtain camera and radar data to determine whether the time difference between camera data and radar data is less than the final timestamp threshold. If so, perform fusion; if so, discard the camera data. Since the radar frequency is lower than the camera frequency, when the time cannot be aligned, discard the camera data. The overall idea is to align the sensor device with a higher sampling frequency to the sensor device with a lower sampling frequency.

[0067] In addition, according to the sampling frequency parameters of the two, if three consecutive frames are discarded (based on the multiple relationship of the sampling frequency of the two), the camera data cannot be aligned to the radar data. At this time, the radar data timestamp is abnormal, and the current frame radar data needs to be discarded, and the next frame radar data is re-acquired to align the timestamp with the new radar data frame, that is, the fusion process is performed again.

[0068] In summary, the main design concept of the present invention is to calculate the timestamp difference of the data collected by different sensors according to the pre-set sensor characteristic parameters; calculate the misalignment spacing according to the vehicle running speed and the timestamp difference; determine the fault tolerance spacing by the misalignment spacing, and obtain the non-solidified timestamp threshold according to the fault tolerance spacing and the maximum vehicle speed; then continuously acquire sensor data, and perform multi-mode data fusion in combination with the obtained timestamp threshold and sensor characteristic parameters, and align and fuse the collected data that meets the timestamp threshold limit conditions according to the sampling frequency of the sensor. The present invention no longer relies solely on experience to set a fixed timestamp threshold, and can effectively solve the problems that the multi-mode fusion soft synchronization timestamp threshold cannot be effectively set and the multi-mode fusion sampling data is seriously misaligned.

[0069] If the expressions expressing orientation are mentioned in the embodiments of the present invention, they are relative concepts based on the embodiments. In addition, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, wherein a, b, c can be single or multiple.

[0070] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings, but the above is only a preferred embodiment of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched into a variety of equivalent schemes by those skilled in the art without departing from or changing the design ideas and technical effects of the present invention; therefore, the present invention is not limited to the scope of implementation shown in the drawings, and all changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the specification and drawings, should be within the protection scope of the present invention.

Claims

1. A vehicle sensor data multi-mode fusion time synchronization method, characterized in that: include: Presetting sensor characteristics, wherein the sensor characteristics include a sampling frequency; Calculating the timestamp difference of data collected by different sensors according to the sensor characteristics; Calculate the misalignment distance based on the vehicle running speed and the time stamp difference of different sensors; Determine the tolerance distance according to the misalignment distance, wherein the tolerance distance is the maximum allowable misalignment distance; Calculate the timestamp threshold based on the fault tolerance interval and the maximum speed of the vehicle; Continuously acquire sensor data, and perform multi-mode data fusion in combination with the timestamp threshold and the sensor characteristics: fuse the data that meets the timestamp threshold according to the sampling frequency of the sensor.

2. The vehicle sensor data multi-mode fusion time synchronization method according to claim 1 is characterized in that: The fusing of data satisfying the timestamp threshold according to the sampling frequency of the sensor includes: If the sampling frequencies of the sensors are different, and the timestamp of the sensor data with a higher sampling frequency meets the timestamp threshold of data alignment, data alignment is performed; If the sampling frequencies of the sensors are the same and the timestamps of the sensor data meet the timestamp threshold for data alignment, data alignment is performed.

3. The vehicle sensor data multi-mode fusion time synchronization method according to claim 2 is characterized in that: The multi-mode data fusion further includes: If the sampling frequencies of the sensors are different, and the timestamp of the sensor data with a higher sampling frequency does not meet the timestamp threshold of data alignment, the sensor data with a higher sampling frequency is discarded; and when the total time of data frames continuously discarded by the sensor data with a higher sampling frequency is greater than the sampling time interval of the sensor data with a lower sampling frequency, the current data of the sensor with a lower sampling frequency is discarded; If the sampling frequencies of sensors are the same and the timestamps of sensor data do not meet the timestamp threshold for data alignment, the sensor data with the smaller timestamp is discarded.

4. The vehicle sensor data multi-mode fusion time synchronization method according to claim 1 is characterized in that: Dislocation distance = vehicle running speed × timestamp difference.

5. The vehicle sensor data multi-mode fusion time synchronization method according to claim 1 is characterized in that: Timestamp threshold = tolerance interval ÷ maximum vehicle speed.

6. The vehicle sensor data multi-mode fusion time synchronization method according to claim 1 is characterized in that: The obtaining of the timestamp threshold comprises: adding a time correction value preset based on the sensor characteristics to the calculated timestamp threshold to obtain a final timestamp threshold.

7. The vehicle sensor data multi-mode fusion time synchronization method according to claim 1 is characterized in that: The fault tolerance interval is constrained according to sensor types, sensor hardware parameters, the number of sensors, and preset sensor data usage conditions.

8. The vehicle sensor data multi-mode fusion time synchronization method according to any one of claims 1 to 7, characterized in that: The timestamp difference is calculated by: Based on the first start timing moment, obtaining a timestamp of the Nth frame of data of the first sensor; Based on the second start timing moment, obtaining the timestamp of the Mth frame data of the second sensor; wherein the start timing moment does not exceed the time interval of one frame data of each sensor; The absolute value of the difference between the two data timestamps is used as the timestamp difference.

Images