Time synchronization method and system for multi-sensor fusion

Abstract

The invention discloses a time synchronization method and system for multi-sensor fusion, and the method comprises the steps: enabling a positioning sensor to transmit correction time to a calculation platform, so as to enable the calculation platform to correct the system time according to the correction time; after the calculation platform carries out time correction, PWM signals of various frequencies are generated, and the laser radar, the inertial navigation and the camera are synchronously driven to carry out data acquisition, so that time synchronization during multi-sensor fusion is realized. According to the method, the sensors synchronously collect data based on the computing platform, data collection is carried out under the unified time reference, accurate synchronization of the data of the multiple sensors can be achieved, in addition, the time synchronization method does not need interpolation operation, the calculation amount can be reduced, computing resource occupation can be reduced, and therefore the requirement for autonomous moving tasks with high real-time performance can be met.

Description

Technical Field

[0001] The present invention relates to the field of time synchronization technology, and in particular to a time synchronization method and system for multi-sensor fusion. Background Art

[0002] At present, autonomous mobile technology continues to develop and mature, and is mainly used in many fields such as drones, autonomous driving, and sweeping robots. With the continuous expansion of the application market of autonomous mobile technology, perception, as a key link, has continuously improved its stability and accuracy. Multi-source sensor fusion is an indispensable means to achieve high robustness and high-precision perception. For example, the common combination of lidar and visual sensors can provide rich and stable perception data at a relatively low cost. Time synchronization is a key technical problem commonly faced when fusing sensor data. Generally, different sensors have their own independent time sources and different sampling frequencies and delays, which makes it impossible for subsequent algorithms to process data under the same time reference and match the data of each sensor corresponding to the same time. If the data cannot be synchronized well, the data will be mismatched during the fusion process, resulting in errors, which will affect the overall performance of the algorithm system.

[0003] At present, mainstream autonomous mobile devices generally achieve sensor time synchronization at the software level, that is, to correct and align the timestamps of different sensor data through algorithms and protocols: taking the time of one of the sensors as the benchmark, using Figure 1 The red matching block in the figure interpolates the data frames of the nearest neighbors of other sensors to the current sensor data frame, thereby synchronizing all sensor times to the same time base.

[0004] However, existing time soft synchronization schemes have the following problems: (1) Although existing time soft synchronization schemes unify sensor data under the same time base at the software level, sensors collect data under their own time systems. Rough matching will force data collected at different times to be aligned to the same time, which will lead to non-negligible system errors. (2) Existing time soft synchronization schemes perform interpolation calculations between data at different times, which will significantly increase the amount of calculation and occupy computing resources. They cannot meet some autonomous mobile tasks with high real-time requirements, and they cannot guarantee high accuracy. (3) Existing time soft synchronization schemes lack additional hardware support, and sensor data cannot be collected simultaneously. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first object of the present invention is to provide a time synchronization method for multi-sensor fusion, in which each sensor synchronously collects data based on a computing platform and collects data under a unified time reference, so that accurate synchronization of multi-sensor data can be achieved. In addition, the time synchronization method does not require interpolation operations, which can reduce the amount of calculation and the occupation of computing resources, thereby meeting the autonomous mobile tasks with high real-time requirements.

[0006] A second object of the present invention is to provide a time synchronization system.

[0007] To achieve the above object, the present invention is implemented through the following technical solutions:

[0008] A time synchronization method for multi-sensor fusion is applied to a time synchronization system, wherein the time synchronization system includes a computing platform and a plurality of sensors, wherein the plurality of sensors include a positioning sensor, a laser radar, an inertial navigation system, and a camera, and each sensor is connected to the computing platform. The method includes:

[0009] The positioning sensor sends the correction time to the computing platform, so that the computing platform corrects its own system time according to the correction time;

[0010] After time correction, the computing platform generates PWM signals of multiple frequencies to synchronously drive the laser radar, inertial navigation and camera to collect data, so as to achieve time synchronization during multi-sensor fusion.

[0011] Preferably, the method further comprises: the positioning sensor sending a PPS signal to the computing platform, so that the computing platform performs time correction according to the correction time at preset intervals based on the PPS signal.

[0012] Preferably, the system time of the computing platform includes the absolute time provided by the independent real-time clock and the relative time provided by the CPU clock. The computing platform corrects its own system time according to the correction time, including: after the computing platform receives the PPS signal, the relative time provided by the CPU clock is cleared, and the absolute time provided by the independent real-time clock is corrected to the correction time to achieve time correction.

[0013] Preferably, the positioning sensor comprises an RTK module, and the positioning sensor sends a GPRMC message to the computing platform through the RTK module, so as to realize the sending of the correction time.

[0014] Preferably, the computing platform includes a PWM generator and a counter, and the computing platform generates PWM signals of multiple frequencies, including:

[0015] The computing platform generates a PWM signal of a preset frequency through the PWM generator;

[0016] The computing platform counts the PWM signal of a preset frequency through the counter, and regenerates PWM signals of different frequencies according to the counting result.

[0017] Preferably, the built-in sampling frequency of the laser radar is the same as the PWM signal driving frequency of the camera.

[0018] Preferably, the method further includes: the computing platform also sends a timestamp to the laser radar and the inertial navigation.

[0019] Preferably, the method further includes: taking the timestamp of the laser radar as a reference, updating the timestamp of the camera itself through the timestamp of the laser radar, so as to achieve the alignment of the timestamps of the laser radar and the camera.

[0020] Preferably, the specific steps of realizing the timestamp alignment between the laser radar and the camera include:

[0021] Put a preset number of data frames collected by the laser radar into the preset shared memory in sequence according to the collection time;

[0022] Determine the timestamp of the data frame first collected by the camera, and search the shared memory for the timestamp with the closest matching timestamp value to update its own timestamp, and remove the timestamp that first enters the shared memory. At the same time, put the data frame data collected by the lidar most recently into the shared memory according to the collection time sequence, and repeat the timestamp alignment step to achieve the timestamp alignment of the next data frame of the camera, until the timestamps of all data frames of the camera are aligned with the timestamps of the data frames collected by the lidar.

[0023] To achieve the above object, the second aspect of the present invention provides a time synchronization system, including a computing platform and a plurality of sensors, wherein the plurality of sensors include a positioning sensor, a laser radar, an inertial navigation system and a camera, and each sensor is connected to the computing platform;

[0024] The positioning sensor is used to send the correction time to the computing platform, so that the computing platform can correct its own system time according to the correction time;

[0025] The computing platform is used to generate PWM signals of multiple frequencies after time correction, and synchronously drive the laser radar, inertial navigation and camera to collect data, so as to achieve time synchronization during multi-sensor fusion.

[0026] The present invention has at least the following technical effects:

[0027] 1. The present invention uses a high-precision time source sent by the RTK module in the positioning sensor to perform time correction on the computing platform. When collecting data, each sensor based on this computing platform is under a unified time reference, and the unified time reference is a more accurate time reference after correction, so it can ensure that each sensor collects data under a more accurate unified time reference; and the present invention also uses the PPS signal to enable the computing platform to update its own system time at preset intervals, thereby ensuring the accuracy of the unified time reference.

[0028] 2. The present invention transmits PWM signals of multiple frequencies through hardware connection, so that each sensor triggers data collection at the same time, thereby ensuring the synchronization of data collection. The present invention also makes the built-in sampling frequency of the laser radar and the PWM signal driving frequency of the camera the same, and the PWM signal driving frequency is realized through a PWM generator and a counter during construction, which can ensure the accurate synchronization of the laser radar and camera data.

[0029] 3. The present invention further aligns the data timestamps at the software level, thereby eliminating the slight data timestamp misalignment caused by the different times between program processes, and further making the data time error-free. In addition, the present invention does not require interpolation operations, which can reduce the amount of calculation and the use of computing resources, so that the time synchronization method can meet the needs of autonomous mobile tasks with high real-time requirements.

[0030] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the existing time soft synchronization solution.

[0032] Figure 2 The figure is a schematic diagram of the structure of a time synchronization system according to an embodiment of the present invention.

[0033] Figure 3 The present invention is a flowchart of a time synchronization method for multi-sensor fusion according to an embodiment of the present invention.

[0034] Figure 4 This is a schematic diagram of the time hard synchronization principle of an embodiment of the present invention.

[0035] Figure 5 Schematic diagram of generating PWM signals of different frequencies according to an embodiment of the present invention.

[0036] Figure 6 Schematic diagram of laser radar and camera data synchronization according to an embodiment of the present invention.

[0037] Figure 7A schematic diagram of timestamp alignment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present embodiment is described in detail below, and examples of the embodiment are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0039] The time synchronization method and system for multi-sensor fusion of this embodiment will be described below with reference to the accompanying drawings.

[0040] The time synchronization method for multi-sensor fusion of this embodiment is applied to a time synchronization system. Figure 2 As shown, it includes a computing platform and multiple sensors, the multiple sensors include positioning sensors, lidar, inertial navigation and cameras, and each sensor is connected to the computing platform.

[0041] Figure 3 FIG. 1 is a flow chart of a time synchronization method for multi-sensor fusion according to an embodiment of the present invention. Figure 3 As shown, the method includes:

[0042] Step S1: The positioning sensor sends the correction time to the computing platform so that the computing platform corrects its own system time according to the correction time.

[0043] In this embodiment, the positioning sensor includes an RTK (real-time kinematic) module. The positioning sensor sends a GPRMC (a protocol message) message to the computing platform through the RTK module. The GPRMC message includes a correction time. After receiving the correction time, the computing platform can correct its own system time according to the correction time.

[0044] In one embodiment of the present invention, the positioning sensor further sends a PPS (pulse per second) signal to the computing platform, so that the computing platform performs time correction according to the correction time at preset intervals based on the PPS signal.

[0045] The system time of the computing platform includes the absolute time provided by the independent real-time clock and the relative time provided by the CPU (central processing unit) clock. The computing platform corrects its own system time according to the correction time, including: after the computing platform receives the PPS signal, the relative time provided by the CPU clock is cleared, and the absolute time provided by the independent real-time clock is corrected to the correction time to achieve time correction.

[0046] Figure 4 FIG. 1 is a schematic diagram of the time hard synchronization principle of an embodiment of the present invention. Figure 4This embodiment discloses a sensor time hard synchronization solution, mainly for the most common "GPS (Global Positioning System, including RTK module) + inertial navigation + laser radar + camera" sensor combination. The solution uses the RTK module to provide the clock source, and the global time is unified under this time reference. At the same time, the nvidiaorin nx development board combined with the HYAI-311UAV switch is used as the computing platform, which also serves as the central hub for synchronizing the time of each sensor in this solution.

[0047] In this embodiment, the RTK module transmits a 1Hz PPS signal through the switch GPIO (general input / output port) port and transmits a GPRMC message through the serial port. Nvidia orin nx, i.e., the computer, once receives the PPS signal (detects the rising edge of the PPS signal) through the interrupt mechanism, it immediately parses the serial port message and reads the GPRMC message, obtains the UTC (universal time) time, i.e., the correction time, and modifies the computer time to achieve 1Hz time correction.

[0048] It should be noted that computer time consists of two parts, namely the absolute time provided by RTC CLOCK (real-time clock, which has an independent power supply) and the CPU time, i.e. the relative time provided by the CPU clock. RTC CLOCK is not very accurate, but has an independent power supply, and generally provides absolute time, which will not be affected by power outages; CPU time is more accurate, but has no independent power supply, and will stop timing when the computer is shut down, so it generally provides relative time. When the computing platform receives the PPS signal, the CPU time is reset, and the absolute time provided by RTCCLOCK is corrected to the UTC time in the GPRMC message, i.e. the correction time.

[0049] Specifically, after receiving the PPS signal, the computing platform will correct the absolute time provided by the RTC CLOCK to the UTC time. Since the normal PPS signal is 1Hz, the computing platform will correct the time every 1 second. Since the UTC time in the GPRMC message comes from the satellite positioning system, its time accuracy is higher. Therefore, the first level of this embodiment is to collect data from other sensors through the computing platform as a time reference. The second level is that the system time of the computing platform is corrected more accurately based on the UTC time in the GPRMC message. The third level is to achieve automatic correction of 1s interval time through the PPS signal, which can prevent the increase of time accumulation error caused by the influence of the computing platform's own system time process. Therefore, the joint effect of the three-layer technical effects can ensure the high accuracy of the unified time reference of each sensor.

[0050] Step S2: After time correction, the computing platform generates PWM signals of multiple frequencies to synchronously drive the lidar, inertial navigation and camera for data acquisition, so as to achieve time synchronization during multi-sensor fusion.

[0051] In one embodiment of the present invention, the computing platform includes a PWM (pulse width modulation) generator and a counter, and the computing platform generates PWM signals of multiple frequencies including: the computing platform generates a PWM signal of a preset frequency through the PWM generator; the computing platform counts the PWM signal of the preset frequency through the counter, and regenerates PWM signals of different frequencies based on the counting results.

[0052] Back to Figure 2 , the laser radar, inertial navigation and camera are all mounted on the computing platform, and receive the 1Hz PPS signal simulated by the PWM signal to trigger the acquisition. The specific implementation process is as follows Figure 5 As shown, the computing platform that has been corrected by the RTK module uses a high-frequency clock (CPU high-frequency part) to generate a PWM signal of a higher frequency, that is, a preset frequency (such as 1kHz), and 1ms corresponds to a PWM cycle; the counter counts the PWM signal to generate a PWM signal of a specific frequency. For example, to simulate a 1Hz PPS signal, a PPS pulse signal is generated when the counter reaches 1000, and to simulate a 10Hz PPS signal (a normal PPS signal is 1Hz, and the 10Hz signal is simulated by a PWM signal), a PPS signal is generated when the counter reaches 100, so 10 PPS signals are generated in 1s to complete 10Hz control.

[0053] Specifically, a PWM generator is used to generate a PWM signal with a preset frequency such as 1kHz, that is, 1ms is a PWM signal. If a 1Hz PPS signal is to be simulated, a counter is used to count. When the count reaches 1000 PWM signals, a PPS signal is sent once, that is, a PPS signal is sent once every 1s to trigger the laser radar to collect data. Therefore, the simulated PPS signal is 1Hz, so the 1Hz PWM signal sent to the laser radar is a simulated 1Hz PPS signal. When a 10Hz simulated PPS signal needs to be sent to the camera, a counter is used to count. When the count reaches 100 PWM signals, a PPS signal is sent once, that is, a PPS signal is sent once every 0.1s, so the simulated PPS signal is 10Hz, so the 10Hz PWM signal sent to the laser radar is a simulated 10Hz PPS signal, that is, the simulated 10Hz PPS signal is used to trigger the camera to collect data.

[0054] Figure 5 The signal simulation control at the hardware level shown ensures that different sensors can collect data at the same time. For example, if the camera and lidar are designed to collect data at the same frequency of 10Hz, and the collection is triggered simultaneously with the PWM signal, each frame of the radar point cloud has a completely corresponding image frame in the timestamp, which greatly improves the accuracy of subsequent point cloud processing.

[0055] In this embodiment, in order to achieve time synchronization during data fusion, the computing platform can simultaneously send PWM signals of different frequencies. The reason for the different frequencies is that in this embodiment, in order to ensure time synchronization, the built-in sampling frequency of the lidar and the PWM signal driving frequency of the camera need to be the same. For example, the camera and the lidar both collect data at the same frequency of 10 Hz, but the frequency of the lidar is the built-in sampling frequency of the radar, and the sampling frequency of the camera is the PWM signal frequency, so the above-mentioned PWM generator and counter need to work together to generate it.

[0056] As an example, Figure 6 As shown in the figure, the computing platform sends a 1Hz PWM signal to the laser radar to simulate a 1Hz PPS signal to trigger the laser radar to collect data. Therefore, the laser radar receives the signal every 1s and samples at its own sampling frequency, i.e., 10Hz. Therefore, data collection is performed at 0.1s, 0.2s, ..., and 0.1s, 0.2s, ..., 1.0s data frames are obtained. At the same time, the computing platform sends a 10Hz PWM signal to the camera to simulate a 10Hz PPS signal to trigger the camera to collect data. Since the camera shutter triggers the collection when the signal is given, the PWM signal frequency of the camera is 10Hz, that is, the computing platform sends a PPS signal to the camera every 0.1s. When the camera receives the signal, it will collect data. Therefore, the camera will also collect data at 0.1s, 0.2s, ..., and obtain 0.1s, 0.2s, ..., 1.0s data frames. This can achieve time synchronization of each data frame of the laser radar and the camera.

[0057] In one embodiment of the present invention, the method also includes: the computing platform also sends a timestamp to the lidar and the inertial navigation system, and uses the timestamp of the lidar as a reference to update the timestamp of the camera itself through the timestamp of the lidar, so as to achieve timestamp alignment between the lidar and the camera.

[0058] In this embodiment, the specific steps for realizing the timestamp alignment of the laser radar and the camera include: placing a preset number of data frame data collected by the laser radar into a preset shared memory in sequence according to the collection time; determining the timestamp of the data frame first collected by the camera, and searching the shared memory for a timestamp with the closest matching timestamp value to update its own timestamp, and removing the timestamp that first enters the shared memory, and at the same time placing the data frame data collected by the laser radar most recently into the shared memory in sequence according to the collection time, and repeating the timestamp alignment step to realize the timestamp alignment of the next data frame of the camera, until the timestamps of all data frames of the camera are aligned with the timestamps of the data frames collected by the laser radar.

[0059] The laser radar and camera are triggered by the simulated PPS signal to collect data at the same time. After the laser radar receives the simulated 1Hz PPS signal, the driving process automatically collects point cloud frames at 10Hz; a 10Hz PWM signal is sent at the same time as the simulated 1Hz PPS signal is sent. Each PWM signal triggers the camera to take a photo once, and finally the laser radar and camera complete the 10Hz synchronous trigger collection.

[0060] After each frame of point cloud and image data is synchronously collected, when the timestamp is marked by the driving process, the time accuracy will be affected by the CPU running speed, and different processes will have different random delays (but the order of magnitude is the same, generally around tens of microseconds when the CPU is running normally), which makes the timestamps of sensor data collected at the same time still slightly different. For example, when the lidar and camera transmit data to the computer, the CPU running speed will affect the final timestamps of the two sensors, and the camera itself will be timestamped, but the timestamps of the camera itself are not based on the computing platform, so the timestamps of the point cloud frame and the image frame need to be processed. Since the timestamps of the lidar and inertial navigation data are obtained based on the computing platform, the timestamps of the camera data can be updated based on the timestamps of the point cloud frame to achieve the alignment of the timestamps of the lidar and the camera. For example, 10 frames of point cloud data can be cached to form a queue, and each frame of image performs nearest neighbor matching on the timestamps in the point cloud frame queue to achieve complete synchronization of the timestamps of each frame of lidar and camera data.

[0061] Figure 7 This is a schematic diagram of timestamp alignment. Figure 7 In the above, lidar represents the laser radar, camera represents the camera, newlidarframe represents the new radar data frame, i.e., the point cloud frame, lidarframei represents the i-th point cloud frame, and cameraframe represents the image frame captured by the camera. Figure 7 As shown in the figure, the 10 point cloud frames collected by the lidar are first placed in the preset shared memory in the order of collection time. After waiting for 10 point cloud frames (such as data frames corresponding to timestamps of 0.1006, 0.2007, etc.) to be placed in, the data frame (image frame) first collected by the camera is taken. If its corresponding timestamp is 0.1007s, the timestamp with the closest timestamp value is found from the shared memory. If 0.1006s is found to be the closest, 0.1006s is used as the timestamp of the image frame to achieve timestamp nearest neighbor matching. Then lidar frame 1, that is, the point cloud frame corresponding to 0.1006s, is moved out of the shared memory, and lidar frame 11 is placed in the shared memory to form a 10-frame queue, and timestamp nearest neighbor matching is performed again until the timestamps of all camera data frames are aligned with the timestamps of the data frames collected by the lidar.

[0062] Therefore, this embodiment performs further fine timestamp alignment on the data at the software level, thereby eliminating the slight data timestamp inconsistency caused by the different times between program processes, and further making the data time error-free. In addition, the present invention does not require interpolation operations, which can reduce the amount of calculation and the use of computing resources, so that the time synchronization method can meet the needs of autonomous mobile tasks with high real-time requirements.

[0063] Furthermore, the present invention also provides a time synchronization system. Figure 2 As shown, the time synchronization system includes a computing platform and multiple sensors, the multiple sensors include positioning sensors, lidar, inertial navigation and cameras, and each sensor is connected to the computing platform.

[0064] Among them, the positioning sensor is used to send the correction time to the computing platform so that the computing platform can correct its own system time according to the correction time; the computing platform is used to generate PWM signals of various frequencies after time correction, and synchronously drive the lidar, inertial navigation and camera for data collection, so as to achieve time synchronization during multi-sensor fusion.

[0065] In one embodiment of the present invention, the positioning sensor is further used to send a PPS signal to the computing platform, so that the computing platform performs time correction according to the correction time at preset intervals based on the PPS signal.

[0066] In one embodiment of the present invention, the system time of the computing platform includes the absolute time provided by the independent real-time clock and the relative time provided by the CPU clock. When the computing platform corrects its own system time according to the correction time, it is specifically used to: after receiving the PPS signal, clear the relative time provided by the CPU clock, and correct the absolute time provided by the independent real-time clock to the correction time to achieve time correction.

[0067] In one embodiment of the present invention, the positioning sensor includes an RTK module, and the positioning sensor is further configured to send a GPRMC message to the computing platform through the RTK module, so as to implement the sending of the correction time.

[0068] In one embodiment of the present invention, the computing platform includes a PWM generator and a counter. When the computing platform generates PWM signals of multiple frequencies, it is specifically used for: the computing platform generates a PWM signal of a preset frequency through the PWM generator; the computing platform counts the PWM signal of the preset frequency through the counter, and regenerates PWM signals of different frequencies according to the counting results.

[0069] In one embodiment of the present invention, the built-in sampling frequency of the laser radar is the same as the PWM signal driving frequency of the camera.

[0070] In one embodiment of the present invention, the computing platform is also used to send timestamps to the laser radar and the inertial navigation.

[0071] In one embodiment of the present invention, the computing platform is also used to update the timestamp of the camera itself based on the timestamp of the lidar, so as to align the timestamps of the lidar and the camera.

[0072] In one embodiment of the present invention, when the computing platform realizes the timestamp alignment of the laser radar and the camera, it is specifically used to: put a preset number of data frame data collected by the laser radar into a preset shared memory in sequence according to the collection time; determine the timestamp of the data frame first collected by the camera, and search for a timestamp with the closest matching timestamp value from the shared memory to update its own timestamp, and remove the timestamp that first enters the shared memory, and at the same time put the data frame data collected by the laser radar most recently into the shared memory in sequence according to the collection time, and repeat the timestamp alignment steps to realize the timestamp alignment of the next data frame of the camera, until the timestamps of all data frames of the camera are aligned with the timestamps of the data frames collected by the laser radar.

[0073] It should be noted that the specific implementation of the time synchronization system in the embodiment of the present invention can refer to the specific implementation of the time synchronization method for multi-sensor fusion mentioned above, and will not be described again here to avoid redundancy.

[0074] In summary, the present invention performs time correction on the computing platform through the high-precision time source sent by the RTK module in the positioning sensor. When collecting data, each sensor based on this computing platform is under a unified time base, and the unified time base is a more accurate time base after correction, so it can be guaranteed that each sensor collects data under a more accurate unified time base; and the present invention also uses the PPS signal to enable the computing platform to update its own system time at preset intervals, thereby ensuring the accuracy of the unified time base; the present invention transmits PWM signals of multiple frequencies through hardware connection, so that each sensor triggers data collection at the same time, thereby The synchronization of data collection can be guaranteed, and the present invention also makes the built-in sampling frequency of the laser radar and the PWM signal driving frequency of the camera the same, and the PWM signal driving frequency is realized by the PWM generator and the counter during construction, which can ensure the accurate synchronization of the laser radar and camera data; the present invention further aligns the data with fine timestamps at the software level, thereby eliminating the slight data timestamp inconsistency caused by the different time between program processes, and thus making the data time zero error, and the present invention does not require interpolation operations, which can reduce the amount of calculation and the occupation of computing resources, so that the time synchronization method can meet the autonomous mobile tasks with high real-time performance.

[0075] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0076] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.

Claims

1. A time synchronization method for multi-sensor fusion, characterized in that: Applied to a time synchronization system, the time synchronization system includes a computing platform and multiple sensors, the multiple sensors include a positioning sensor, a laser radar, an inertial navigation system and a camera, each sensor is connected to the computing platform, and the method includes: The positioning sensor sends the correction time to the computing platform, so that the computing platform corrects its own system time according to the correction time; After time correction, the computing platform generates PWM signals of multiple frequencies to synchronously drive the laser radar, inertial navigation and camera to collect data, so as to achieve time synchronization during multi-sensor fusion.

2. The time synchronization method for multi-sensor fusion according to claim 1, characterized in that: The method further comprises: The positioning sensor sends a PPS signal to the computing platform, so that the computing platform performs time correction according to the correction time at every preset time based on the PPS signal.

3. The time synchronization method for multi-sensor fusion according to claim 2, characterized in that: The system time of the computing platform includes the absolute time provided by the independent real-time clock and the relative time provided by the CPU clock. The computing platform corrects its own system time according to the correction time, including: After receiving the PPS signal, the computing platform clears the relative time provided by the CPU clock and corrects the absolute time provided by the independent real-time clock to the correction time, so as to achieve time correction.

4. The time synchronization method for multi-sensor fusion according to claim 1, characterized in that: The positioning sensor includes an RTK module, and the positioning sensor sends a GPRMC message to the computing platform through the RTK module, so as to realize the sending of the correction time.

5. The time synchronization method for multi-sensor fusion according to claim 1, characterized in that: The computing platform includes a PWM generator and a counter, and the computing platform generates PWM signals of multiple frequencies, including: The computing platform generates a PWM signal of a preset frequency through the PWM generator; The computing platform counts the PWM signal of a preset frequency through the counter, and regenerates PWM signals of different frequencies according to the counting result.

6. The time synchronization method for multi-sensor fusion according to claim 1, characterized in that: The built-in sampling frequency of the laser radar is the same as the PWM signal driving frequency of the camera.

7. The time synchronization method for multi-sensor fusion according to claim 1, characterized in that: The method also includes: the computing platform also sends a timestamp to the laser radar and the inertial navigation.

8. The time synchronization method for multi-sensor fusion according to claim 7, characterized in that: The method further comprises: The timestamp of the laser radar is used as a reference, and the timestamp of the camera itself is updated by the timestamp of the laser radar, so as to align the timestamps of the laser radar and the camera.

9. The time synchronization method for multi-sensor fusion according to claim 8, characterized in that: The specific steps of realizing the timestamp alignment between the laser radar and the camera include: Put a preset number of data frames collected by the laser radar into the preset shared memory in sequence according to the collection time; Determine the timestamp of the data frame first collected by the camera, and search the shared memory for the timestamp with the closest matching timestamp value to update its own timestamp, and remove the timestamp that first enters the shared memory. At the same time, put the data frame data collected by the lidar most recently into the shared memory according to the collection time sequence, and repeat the timestamp alignment step to achieve the timestamp alignment of the next data frame of the camera, until the timestamps of all data frames of the camera are aligned with the timestamps of the data frames collected by the lidar.

10. A time synchronization system, characterized in that: It includes a computing platform and a plurality of sensors, wherein the plurality of sensors include a positioning sensor, a laser radar, an inertial navigation system and a camera, and each sensor is connected to the computing platform; The positioning sensor is used to send the correction time to the computing platform, so that the computing platform can correct its own system time according to the correction time; The computing platform is used to generate PWM signals of multiple frequencies after time correction, and synchronously drive the laser radar, inertial navigation and camera to collect data, so as to achieve time synchronization during multi-sensor fusion.

Images