Multi-sensor time synchronization method, device and system, medium and program product

Abstract

The embodiment of the invention relates to a multi-sensor time synchronization method, device and system, a medium and a program product. The method comprises the following steps: acquiring system time from a plurality of high-precision time sources according to priorities of the high-precision time sources; broadcasting the system time to a first sensor, so that the first sensor sets a timestamp for collected first sensing data according to the system time; and acquiring second sensing data from a second sensor, and setting a timestamp for the second sensing data according to the system time. By adopting the method, time synchronization of multiple sensors can be realized, and the accuracy of automatic driving and the safety of vehicles and personnel are improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of autonomous driving technology, and in particular, to a multi-sensor time synchronization method, apparatus, system, medium, and program product. Background Art

[0002] With the development of Internet and automotive technologies, more and more vehicles are equipped with autonomous driving functions. Generally, an autonomous driving vehicle collects surrounding environmental data through multiple sensors and performs autonomous driving based on the collected multiple environmental data.

[0003] Autonomous driving vehicles have extremely high requirements for the data time of sensors. If the sensors do not provide timestamps or the provided timestamps are inaccurate, it will affect the accuracy of autonomous driving, and further affect the safety of vehicles and personnel. Summary of the Invention

[0004] Embodiments of the present disclosure provide a multi-sensor time synchronization method, apparatus, system, medium, and program product, which can be used to provide a relatively accurate system time, so as to synchronize the times of multiple sensors, improve the accuracy of autonomous driving, and enhance the safety of vehicles and personnel.

[0005] In a first aspect, embodiments of the present disclosure provide a multi-sensor time synchronization method, the method comprising:

[0006] Obtaining system time from multiple high-precision time sources according to the priority of the high-precision time sources;

[0007] Broadcasting the system time to a first sensor for the first sensor to set a timestamp for the first sensed data collected according to the system time;

[0008] Obtaining second sensed data from a second sensor and setting a timestamp for the second sensed data according to the system time.

[0009] In a second aspect, embodiments of the present disclosure provide a multi-sensor time synchronization apparatus, the apparatus comprising:

[0010] A system time acquisition module, configured to obtain system time from multiple high-precision time sources according to the priority of the high-precision time sources;

[0011] A first time setting module, configured to broadcast the system time to a first sensor for the first sensor to set a timestamp for the first sensed data according to the system time;

[0012] A second time setting module, configured to obtain second sensed data from a second sensor and set a timestamp for the second sensed data according to the system time.

[0013] In a third aspect, embodiments of the present disclosure provide an autonomous driving system, which includes a data processor, multiple high-precision time sources, and multiple sensors; the data processor is communicatively connected to each high-precision time source and each sensor respectively;

[0014] The data processor is configured to execute the method as described in the first aspect;

[0015] The high-precision time source is configured to obtain time information;

[0016] The sensor is configured to collect data on the surrounding environment of the vehicle.

[0017] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0018] In a fifth aspect, embodiments of the present disclosure provide a computer program product, including a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0019] The multi-sensor time synchronization method, device, system, medium, and program product provided by the embodiments of the present disclosure obtain the system time from multiple high-precision time sources according to the priority of the high-precision time sources; broadcast the system time to the first sensor for the first sensor to set a timestamp for the first sensed data collected according to the system time; obtain the second sensed data from the second sensor and set a timestamp for the second sensed data according to the system time. The embodiments of the present disclosure can provide a relatively accurate system time through the high-precision time source, enable multi-sensor time synchronization, thereby improving the accuracy of autonomous driving and the safety of vehicles and personnel. Description of the Drawings

[0020] Figure 1 It is an application environment diagram of the multi-sensor time synchronization method in an embodiment;

[0021] Figure 2 It is a flowchart of the multi-sensor time synchronization method in an embodiment;

[0022] Figure 3 It is a flowchart of the step of obtaining the system time in an embodiment;

[0023] Figure 4a It is a schematic diagram of misaligned trigger moments in an embodiment;

[0024] Figure 4b It is a schematic diagram of the same acquisition perspective and acquisition scene in an embodiment;

[0025] Figure 5 It is a flowchart of the trigger alignment processing step in an embodiment;

[0026] Figure 6a One of the schematic diagrams of the trigger moment in an embodiment;

[0027] Figure 6b Another schematic diagram of the trigger moment in an embodiment;

[0028] Figure 6c Schematic diagram of the theoretical trigger period and the actual trigger period in an embodiment;

[0029] Figure 7 Flow schematic diagram of the trigger alignment processing step in an embodiment;

[0030] Figure 8 Schematic diagram of the trigger moment in the trigger correction processing in an embodiment;

[0031] Figure 9 Structural block diagram of a multi-sensor time synchronization device in an embodiment;

[0032] Figure 10 Structural diagram of an autonomous driving system in an embodiment. Detailed implementation manners

[0033] In order to make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present disclosure, and are not used to limit the embodiments of the present disclosure.

[0034] First, before specifically introducing the technical solutions of the embodiments of the present disclosure, the technical background or the technical evolution context on which the embodiments of the present disclosure are based will be introduced. With the development of Internet and automotive technologies, more and more vehicles with autonomous driving functions are available. Generally, autonomous driving vehicles collect surrounding environmental data through multiple sensors, such as cameras, radars, etc., and perform autonomous driving based on the multiple pieces of collected environmental data. After conducting experiments on autonomous driving and collecting data, it is found that autonomous driving vehicles have extremely high requirements for the time of sensor data. If the sensors do not provide timestamps or the provided timestamps are inaccurate, it will affect the accuracy of autonomous driving, and further affect the safety of vehicles and personnel. It should be noted that the applicant has made a large amount of creative labor both in terms of the extremely high requirements of autonomous driving vehicles for the time of sensor data and the technical solutions introduced in the following embodiments.

[0035] Next, the technical solutions involved in the embodiments of the present disclosure will be introduced in combination with the scenarios to which the embodiments of the present disclosure are applied.

[0036] The multi-sensor time synchronization method provided by the embodiments of the present disclosure can be applied to, for example Figure 1In the application environment shown. The application environment includes a vehicle with autonomous driving capabilities, in which an autonomous driving system 10 and a vehicle controller (Vehicle) 11 can be set. The autonomous driving system 10 plays a role in data collection, processing, and autonomous driving decision-making; the vehicle controller 11 plays an execution role. For example, the vehicle controller 11 can control components such as cylinders, brakes, and wheels to take corresponding actions according to the autonomous driving decision. Among them, the autonomous driving system 10 and the vehicle controller 11 communicate through a CAN (Controller Area Network) bus. The CAN bus is a serial communication protocol bus for real-time applications. It can use twisted pairs to transmit signals and is one of the most widely used field buses. The CAN protocol is used for communication between various different components in the vehicle. The characteristics of the CAN protocol include complete serial data communication, providing real-time support, a transmission rate of up to 1 Mb / s, 11-bit addressing, and error detection capabilities, etc.

[0037] In the autonomous driving system 10, a data processor 101, multiple high-precision time sources 102, and multiple sensors 103 are provided. Among them, the data processor 101 is a processor chip module that provides high computing power and high access capabilities. The high-precision time source 102 is used to obtain time information, and the accuracy of the time information is higher than a preset accuracy threshold. The sensors 103 include but are not limited to various cameras, millimeter-wave radars (Radar), and lidars (Lidar), etc.

[0038] In one embodiment, as Figure 2 shown, a multi-sensor time synchronization method is provided. Taking the data processor of the autonomous driving system in Figure 1 as an example, the method includes the following steps:

[0039] Step 201, obtain the system time from multiple high-precision time sources according to the priority of the high-precision time sources.

[0040] Multiple high-precision time sources are set in the autonomous driving system, and the multiple high-precision time sources correspond to different priorities. For example, the priorities corresponding to the first high-precision time source, the second high-precision time source, and the third high-precision time source are high, medium, and low, respectively.

[0041] In the automatic driving system, the data processor, in the order of decreasing priority, first obtains time information from the first high-precision time source. If the time information is obtained, the system time is determined according to the time information. If the time information is not obtained from the first high-precision time source, the time information is obtained from the second high-precision time source. Similarly, if the time information is obtained, the system time is determined according to the time information. If the time information is not obtained from the second high-precision time source, the time information is obtained from the third high-precision time source, and the system time is determined according to the time information.

[0042] In some embodiments, if the time information is not obtained from the first high-precision time source, it can be repeatedly obtained; in the case where the number of repeated acquisitions is greater than the first preset number, the time information is obtained from the second high-precision time source. Similarly, if the time information is not obtained from the second high-precision time source, it can also be repeatedly obtained, and in the case where the number of repeated acquisitions is greater than the second preset number, the time information is obtained from the third high-precision time source. It should be noted that the second preset number can be the same as the first preset number or different from the first preset number.

[0043] Step 202: Broadcast the system time to the first sensor for the first sensor to set a timestamp for the first sensed data collected according to the system time.

[0044] In the automatic driving system, the first sensor can be a sensor capable of providing a timestamp, such as a millimeter-wave radar, a lidar, etc. For such sensors, the data processor broadcasts the system time to the first sensor. The first sensor receives the system time broadcast by the data processor, and then sets a timestamp for the first sensed data collected according to the system time.

[0045] In some embodiments, the data processor broadcasts the system time to the first sensor through an eth (Ethernet) bus based on the gPTP (generalized Precision Time Protocol) method. The above gPTP has carried out a series of optimizations based on the PTP (IEEE 1588v2) protocol, forming a more targeted time synchronization mechanism, which can achieve a synchronization accuracy of the order of μs.

[0046] It can be understood that the system time obtained by the data processor from the high-precision time source is relatively accurate. Therefore, if the first sensor sets a timestamp according to the system time, the data can have a relatively accurate timestamp, thus meeting the requirements of the automatic driving for the data time.

[0047] Step 203: Obtain the second sensed data from the second sensor and set a timestamp for the second sensed data according to the system time.

[0048] In an autonomous driving system, the second sensor can be a sensor that cannot provide a timestamp, such as a camera. For such sensors, the data processor first obtains the second sensing data from the second sensor, and then sets a timestamp for the second sensing data according to the system time.

[0049] Understandably, the system time obtained by the data processor from the high-precision time source is relatively accurate. Therefore, when the data processor sets a timestamp for the second sensing data according to the system time, the data can have a relatively accurate timestamp, thus meeting the requirements of autonomous driving for data time.

[0050] In the above embodiment, according to the priority of the high-precision time source, the system time is obtained from multiple high-precision time sources; the system time is broadcast to the first sensor for the first sensor to set a timestamp for the collected first sensing data according to the system time; the second sensing data is obtained from the second sensor, and a timestamp is set for the second sensing data according to the system time. By providing a relatively accurate system time through the high-precision time source, the embodiments of the present disclosure can synchronize the times of multiple sensors, thereby improving the accuracy of autonomous driving and the safety of vehicles and personnel.

[0051] In one embodiment, as Figure 3 shown, the process of obtaining the system time from multiple high-precision time sources according to the priority of the high-precision time source may include the following steps:

[0052] Step 301, obtain multiple time information from multiple high-precision time sources.

[0053] In the scenario of the above embodiment, the data processor may sequentially obtain time information from multiple high-precision time sources in the order of decreasing priority. In another scenario, the data processor may simultaneously obtain multiple time information from multiple high-precision time sources. For example, the data processor simultaneously obtains the first time information, the second time information, and the third time information from the first high-precision time source, the second high-precision time source, and the third high-precision time source.

[0054] Step 302, sort the multiple time information according to the priority of the high-precision time source, and determine the system time according to the sorting result.

[0055] After obtaining multiple time information, the data processor sorts the multiple time information according to the priority of the high-precision time source. For example, the priorities corresponding to the first high-precision time source, the second high-precision time source, and the third high-precision time source are high, medium, and low respectively. According to the order of priority, the first time information is ranked first, the second time information is ranked second, and the third time information is ranked third. Then, the system time is determined according to the first time information.

[0056] In some embodiments, there is a situation where time information cannot be obtained from the first high-precision time source and / or the second high-precision time source. In this case, the system time is also determined according to the sorting result. For example, the first time information cannot be obtained from the first high-precision time source, while the second time information is obtained from the second high-precision time source, and the third time information is obtained from the third high-precision time source. Sorted in descending order of priority, if the second time information ranks first, then the system time is determined according to the second time information. Another example is that the second time information cannot be obtained from the second high-precision time source, the first time information is obtained from the first high-precision time source, and the third time information is obtained from the third high-precision time source. Sorted in descending order of priority, if the first time information ranks first, then the system time is determined according to the first time information.

[0057] In the above embodiments, multiple time information is obtained from multiple high-precision time sources; the multiple time information is sorted according to the priority of the high-precision time sources, and the system time is determined according to the sorting result. In the embodiments of the present disclosure, obtaining time information from multiple high-precision time sources can reduce the risk of data fusion failure due to the inability to obtain the system time, thereby improving the reliability of autonomous driving.

[0058] In one embodiment, the high-precision time source includes a global positioning module, a mobile communication module, and a clock chip. The step of obtaining multiple time information from multiple high-precision time sources may include: obtaining time information from the global positioning module through a first communication method; obtaining time information from the mobile communication module through a second communication method; obtaining time information from the clock chip (Real_Time Clock, RTC) through a third communication method.

[0059] The above global positioning module can obtain time information from the Global Positioning System (GPS) through an antenna, and then the data processor obtains the time information from the global positioning module through the first communication method.

[0060] The above global positioning system is a high-precision radio navigation positioning system based on artificial earth satellites, which can provide accurate geographical location, vehicle speed and precise time information anywhere in the world and in near-earth space. The above first communication method may include asynchronous communication and the way of general input / output superimposed Ethernet. Asynchronous communication uses a universal asynchronous receiver / transmitter Uart, which is a general-purpose serial data bus that can communicate bidirectionally and achieve full-duplex transmission and reception. General-purpose input / output Gpio (General-purpose input / output), whose function is similar to P0 - P3 of 8051, and its pins can be freely used by the user through programming. The PIN pins can be used as general input (GPI), general output (GPO) or general input and output (GPIO) according to actual considerations.

[0061] The above mobile communication module can obtain time information from the mobile communication network through an antenna, and then the data processor can obtain time information from the mobile communication module through the second communication method.

[0062] The above mobile communication network may include 4G and / or 5G mobile communication networks. Among them, 5G (5th Generation Mobile Communication Technology) is a new generation of broadband mobile communication technology with the characteristics of high speed, low latency and large connection. 5G communication facilities are the network infrastructure for realizing the interconnection of humans, machines and things. The above second communication method may include the way of USB superimposed Ethernet. USB (Universal Serial Bus) is a serial bus standard and also a technical specification for input / output interfaces. The latest generation is USB4, with a transmission speed of 40Gbit / s, three-stage voltage of 5V / 12V / 20V, a maximum power supply of 100W, and the new Type C interface allows blind insertion in both directions.

[0063] The above clock chip can be an integrated circuit that can provide accurate real-time time or time reference. Most clock chips use crystal oscillators with higher precision as the clock source. Some clock chips need to be powered by an external battery in order to still work when the main power supply is cut off. The above third communication method can adopt asynchronous communication, that is, use Uart for communication.

[0064] In the above embodiments, time information is obtained from the global positioning module through the first communication method; time information is obtained from the mobile communication module through the second communication method; time information is obtained from the clock chip through the third communication method. In the embodiments of the present disclosure, time information is obtained from different high-precision time sources through multiple communication methods, which can improve the acquisition efficiency and accuracy of time information and provide support for subsequent time synchronization of multiple sensors.

[0065] In practical applications, due to the different trigger cycles and trigger times of sensors, there will be a problem that the sensing data collected by multiple sensors is not aligned. For example Figure 4a As shown, it affects the efficiency and accuracy of the fusion of various sensing data. The common solution is that after multiple sensors collect data separately, first align the data frames of multiple sensing data according to the timestamps, and then perform data fusion processing. However, on the one hand, autonomous driving has high requirements for data time. Aligning data frames takes time and will affect the timeliness of autonomous driving decisions. On the other hand, due to the different trigger times of sensors, the collected sensing data not only has the problem of time misalignment, but also due to the movement of the vehicle, there will be problems of inconsistent acquisition perspectives and acquisition scenarios, resulting in data that cannot be fused and affecting the decision-making accuracy of autonomous driving.

[0066] In view of the above problems, the embodiments of the present disclosure provide a solution: perform trigger alignment processing on the first sensor and the second sensor to dynamically keep the first sensor and the second sensor triggered at the same time.

[0067] During the entire autonomous driving process, the data processor performs trigger alignment processing on the first sensor and the second sensor, controls the first sensor and / or the second sensor, so that the first sensor and the second sensor are triggered at the same time within each trigger period.

[0068] It can be understood that when the first sensor and the second sensor are triggered at the same time and exposed at the same time, the collected sensing data can naturally have the characteristic of alignment; in addition, as Figure 4b As shown, Actor is the acquisition object and lidar is the radar, which can make the acquisition perspectives and acquisition scenarios the same, thereby reducing the risk of data that cannot be fused and improving the decision-making speed and decision-making accuracy of autonomous driving.

[0069] In one embodiment, as Figure 5 As shown, the process of performing trigger alignment processing on the first sensor and the second sensor may include the following steps:

[0070] Step 401, determine the trigger cycle of the second sensor according to the trigger cycle of the first sensor.

[0071] Among them, the trigger period is the time interval between two trigger moments. The first sensor can be an actively triggered sensor, and the second sensor can be a passively triggered sensor. For example, the first sensor is a millimeter-wave radar, lidar, etc. After these sensors are started, they have a relatively fixed trigger period, that is, they will be actively triggered once every preset time interval. The second sensor is a camera, etc. After this type of sensor is started, it will be triggered only when it receives a trigger signal. Based on this situation, the trigger period of the second sensor can be determined according to the trigger period of the first sensor, so that within the same trigger period, the first sensor and the second sensor are triggered at the same moment.

[0072] In practical applications, the trigger period of the first sensor is the same as that of the second sensor, or the trigger period of the first sensor is an integer multiple of the trigger period of the second sensor. For example, if the trigger period of the first sensor is T1 and the trigger period of the second sensor is T2, then T1 = N * T2, where N is a positive integer.

[0073] Based on the above relationship, after determining the trigger period of the first sensor, the data processor can determine the trigger period of the second sensor according to the trigger period of the first sensor.

[0074] Step 402, after starting the first sensor, send a control instruction to the second sensor according to the trigger period of the second sensor. The control instruction is used to trigger the second sensor, and the trigger moment of the second sensor matches the trigger moment of the first sensor.

[0075] After starting the first sensor, the data processor can obtain the timestamp of the first frame of sensing data collected by the first sensor. According to the timestamp of the first frame of sensing data and the trigger period of the first sensor, the trigger moment when the second sensor is first triggered can be determined. Then, according to the trigger moment when the second sensor is first triggered and the trigger period of the second sensor, the subsequent trigger moments of the second sensor can be determined. Next, control instructions are generated according to the trigger moment of the second sensor each time, and the control instructions are sent to the second sensor. In this way, after receiving the control instruction, the second sensor is triggered under the control of the control instruction, and the trigger moment of the second sensor can be made to match the trigger moment of the first sensor.

[0076] For example, the trigger period T1 of the first sensor is the same as the trigger period T2 of the second sensor. After the first sensor is started, the timestamp of the first frame of sensing data collected by the first sensor is t1. According to the timestamp t1 and the trigger period T1 of the first sensor, the data processor determines that the first trigger time of the second sensor is t1' = t1 + T2. The second trigger time t2 of the first sensor is t1 + T1, and the second trigger time of the second sensor is t2' = t1' + T2. By analogy, it can be determined that in subsequent trigger periods, the first sensor and the second sensor are triggered at the same time each time, as Figure 6a shown.

[0077] For another example, the trigger period T1 of the first sensor is twice the trigger period T2 of the second sensor. After the first sensor is started, the timestamp of the first frame of sensing data collected by the first sensor is t1. According to the timestamp t1 and the trigger period T1 of the first sensor, the data processor determines that the first trigger time of the second sensor is t1' = t1 + T2 = t1 + T1 / 2. The second trigger time t2 of the first sensor is t1 + T1, and the second trigger time of the second sensor is t2' = t1' + T2 = t1 + T1. By analogy, it can be determined that in subsequent trigger periods, each time the first sensor is triggered, it is at the same time as the second sensor's even-numbered triggers, as Figure 6b shown.

[0078] The above-mentioned sending of control instructions to the second sensor can be implemented by means of IO (Input / Output) control. For example, a square wave signal is sent to the second sensor to trigger the second sensor periodically with a high level or a low level. It can also be implemented by adding an embedded line, that is, each time the second sensor is triggered, a delayed exposure instruction or an early exposure instruction is sent to the second sensor.

[0079] In the above embodiments, the trigger period of the second sensor is determined according to the trigger period of the first sensor; after the first sensor is started, a control instruction is sent to the second sensor according to the trigger period of the second sensor. The control instruction is used to trigger the second sensor, and the trigger time of the second sensor matches the trigger time of the first sensor. By controlling the second sensor in the embodiments of the present disclosure, the second sensor and the first sensor can be triggered at the same time, expose and collect data at the same time, so as to achieve data alignment, consistent acquisition perspectives and acquisition scenarios, facilitate subsequent data fusion processing, and thus improve the decision-making efficiency and decision-making accuracy of the autonomous driving system.

[0080] In one embodiment, both the first sensor and the second sensor are unstable. For example, there are slight differences between the theoretical trigger period and the actual trigger period of the sensor, such as Figure 6cAs shown. Therefore, after running for a period of time, there may be a large deviation in the trigger moments between the first sensor and the second sensor, affecting data fusion. For example, assuming that the average jitter duration of the second sensor is Δt, after n rounds of cycling, there will be a deviation of n*Δt between the trigger moment of the first sensor and the trigger moment of the second sensor.

[0081] Considering the problems caused by the above instability, the step of determining the trigger period of the second sensor according to the trigger period of the first sensor may include: determining the trigger period of the second sensor according to the trigger period of the first sensor and the average jitter duration of the second sensor.

[0082] For example, if the trigger period of the first sensor is T1, the trigger period of the second sensor is T2, and the average jitter duration is Δt, and it is determined that T1 = T2 + Δt, then the trigger period T2 of the second sensor is determined to be T2 = T1 - Δt.

[0083] Based on the above embodiments, the following method can be used to solve the jitter problem, as Figure 7 shown, the embodiments of the present disclosure may further include the following steps:

[0084] Step 501, respectively obtain the trigger moments of the first sensor and the second sensor within the first trigger period, and determine the interval duration between the two trigger moments.

[0085] Among them, the first trigger period may be each trigger period or a preset trigger period. The trigger period is the period when the first sensor and the second sensor need to be triggered at the same moment.

[0086] In practical applications, the data processor may determine the trigger moment of the first sensor and the trigger moment of the second sensor within each trigger period, and determine the interval duration between the two trigger moments. The data processor may also determine the trigger moment of the first sensor and the trigger moment of the second sensor within a preset trigger period, and determine the interval duration between the two trigger moments.

[0087] Step 502, if the interval duration is greater than or equal to the preset duration, perform trigger correction processing on the second sensor so that the second trigger is triggered at the same moment as the first sensor within the second trigger period.

[0088] Among them, the second trigger period is the next trigger period adjacent to the first trigger period. The preset duration is determined according to the maximum deviation that the system can tolerate between the trigger moment of the first sensor and the trigger moment of the second sensor. For example, if the maximum deviation tolerated by the system is ΔT, the preset duration can be determined to be ΔT.

[0089] Compare the interval duration with the preset duration. If the interval duration is less than the preset duration, it indicates that the deviation between the trigger time of the first sensor and the trigger time of the second sensor has not reached the maximum deviation tolerable by the system. In this case, the first sensor and the second sensor can continue with subsequent triggering and data acquisition in their current states, and the sensed data collected has a certain degree of alignment, which will not affect data fusion and the decision-making of autonomous driving.

[0090] If the interval duration is greater than or equal to the preset duration, it indicates that the deviation between the trigger time of the first sensor and the trigger time of the second sensor has reached the maximum deviation tolerable by the system. If no measures are taken, the deviation between the trigger times of the two types of sensors will continue to increase, which will affect data alignment and data fusion.

[0091] Therefore, when the interval duration is greater than or equal to the preset duration, trigger correction processing is performed on the second sensor. The process of trigger correction processing may include: determining the trigger time of the first sensor in the second trigger period according to the trigger period of the first sensor and the trigger time in the first trigger period; then determining the trigger time of the second sensor in the second trigger period according to the trigger time of the first sensor in the second trigger period and the trigger period of the second sensor, and controlling the second sensor to be triggered at the corresponding trigger time. In this way, the second trigger will be triggered at the same time as the first sensor in the second trigger period, as Figure 8 shown.

[0092] In the above embodiments, the trigger times of the first sensor and the second sensor in the first trigger period are respectively obtained, and the interval duration between the two trigger times is determined; if the interval duration is greater than or equal to the preset duration, trigger correction processing is performed on the second sensor so that the second trigger is triggered at the same time as the first sensor in the second trigger period. By performing trigger correction processing on the second sensor in the embodiments of the present disclosure, the second sensor can be triggered simultaneously with the first sensor again, reducing the deviation caused by jitter, making the data alignment, the acquisition perspectives, and the acquisition scenarios consistent, thereby improving the decision-making efficiency and decision-making accuracy of the autonomous driving system.

[0093] Based on the above embodiments, the embodiments of the present disclosure may further include: broadcasting the system time to the vehicle controller via the CAN bus for the vehicle controller to perform vehicle control according to the system time.

[0094] After determining the system time, the data processor can broadcast the system time to the vehicle controller via the CAN bus. The vehicle controller receives the system time and performs vehicle control according to the system time.

[0095] Understandably, the vehicle controller and the sensors adopt the same system time, and the vehicle control and sensing data are relatively matched in time. Therefore, the safety and reliability of autonomous driving can be improved.

[0096] In some embodiments, the above-mentioned broadcasting of the system time to the vehicle controller via the CAN bus may include: broadcasting the system time to the vehicle controller via the CAN bus according to a preset time format and a preset broadcasting speed.

[0097] In practical applications, the system time is encapsulated into a time message according to a preset time format and the time message is broadcast to the vehicle controller. The preset time format may be as follows:

[0098] Message identifier Year Month Day Hour Minute Second Millisecond

[0099] In practical applications, the time message can also be broadcast at a preset broadcasting speed. For example, the time message is broadcast every 10 ms.

[0100] The vehicle controller can detect whether the broadcasting speed of the time message is stable according to its own timer. If the broadcasting speed is unstable, the vehicle controller uses its own timer for timing and does not adopt the system time in the time message. If the broadcasting speed is stable, the vehicle controller adopts the system time in the time message.

[0101] The process of detecting whether the broadcasting speed of the time message is stable may include: receiving at least two time messages and determining the receiving time interval between every two adjacent time messages. If the receiving time interval is consistent with the preset broadcasting speed, it is determined that the broadcasting speed of the time message is stable; if the receiving time interval is inconsistent with the preset broadcasting speed, it is determined that the broadcasting speed of the time message is unstable.

[0102] It should be noted that the preset time format and the preset broadcasting speed are not limited to the above description and other formats can also be adopted.

[0103] In the above embodiments, the system time is broadcast to the vehicle controller via the CAN bus according to a preset time format and a preset broadcasting speed. The embodiments of the present disclosure can enable the vehicle controller and the sensors to adopt the same system time, so that the vehicle control and sensing data are relatively matched in time, thereby improving the safety and reliability of autonomous driving.

[0104] In one embodiment, a multi-sensor time synchronization method is provided. Taking the example that the method is applied to the data processor of the Figure 1 autonomous driving system as an example for description, the embodiments of the present disclosure may include the following steps:

[0105] Step 1, obtain time information from the global positioning module through the first communication method; obtain time information from the mobile communication module through the second communication method; obtain time information from the clock chip through the third communication method.

[0106] Step 2, sort multiple time information according to the priorities of the global positioning module, the mobile communication module, and the clock chip, and determine the system time according to the sorting result.

[0107] Step 3, determine the trigger period of the second sensor according to the trigger period of the first sensor.

[0108] Among them, the first sensor may include a lidar, a millimeter-wave radar, etc.; the second sensor may include a camera, etc.

[0109] Step 4, determine the trigger period of the second sensor according to the trigger period of the first sensor and the average jitter duration of the second sensor.

[0110] Step 5, after starting the first sensor, send a control instruction to the second sensor according to the trigger period of the second sensor.

[0111] Among them, the control instruction is used to trigger the second sensor, and the trigger moment of the second sensor matches the trigger moment of the first sensor.

[0112] Step 6, broadcast the system time to the first sensor for the first sensor to set a timestamp for the collected first sensing data according to the system time.

[0113] Step 7, obtain the second sensing data from the second sensor and set a timestamp for the second sensing data according to the system time.

[0114] Step 8, broadcast the system time to the vehicle controller through the CAN bus according to the preset time format and the preset broadcast speed.

[0115] Step 9, respectively obtain the trigger moments of the first sensor and the second sensor within the first trigger period, and determine the interval duration between the two trigger moments.

[0116] Step 10, if the interval duration is greater than or equal to the preset duration, perform trigger correction processing on the second sensor so that the second trigger is triggered at the same moment as the first sensor within the second trigger period.

[0117] Among them, the second trigger period is the next trigger period adjacent to the first trigger period.

[0118] It should be understood that although the steps in the above flowcharts are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turns with at least a part of other steps or steps or stages in other steps.

[0119] In one embodiment, as Figure 9 shown, a multi-sensor time synchronization device is provided, including:

[0120] A system time acquisition module 601, configured to acquire system time from multiple high-precision time sources according to the priority of the high-precision time sources;

[0121] A first time setting module 602, configured to broadcast the system time to a first sensor for the first sensor to set a timestamp of first sensing data according to the system time;

[0122] A second time setting module 603, configured to acquire second sensing data from a second sensor and set a timestamp of the second sensing data according to the system time.

[0123] In one embodiment, the system time acquisition module 601 is specifically configured to acquire multiple time information from multiple high-precision time sources; sort the multiple time information according to the priority of the high-precision time sources, and determine the system time according to the sorting result.

[0124] In one embodiment, the high-precision time sources include a global positioning module, a mobile communication module, and a clock chip. The system time acquisition module 601 is specifically configured to acquire time information from the global positioning module through a first communication method; acquire time information from the mobile communication module through a second communication method; acquire time information from the clock chip through a third communication method.

[0125] In one embodiment, the device further further includes:

[0126] An alignment processing module, configured to perform trigger alignment processing on the first sensor and the second sensor to dynamically keep the first sensor and the second sensor triggered at the same moment.

[0127] In one embodiment, the alignment processing module is specifically configured to determine the trigger period of the second sensor according to the trigger period of the first sensor; after starting the first sensor, send a control instruction to the second sensor according to the trigger period of the second sensor, where the control instruction is used to trigger the second sensor, and the trigger moment of the second sensor matches the trigger moment of the first sensor.

[0128] In one embodiment, the alignment processing module is specifically configured to determine the trigger period of the second sensor according to the trigger period of the first sensor and the average jitter duration of the second sensor.

[0129] In one embodiment, the device further includes:

[0130] The interval acquisition module is configured to respectively acquire the trigger moments of the first sensor and the second sensor within the first trigger period, and determine the interval duration between the two trigger moments;

[0131] The correction processing module is configured to, if the interval duration is greater than or equal to a preset duration, perform trigger correction processing on the second sensor, so that the second trigger is triggered at the same moment as the first sensor within the second trigger period, where the second trigger period is the next trigger period adjacent to the first trigger period.

[0132] In one embodiment, the device further includes:

[0133] The broadcast module is configured to broadcast the system time to the vehicle controller through the CAN bus for the vehicle controller to perform vehicle control according to the system time.

[0134] In one embodiment, the broadcast module is specifically configured to broadcast the system time to the vehicle controller through the CAN bus according to a preset time format and a preset broadcast speed.

[0135] For the specific limitations of the multi-sensor time synchronization device, reference can be made to the limitations of the multi-sensor time synchronization method in the above text, which will not be elaborated here. Each module in the above multi-sensor time synchronization device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0136] In one embodiment, as Figure 1As shown, an autonomous driving system is provided. The autonomous driving system 10 includes a data processor 101, multiple high-precision time sources 102, and multiple sensors 103. The data processor 101 is communicatively connected to each high-precision time source 102 and each sensor 103 respectively. The data processor 101 is configured to execute the method in the above-mentioned embodiments. The high-precision time source 102 is configured to obtain time information. The sensor 103 is configured to collect data on the vehicle's surrounding environment.

[0137] In the embodiments of the present disclosure, the autonomous driving system 10 includes a data processor 101, multiple high-precision time sources 102, and multiple sensors 103. The data processor 101 is communicatively connected to each high-precision time source 102 and each sensor 103 respectively.

[0138] Each high-precision time source 102 obtains time information and transmits the time information to the data processor 101. The data processor 101 determines the system time based on the priority of the high-precision time source and the time information. Each sensor 103 collects data on the vehicle's surrounding environment. For the first sensor that can set a timestamp, the data processor 101 broadcasts the system time to the first sensor. After receiving the system time, the first sensor sets a timestamp for the first sensing data collected according to the system time. For the second sensor that cannot set a timestamp, the data processor 101 obtains the second sensing data from the second sensor and sets a timestamp for the second sensing data according to the system time.

[0139] In the above-mentioned embodiments, the autonomous driving system can obtain a relatively accurate system time from the high-precision time source, thereby synchronizing the time of multiple sensors, improving the accuracy of autonomous driving, and enhancing the safety of the vehicle and personnel.

[0140] In one embodiment, as Figure 10 shown, the multiple high-precision time sources 102 include a global positioning module 1021, a mobile communication module 1022, and a clock chip 1023. The autonomous driving system 10 further includes a network processor 104. The global positioning module 1021 is communicatively connected to the data processor 104 through the network processor 104. The mobile communication module 1022 is communicatively connected to the data processor 101 through the network processor 104. The clock chip 1023 is communicatively connected to the data processor 104.

[0141] In the embodiments of the present disclosure, the multiple high-precision time sources 102 include a global positioning module 1021, a mobile communication module 1022, and a clock chip 1023. The autonomous driving system 10 further includes a network processor 104. The global positioning module 1021 is communicatively connected to the network processor 104 through Uart and Gpio. The mobile communication module 1022 is communicatively connected to the network processor 104 through USB. The network processor 104 is communicatively connected to the data processor 101 through eth. The clock chip 1023 is communicatively connected to the data processor through Uart.

[0142] In some embodiments, the network processor 104 is communicatively connected to the vehicle controller 11 through a CAN bus.

[0143] The above-mentioned network processor 104 combines ASIL D-level security, high-performance real-time and application processing, and network acceleration functions, and supports the requirements of new automotive architectures: service gateways, domain controllers, regional processors, security processors, etc.

[0144] In the above embodiments, the multiple high-precision time sources include a global positioning module, a mobile communication module, and a clock chip; the autonomous driving system further includes a network processor. The multiple high-precision time sources provided in the embodiments of the present disclosure can improve the accuracy of the system time. Moreover, the provided network processor can process the data transmission between the multiple high-precision time sources and the data processor, making the data more suitable for processing by the data processor.

[0145] In one embodiment, as Figure 10 shown, the multiple sensors 103 include a camera 1031 and a radar 1032. The autonomous driving system 10 further includes a signal mixer 105 and an Ethernet switch 106; the camera 1031 is communicatively connected to the data processor 101 through the signal mixer 105; the radar 1032 is communicatively connected to the data processor 101 through the Ethernet switch 106.

[0146] In the embodiments of the present disclosure, the multiple sensors 103 include a camera 1031 and a radar 1032. The autonomous driving system includes a signal mixer 105 and an Ethernet switch 106. The camera 1031 is communicatively connected to the signal mixer 105 through GMSL (Gigabit Multimedia Serial Links); the signal mixer 105 is communicatively connected to the data processor 101 through Mipi (Mobile Industry Processor Interface). The radar 1032 is communicatively connected to the Ethernet switch 106 through eth, and the Ethernet switch 106 is communicatively connected to the data processor 101 through eth.

[0147] The above signal mixer 105 can support up to 4 unidirectional channels, and the maximum data rate of each channel can reach 6.144 Gbps. Its main purpose is to convert Mipi into a GMSL signal that can be transmitted over long distances. The above GMSL is a high-speed serial interface launched by Maxim Corporation and is suitable for the transmission of audio, video, and control signals. The above Mipi is an open standard and a specification developed by the MIPI Alliance for mobile application processors.

[0148] The above Ethernet switch 106 has an 11-port Ethernet gigabit capacity, complies with the IEEE 802.3 automotive standard, and provides high performance and low power consumption.

[0149] In the above embodiments, the multiple sensors include cameras and radars, and the autonomous driving system further includes a signal mixer and an Ethernet switch. The signal mixer and Ethernet switch provided by the embodiments of the present disclosure can handle data transmission between multiple sensors and a data processor, enabling the data processor to better perform time synchronization for multiple sensors.

[0150] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions. The above instructions can be executed by the data processor 101 of the autonomous driving system 10 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0151] In an exemplary embodiment, a computer program product is also provided. When the computer program is executed by a processor, the above method can be implemented. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, part or all of the above method can be implemented in accordance with the process or function described in the embodiments of the present disclosure.

[0152] It should be noted that for the solutions described in this specification and embodiments, if they involve personal information processing, they will be processed on the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for performing a contract, etc.), and will only be processed within the specified or agreed scope. When a user refuses to process personal information other than the necessary information required for basic functions, it will not affect the user's use of basic functions.

[0153] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0154] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0155] The above-described embodiments merely represent several implementation manners of the embodiments of the present disclosure. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present disclosure, several modifications and improvements can be made, and these all belong to the protection scope of the embodiments of the present disclosure. Therefore, the protection scope of the patent of the embodiments of the present disclosure should be subject to the appended claims.

Claims

1. A multi-sensor time synchronization method, characterized in that, the method includes: Obtaining the system time from multiple high-precision time sources according to the priority of the high-precision time sources; Broadcasting the system time to the first sensor for the first sensor to set a timestamp for the first sensed data collected according to the system time; Obtaining second sensed data from the second sensor and setting a timestamp for the second sensed data according to the system time.

2. The method according to claim 1, characterized in that, the obtaining the system time from multiple high-precision time sources according to the priority of the high-precision time sources includes: Obtaining multiple time information from multiple high-precision time sources; Sorting the multiple time information according to the priority of the high-precision time sources and determining the system time according to the sorting result.

3. The method according to claim 2, characterized in that, the high-precision time sources include a global positioning module, a mobile communication module and a clock chip, and the obtaining multiple time information from multiple high-precision time sources includes: Obtaining the time information from the global positioning module through a first communication method; Obtaining the time information from the mobile communication module through a second communication method; Obtaining the time information from the clock chip through a third communication method.

4. The method according to any one of claims 1-3, characterized in that, the method further includes: Performing trigger alignment processing on the first sensor and the second sensor to dynamically keep the first sensor and the second sensor triggered at the same moment.

5. The method according to claim 4, characterized in that, the performing trigger alignment processing on the first sensor and the second sensor includes: Determining the trigger period of the second sensor according to the trigger period of the first sensor; After starting the first sensor, sending a control instruction to the second sensor according to the trigger period of the second sensor, the control instruction is used to trigger the second sensor, and the trigger moment of the second sensor matches the trigger moment of the first sensor.

6. The method according to claim 5, characterized in that, the determining the trigger period of the second sensor according to the trigger period of the first sensor includes: Determining the trigger period of the second sensor according to the trigger period of the first sensor and the average jitter duration of the second sensor.

7. The method according to claim 5, characterized in that, the method further includes: Respectively obtaining the trigger moments of the first sensor and the second sensor within a first trigger period and determining the interval duration between the two trigger moments; If the interval duration is greater than or equal to a preset duration, performing trigger correction processing on the second sensor so that the second trigger is triggered at the same moment as the first sensor within a second trigger period, where the second trigger period is the next trigger period adjacent to the first trigger period.

8. The method according to claim 1, characterized in that, the method further includes: Broadcast the system time to the vehicle controller via the CAN bus for the vehicle controller to perform vehicle control based on the system time.

9. The method according to claim 8, wherein, the broadcasting the system time to the vehicle controller via the CAN bus includes: broadcasting the system time to the vehicle controller via the CAN bus according to a preset time format and a preset broadcast speed.

10. A multi-sensor time synchronization device, wherein, the device includes: a system time acquisition module, configured to acquire the system time from multiple high-precision time sources according to the priority of the high-precision time sources; a first time setting module, configured to broadcast the system time to a first sensor for the first sensor to set the timestamp of the first sensing data according to the system time; a second time setting module, configured to acquire second sensing data from a second sensor and set the timestamp of the second sensing data according to the system time.

11. An autonomous driving system, wherein, the autonomous driving system includes a data processor, multiple high-precision time sources, and multiple sensors; the data processor is communicatively connected to each of the high-precision time sources and each of the sensors; the data processor is configured to execute the method according to any one of claims 1-9; the high-precision time sources are configured to acquire time information; the sensors are configured to collect data on the vehicle's surrounding environment.

12. The autonomous driving system according to claim 11, wherein, the multiple high-precision time sources include a global positioning module, a mobile communication module, and a clock chip; the autonomous driving system further includes a network processor; the global positioning module is communicatively connected to the data processor through the network processor; the mobile communication module is communicatively connected to the data processor through the network processor; the clock chip is communicatively connected to the data processor.

13. The autonomous driving system according to claim 11, wherein, the multiple sensors include a camera and a radar, and the autonomous driving system includes a signal mixer and an Ethernet switch; the camera is communicatively connected to the data processor through the signal mixer; the radar is communicatively connected to the data processor through the Ethernet switch.

14. A storage medium, on which a computer program is stored, wherein, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

15. A computer program product, including a computer program, wherein, when the computer program is executed by a processor, the steps of the method according to any one of claims 1-9 are implemented.

Images