Time synchronization method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide a time synchronization method and device, electronic equipment and storage medium. The method comprises: determining a target time source corresponding to a calculation unit module in a controller based on a first time source corresponding to a processor module in the controller and target point cloud data sent by a laser radar device to the calculation unit module in the controller; the first time source is determined based on a standard clock signal for time service of the processor module by a logic module in the controller; the first time source is used for time synchronization of the calculation unit module; and the target point cloud data is used for time calibration of the calculation unit module after time synchronization; and time service is performed on a camera device based on the target time source, a first network delay value corresponding to the calculation unit module and the camera device, and a first clock bias value, so as to realize time synchronization of modules in the controller and time synchronization among the controller, the laser radar device and the camera device.

Description

Technical Field

[0001] This invention belongs to the field of vehicle technology, and in particular relates to a time synchronization method, device, electronic device, and storage medium. Background Technology

[0002] Autonomous driving is a technology that uses various types of sensors to acquire environmental information (obstacles, location, road information, traffic information), performs information calculations and processing based on a high-performance computing platform, and then sends the calculation results to the vehicle chassis to enable the vehicle to drive automatically.

[0003] The stable operation of autonomous driving systems relies on the time synchronization of multiple computing unit modules within the autonomous driving computing platform, as well as the time synchronization between the computing platform and sensors. Related technologies often depend on the integrated navigation receiver acquiring the PPS+GPRM signal source and then distributing it to different devices. However, since PPS is a low-power pulse signal, it cannot simultaneously provide PPS signals to multiple devices. This causes time asynchrony among the computing unit modules, controllers, and sensors, resulting in a deviation of the entire system's time from the standard time, which affects the safety and reliability of autonomous driving. Summary of the Invention

[0004] To overcome the problems existing in related technologies, the present invention provides a time synchronization method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, the present invention provides a time synchronization method applied to a vehicle, the vehicle including a controller, a lidar device, and a camera device; the method includes:

[0006] Based on the first time source corresponding to the processor module in the controller and the target point cloud data sent by the lidar device to the computing unit module in the controller, the target time source corresponding to the computing unit module is determined.

[0007] The first time source is determined based on a standard clock signal used by the logic module in the controller to synchronize the processor module. The first time source is used to synchronize the time of the computing unit module, and the target point cloud data is used to calibrate the time of the synchronized computing unit module.

[0008] Based on the target time source and the first network delay value and first clock deviation value corresponding to the computing unit module and the camera device, the camera device is timed to achieve time synchronization of each module inside the controller and time synchronization between the controller, the lidar device and the camera device.

[0009] Optionally, the method further includes:

[0010] When the processor module receives a standard clock signal sent by the logic module, the first time corresponding to the standard clock signal is set as the first time source of the processor module;

[0011] The second time corresponding to the first clock signal is set as the second time source of the processor module; the second time is the network time obtained by the processor module.

[0012] When the time mode of the processor module is the second time source, the second time source is calibrated based on the predetermined target time difference between the first time and the second time.

[0013] Optionally, the logic module is used to distribute the received standard clock signal to the processor module and the lidar device to provide time synchronization for the processor module and the lidar device.

[0014] Optionally, determining the target time source corresponding to the computing unit module based on the first time source corresponding to the processor module in the controller and the target point cloud data sent from the lidar device to the computing unit module in the controller includes:

[0015] Based on the first time source corresponding to the processor module, the computing unit module is timed to obtain the first time to be calibrated.

[0016] When the computing unit module receives target point cloud data sent by the lidar device to the computing unit module, it obtains the timestamp information carried by the target point cloud data and determines the first time information of the computing unit module receiving the target point cloud data.

[0017] When the computing unit module receives the first computing message sent by the lidar device, it obtains the second time information carried in the first computing message and determines the third time information of the first feedback message sent by the computing unit module.

[0018] Wherein, the first calculation message is sent by the lidar device upon receiving the first feedback message, and the second time information is the reception time information of the lidar device upon receiving the first feedback message;

[0019] Based on the timestamp information, the first time information, the second time information, and the third time information, the first time to be calibrated is calibrated, and the target time source is determined.

[0020] Optionally, calibrating the first time to be calibrated based on the timestamp information, the first time information, the second time information, and the third time information, and determining the target time source, includes:

[0021] Based on the timestamp information, the first difference between the first time information, and the second difference between the second time information and the third time information, a second network delay value and a second clock offset value are determined.

[0022] Based on the second network delay value and the second clock deviation value, the first time to be calibrated is calibrated to obtain the system time corresponding to the computing unit module;

[0023] The system time corresponding to the computing unit module is determined as the target time source.

[0024] Optionally, before timing the camera device based on the target time source and the first network delay value and first clock offset value corresponding to the computing unit module and the camera device, the method further includes:

[0025] Obtain the first transmission time and the first reception time of the first synchronization message; the first synchronization message is sent by the computing unit module to the camera device;

[0026] Obtain the second sending time and the second receiving time of the second synchronization message; the second synchronization message is sent by the camera device to the computing unit module;

[0027] Based on the first sending time and the first receiving time, a first correspondence is determined, and based on the second sending time and the second receiving time, a second correspondence is determined;

[0028] Based on the first correspondence and the second correspondence, the first network delay value and the first clock offset value are determined.

[0029] Optionally, the step of providing time synchronization to the camera device based on the target time source, the first network delay value corresponding to the computing unit module and the camera device, and the first clock offset value includes:

[0030] Based on the target time source, the calculation unit module provides time synchronization to the camera device to obtain the second time to be calibrated;

[0031] The second calibration time is used for the camera device to determine the system time of the camera device based on the first network delay value and the first clock deviation value, so as to achieve time synchronization between the camera device, the controller and the lidar device.

[0032] In a second aspect, the present invention provides a time synchronization device applied to a vehicle, the vehicle including a controller, a lidar device, and a camera device; the device includes:

[0033] The first determining module is used to determine the target time source corresponding to the computing unit module based on the first time source corresponding to the processor module in the controller and the target point cloud data sent by the lidar device to the computing unit module in the controller.

[0034] The first time source is determined based on a standard clock signal used by the logic module in the controller to synchronize the processor module. The first time source is used to synchronize the time of the computing unit module, and the target point cloud data is used to calibrate the time of the synchronized computing unit module.

[0035] The first timing module is used to synchronize the time of the camera device based on the target time source and the first network delay value and the first clock deviation value corresponding to the computing unit module and the camera device, so as to realize the time synchronization of each module inside the controller and the time synchronization between the controller, the lidar device and the camera device.

[0036] Thirdly, the present invention provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the time synchronization method described in any one of the first aspects above.

[0037] Fourthly, the present invention provides a readable storage medium that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform steps in the time synchronization method as described in any of the embodiments of the first aspect above.

[0038] In this embodiment of the invention, based on the first time source corresponding to the processor module in the controller and the target point cloud data sent from the LiDAR device to the computing unit module, the target time source corresponding to the computing unit module is determined, enabling time synchronization of the computing unit module. Furthermore, since the first time source is determined based on a standard clock signal, and the target time source is determined by time synchronization based on the first time source and calibration based on the target point cloud data, the calibrated target time source of the computing unit module remains consistent with the standard clock signal, ensuring the accuracy of the target time source of the computing unit module. Further, by timing the camera device based on the target time source, a first network delay value, and a first clock deviation value, the clock error caused by network delay and clock deviation is considered, ensuring that the system time of the camera device after timing synchronization remains consistent with the target time source of the computing unit module. In other words, the various modules within the controller, namely the processor module and the computing unit module, achieve time synchronization, and the controller also achieves time synchronization with the LiDAR device and the camera device, ensuring the time consistency and accuracy of the controller, LiDAR device, and camera device. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of the steps of a time synchronization method provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the architecture of a controller provided in an embodiment of the present invention;

[0042] Figure 3 This is an interactive schematic diagram of a time synchronization method provided in an embodiment of the present invention;

[0043] Figure 4 This is a structural diagram of a time synchronization device provided in an embodiment of the present invention;

[0044] Figure 5 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Figure 1 This is a flowchart of the steps of a time synchronization method provided in an embodiment of the present invention, as follows: Figure 1 As shown, it is applied to a vehicle, which includes a controller, a lidar device, and a camera device.

[0047] In this embodiment of the invention, the controller can be a model for decision-making, planning, and control in an autonomous driving system. The controller may include a processor module, a computing unit module, and a logic module. The processor module can be an x86 module, used for handling logical operations, such as decision-making, planning, and control for autonomous driving. The computing unit module can be a unit used to process algorithms required by the autonomous driving system, for example, an NVIDIA Orin embedded computing unit module, specifically used for processing sensor data from lasers, cameras, etc., and AI calculations, such as image preprocessing, perception, perception fusion, point cloud processing, and fusion localization algorithms. The logic module can be a digital integrated circuit with logic functions constructed by the user as needed, such as a Complex Programmable Logic Device (CPLD). This logic module is used to receive the second pulse and clock signal (PPS+GPRMC) sent by the integrated navigation receiver and distribute the PPS+GPRMC to different modules for time synchronization. Communication between the processing module and the computing unit module can be achieved via Ethernet communication through an onboard switch, or they can be connected to a PCIe switch via PCIe to achieve PCIe communication between the modules. It is understood that the controller may include multiple computing unit modules, and these modules can communicate with the processor module via a switch. For example, Figure 2 A schematic diagram of a controller architecture is shown. The internal architecture of the controller may include, for example: Figure 2 The modules shown may specifically include an x86 module, a logic module (CPLD), a switch, and OrinA, OrinB, OrinC, and OrinD.

[0048] In this embodiment of the invention, the processor module can be used as the master clock node for time synchronization, the computing unit module as the boundary clock node, and the sensors (camera devices and lidar devices) as ordinary clock nodes. Through the time synchronization method of this embodiment of the invention, time synchronization of each module (processor module and computing unit module) within the controller and time synchronization between the controller and the sensors can be achieved.

[0049] The controller can be deployed in a vehicle, which can also be a smart driving device, a smart terminal device, or other device with data processing, network communication, and program execution functions; this embodiment of the invention does not impose any limitations on this. The vehicle is equipped with an autonomous driving system, which may include a controller, a lidar device, and a camera device. The lidar device emits a detection signal (laser beam) towards the target and then measures parameters such as the arrival time and intensity of the reflected or scattered signal to determine the target's distance, orientation, motion state, and surface optical properties. By sending the collected point cloud data to the computing unit module, the lidar device enables the computing unit module to perform calculations and decisions based on the point cloud data for subsequent assisted driving.

[0050] The method may include:

[0051] Step 101: Based on the first time source corresponding to the processor module in the controller and the target point cloud data sent by the lidar device to the computing unit module in the controller, determine the target time source corresponding to the computing unit module.

[0052] The first time source is determined based on a standard clock signal used by the logic module in the controller to synchronize the processor module. The first time source is used to synchronize the time of the computing unit module, and the target point cloud data is used to calibrate the time of the synchronized computing unit module.

[0053] In this embodiment of the invention, the logic module (CPLD), acting as the receiving and transmitting node of PPS+GPRMC, can be configured with a signal input interface and a signal output interface. The signal input interface is used to receive the second pulse signal (PPS signal) and the clock signal (GPRMC signal). The signal output interface is used to forward, transmit, and frequency-convert the PPS+GPRMC signal to achieve lossless signal multi-functionality. After receiving the standard clock signal (PPS+GPRMC) sent by the integrated navigation receiver through the signal input interface, the logic module can forward the standard clock signal (PPS+GPRMC) to the processor module and the lidar device through the signal output interface to synchronize the time of the processor module and the lidar device. In this way, the system time of the processor module and the system time of the lidar device are synchronized. Since the standard clock signal (PPS+GPRMC) includes Coordinated Universal Time (UTC), the processor module, upon receiving the standard clock signal (PPS+GPRMC), can directly determine it as the primary time source for the processor module. When the processor module's time mode is in the first mode, this primary time source is used as the system time source. In other words, the processor module's system time is consistent with Coordinated Universal Time at this time. For example, upon receiving the second pulse signal (PPS), the processor module clears the millisecond and smaller times in its internal system time (based on a crystal oscillator) to zero and begins calculating milliseconds, while waiting for the GPRMC statement to be sent via the serial port. Upon receiving the GPRMC message data, it extracts the hour, minute, second, year, month, and day (UTC time) from the message, corresponding to the rising edge time of the previous PPS. When the next rising edge of the PPS signal arrives, its corresponding time is the known UTC time + 1 second. The processor module's real-time clock (RTC) can be adjusted using this pulse signal and the known time.

[0054] In one possible implementation, if the logic module cannot receive the standard clock signal (PPS+GPRMC) sent by the integrated navigation receiver, the logic module can be timed using the processor module's own system time.

[0055] Furthermore, the processor module, acting as the master clock node, can use the first time source as a clock synchronization source to synchronize the time of the computing unit module. Specifically, the processor module can use Ethernet transmission protocols, such as the Precision Time Protocol (PTP) standard for network measurement and control systems, to synchronize the time of the computing unit module based on the first time source. However, due to potential transmission time differences caused by network latency and time differences between the processor module and the computing unit module during network communication for time synchronization, calibration can be performed based on the target point cloud data sent to the computing unit module by the LiDAR device to determine the target time source corresponding to the computing unit module. This target time source is used as the time source for the computing unit module's system time. It can be understood that the system time of the computing unit module is consistent with the system time of the processor module at this time. Since the LiDAR device is directly synchronized by the logic module, its system time is consistent with standard UTC. Therefore, based on the target point cloud data sent to the computing unit module by the LiDAR device, the time of the computing unit module can be further calibrated to ensure that its system time is consistent with standard UTC.

[0056] Step 102: Based on the target time source and the first network delay value and first clock deviation value corresponding to the computing unit module and the camera device, the camera device is timed to achieve time synchronization of each module inside the controller and time synchronization between the controller, the lidar device and the camera device.

[0057] In this embodiment of the invention, the target time source corresponding to the computing unit module is used as the clock synchronization source for the camera device to provide time synchronization. For example, the computing unit module can provide time synchronization to the camera device outside the controller via the camera network. However, since there may be transmission time differences due to network latency and time differences between the computing unit module and the camera network during network time synchronization between the computing unit module and the camera network, the target time source received by the camera device can be calibrated based on the first network delay value and the first clock deviation value corresponding to the computing unit module and the camera device to achieve time synchronization for the camera device. Thus, through the above time synchronization method, the target time source corresponding to the computing unit module can be kept consistent with the first time source corresponding to the processor module. Furthermore, through the time synchronization process between the computing unit module and the camera device, the system time of the camera device can be made consistent with the system time of the computing unit module. Since the target time source of the computing unit module is determined based on the target point cloud data of the LiDAR device, and the target time source is also the clock synchronization source for time synchronization of the camera device, time synchronization between the LiDAR device and the camera device can be achieved, facilitating synchronous triggering of the LiDAR device and the camera device. Furthermore, it simultaneously achieves time synchronization among the various modules within the controller (processor module and computing unit module) and among the controller, LiDAR device, and camera device. In other words, the time synchronization method in this embodiment ensures that the system clocks of the controller's internal and external components—the processor module, computing unit module, LiDAR device, and camera device—are all synchronized with Coordinated Universal Time (UTC), guaranteeing time consistency and accuracy among the modules in the vehicle.

[0058] In one possible implementation, in the actual use scenario of autonomous driving, since the lidar device and the camera device are synchronized in time, the lidar device can send a trigger pulse to the camera device by rotating the lidar device to control the corresponding camera device to perform synchronous exposure.

[0059] In summary, in this embodiment of the invention, the target time source corresponding to the computing unit module is determined based on the first time source corresponding to the processor module in the controller and the target point cloud data sent from the LiDAR device to the computing unit module. This enables time synchronization for the computing unit module. Furthermore, since the first time source is determined based on a standard clock signal, and the target time source is determined by time synchronization based on the first time source and calibration based on the target point cloud data, the target time source of the computing unit module remains consistent with the standard clock signal, ensuring the accuracy of the target time source. Further, by timing the camera device based on the target time source, a first network delay value, and a first clock deviation value, the clock error caused by network delay and clock deviation is considered, ensuring that the system time of the camera device after timing synchronization is consistent with the target time source of the computing unit module. In other words, the various modules within the controller—the processor module and the computing unit module—achieve time synchronization, and the controller also achieves time synchronization with the LiDAR device and the camera device, ensuring the time consistency and accuracy of the controller, LiDAR device, and camera device.

[0060] Optionally, embodiments of the present invention may include the following steps:

[0061] Step 201: When the processor module receives the standard clock signal sent by the logic module, the first time corresponding to the standard clock signal is set as the first time source of the processor module.

[0062] In this embodiment of the invention, when the logic module receives a standard clock signal from the integrated navigation receiver, it sends the standard clock signal to the processor module. Upon receiving the standard clock signal, the processor module sets the first time corresponding to the standard clock signal as its first time source. The standard clock signal can be PPS+GPRMC, and the first time can be Standard Universal Time (UTC).

[0063] Step 202: Set the second time corresponding to the first clock signal as the second time source of the processor module; the second time is the network time obtained by the processor module.

[0064] In this embodiment of the invention, when the processor module acquires a standard clock signal, in order to improve the stability of the system time of the processor module, the second time corresponding to the first clock signal can be set as the second time source of the processor module. The first clock signal can be the clock signal corresponding to the network time source acquired by the processor module. That is, the first clock signal can be the clock signal sent by the communication module (4G / 5G module), and the second time can be the network time acquired by the processor module.

[0065] Furthermore, the processor module is also equipped with an internal crystal oscillator timekeeping function, that is, the processor module can determine and display its own system time based on the internal timing module.

[0066] Step 203: When the time mode of the processor module is the second time source, the second time source is calibrated based on the predetermined target time difference between the first time and the second time.

[0067] In this embodiment of the invention, the processor module's time mode preferentially uses a first time source, namely satellite time. If satellite time cannot be obtained due to GPS malfunction or other reasons, the processor module's time mode can be switched to a second time source. If satellite or network time cannot be obtained due to satellite system or network malfunctions, the processor module's time mode switches to an internal timekeeping state. It can be understood that, when a stable satellite time is obtained, the processor module's time mode can be directly switched to the first time source.

[0068] When the processor module acquires both satellite time and network time, since satellite time is more accurate than network time, a target time difference between the first and second times can be predetermined. Then, when the processor module's time mode is set to the second time source, the second time source can be calibrated based on this target time difference. For example, the target time difference can be determined by multi-frame calculation, i.e., by obtaining the time difference between the first and second times within a preset time interval. This way, even when the processor module's time mode is set to the second time source, the time accuracy of the processor module can be ensured to a certain extent.

[0069] It is understandable that, when the processor module's time mode is the first time source, in step 101, during the process of the processor module providing time synchronization to the computing unit module, the target time source is determined based on the first time source corresponding to the processor module and the target point cloud data sent to the computing unit module by the radar device; when the processor module's time mode is the second time source, in step 101, during the process of the processor module providing time synchronization to the computing unit module, the target time source is determined based on the second time source corresponding to the processor module and the target point cloud data sent to the computing unit module by the radar device.

[0070] In this embodiment of the invention, by setting two time sources for the processor module, the processor module can select either time source as the time mode of the processor module, so that the processor module acts as the master clock node and has a clock switching function. This realizes the controller's all-weather, all-scene time synchronization function for sensors (LiDAR devices and camera devices), improves the time comprehensiveness of the controller's time synchronization, and ensures the stability of the controller's own time and the stability of external time synchronization.

[0071] Optionally, step 101 may include the following steps:

[0072] Step 301: Based on the first time source corresponding to the processor module, the computing unit module is timed to obtain the first time to be calibrated.

[0073] In this embodiment of the invention, the computing unit module is time-synchronized based on a first time source corresponding to the processor module. After the computing unit module obtains the clock signal sent by the processor module based on the first time source via Ethernet, it determines a first time to be calibrated. In the absence of network latency and clock deviation, the first time to be calibrated is consistent with standard UTC. However, due to network latency and clock deviation during actual signal transmission, the first time to be calibrated can be a time with a certain time error compared to standard UTC.

[0074] Step 302: When the computing unit module receives the target point cloud data sent by the lidar device to the computing unit module, obtain the timestamp information carried by the target point cloud data, and determine the first time information of the computing unit module receiving the target point cloud data.

[0075] In this embodiment of the invention, since the first time to be calibrated is not consistent with standard UTC, further error correction is required. Because the system time of the lidar device is determined based on the standard clock signal sent by the logic module, meaning the lidar device's system time is standard UTC, the first time to be calibrated can be calibrated based on the lidar device. Since the lidar device is used to collect point cloud data and send it to the computing unit module for subsequent analysis, the computing unit module, upon receiving the target point cloud data, can obtain the timestamp information carried by the target point cloud data and determine the first time information when the computing unit module receives the target point cloud data. Specifically, when the lidar device sends point cloud data to the computing unit module, it can determine the timestamp information based on its own system time and send the timestamp information to the computing unit module. This timestamp information is used to characterize the time corresponding to the lidar device sending the target point cloud data. The first time information is used to characterize the time corresponding to the computing unit module receiving the target point cloud data. Since the target point cloud data is sent from the LiDAR device to the computing unit module, the difference between the timestamp information and the first time information represents the second network delay value and the second clock deviation value between the LiDAR device and the computing unit module. That is, timestamp information + second network delay value + second clock deviation value = first time information.

[0076] Step 303: When the computing unit module receives the first computing message sent by the lidar device, it obtains the second time information carried in the first computing message and determines the third time information of the first feedback message sent by the computing unit module.

[0077] Wherein, the first calculation message is sent by the lidar device upon receiving the first feedback message, and the second time information is the reception time information of the lidar device upon receiving the first feedback message.

[0078] In this embodiment of the invention, the computing unit module sends a first feedback message to the lidar device and records the transmission time information of the first feedback message as third time information. Upon receiving the first feedback message, the lidar device acquires second time information indicating that it received the first feedback message. The lidar device then sends a first calculation message carrying the second time information to the computing unit module. Upon receiving the first calculation message, the computing unit module parses the first calculation message to obtain the second time information. The second time information characterizes the reception time of the first feedback message received by the lidar device, and the third time information characterizes the transmission time of the first feedback message sent by the computing unit device. Since the first feedback message is sent by the computing unit module to the lidar device, the third time information + the second network delay value - the second clock offset value = the second time information.

[0079] Step 304: Based on the timestamp information, the first time information, the second time information, and the third time information, calibrate the first time to be calibrated, and determine the target time source.

[0080] In this embodiment of the invention, a first time to be calibrated is performed based on timestamp information, first time information, second time information, and third time information. Based on the timestamp information, first time information, second time information, and third time information, a second network delay value and a second clock offset value can be determined. Subtracting the second network delay value and the second clock offset value from the first time to be calibrated yields a calibrated time that is consistent with standard world time, i.e., the system time corresponding to the processor module and the system time corresponding to the lidar device. The calibrated first time to be calibrated is then used as the system time of the computing unit module and determined as the target time source.

[0081] In this embodiment of the invention, the first time to be calibrated is calibrated by using the receiving and sending time information of the target point cloud data sent by the lidar device to the computing unit module, and the sending and receiving time information of the first feedback message sent by the computing unit module to the lidar device, in order to eliminate errors caused by network latency and clock deviation, and to ensure the accuracy of the target time source of the computing unit module and the time consistency with the processor module and the lidar device.

[0082] Optionally, step 304 may include:

[0083] Step 401: Based on the timestamp information, the first difference between the first time information, the second difference between the second time information and the third time information, determine the second network delay value and the second clock offset value.

[0084] In this embodiment of the invention, a first difference can be determined based on timestamp information and first time information. The first difference is the sum of a second network delay value and a second clock deviation value. According to the formula: Timestamp Information + Second Network Delay Value + Second Clock Deviation Value = First Time Information, the first difference can be determined as: First Time Information - Timestamp Information = Second Network Delay Value + Second Clock Deviation Value. A second difference can be determined based on second time information and third time information. The second difference is the difference between the second network delay value and the second clock deviation value. According to the formula: Third Time Information + Second Network Delay Value - Second Clock Deviation Value = Second Time Information, the second difference can be determined as: Second Time Information - Third Time Information = Second Network Delay Value - Second Clock Deviation Value. Thus, the second network delay value and the second clock deviation value can be determined based on the first difference and the second difference. The second network delay value characterizes the network latency generated when the computing unit module and the lidar device communicate via the network, and the second clock deviation value characterizes the clock deviation between the computing unit module and the lidar device.

[0085] For example, the second network delay value = [first difference + second difference] / 2, and the second clock skew value = [first difference - second difference] / 2.

[0086] Step 402: Based on the second network delay value and the second clock deviation value, calibrate the first time to be calibrated to obtain the system time corresponding to the computing unit module.

[0087] Step 403: Determine the system time corresponding to the computing unit module as the target time source.

[0088] In this embodiment of the invention, the first time to be calibrated is calibrated based on the second network delay value and the second clock deviation value to obtain the system time corresponding to the computing unit module. Specifically, the second network delay value and the second clock deviation value can be subtracted from the first time to be calibrated to obtain the system time corresponding to the computing unit module. The system time corresponding to the computing unit module is then determined as the target time source corresponding to the computing unit module.

[0089] In this embodiment of the invention, by determining the second network delay value and the second clock deviation value, the first time to be calibrated can be calibrated based on the second network delay value and the second clock deviation value, so that the system time corresponding to the computing unit module is more accurate and consistent with the system time of the processor module and the lidar device.

[0090] Optionally, embodiments of the present invention may include the following steps:

[0091] Step 501: Obtain the first sending time and the first receiving time of the first synchronization message; the first synchronization message is sent by the computing unit module to the camera device.

[0092] In this embodiment of the invention, the computing unit module can send a first synchronization message to the camera device, wherein the first synchronization message can be a Sync message. When sending the first synchronization message to the camera device, the computing unit module records the first transmission time of the first synchronization message. Upon receiving the first synchronization message, the camera device records the first reception time. In a practical scenario, the computing unit module can send a message carrying the first transmission time (a follow_up message) to the camera device, so that the camera device can parse the first transmission time upon receiving the message.

[0093] Step 502: Obtain the second sending time and the second receiving time of the second synchronization message; the second synchronization message is sent by the camera device to the computing unit module.

[0094] In this embodiment of the invention, the camera device can send a second synchronization message to the computing unit module. The second synchronization message can be a delayed feedback message. When sending the second synchronization message, the camera device can record a second sending time. Upon receiving the second synchronization message, the computing unit module records a second receiving time. In a practical scenario, the computing unit module can send a message carrying the second receiving time (a delayed calculation message) to the camera device, allowing the camera device to parse the second receiving time from the received message.

[0095] Step 503: Determine a first correspondence based on the first sending time and the first receiving time, and determine a second correspondence based on the second sending time and the second receiving time.

[0096] In this embodiment of the invention, since the first transmission time and the first reception time correspond to the data transmission process from the computing unit module to the camera device, a first correspondence can be determined based on the first transmission time and the first reception time, combined with the network latency between the computing unit module and the camera device and the clock deviation between the computing unit module and the camera device. The first correspondence can be: first transmission time + first network latency value + first clock deviation value = first reception time. Correspondingly, since the second transmission time and the second reception time correspond to the data transmission process from the camera device to the computing unit module, a second correspondence can be determined based on the second transmission time and the second reception time, combined with the network latency between the computing unit module and the camera device and the clock deviation between the computing unit module and the camera device. The second correspondence can be: second transmission time + first network latency value - first clock deviation value = second reception time.

[0097] Step 504: Based on the first correspondence and the second correspondence, determine the first network delay value and the first clock offset value.

[0098] In this embodiment of the invention, based on the first correspondence and the second correspondence, the first network delay value and the first clock deviation value are calculated and determined. Specifically, the first network delay value = [(first receiving time - first sending time) + (second receiving time - second sending time)] / 2, and the first clock deviation value = [(first receiving time - first sending time) - (second receiving time - second sending time)] / 2.

[0099] In this embodiment of the invention, by transmitting the first synchronization message and the second synchronization message between the computing unit module and the camera device, the first network delay value and the first clock deviation value can be determined. By determining the first network delay value and the first clock deviation value, the time of the camera device can be calibrated to achieve the accuracy of the camera device's time synchronization and the time consistency with the computing unit module.

[0100] Optionally, step 102 may include the following steps:

[0101] Step 601: Based on the target time source, the calculation unit module provides time synchronization to the camera device to obtain the second time to be calibrated;

[0102] The second calibration time is used for the camera device to determine the system time of the camera device based on the first network delay value and the first clock deviation value, so as to achieve time synchronization between the camera device, the controller and the lidar device.

[0103] In this embodiment of the invention, the computing unit module synchronizes the camera device's time based on the target time source through the camera network to obtain a second time to be calibrated for the camera device. This second time to be calibrated is used by the camera device to determine its system time based on a first network delay value and a first clock deviation value. Specifically, the second time to be calibrated can be calibrated based on the first network delay value and the first clock deviation value. After determining the second time to be calibrated, the camera device subtracts the first network delay value and the first clock deviation value from it to determine the system time of the camera device. This enables time synchronization between the camera device, the controller, and the LiDAR.

[0104] In this embodiment of the invention, the second time to be calibrated is calibrated based on the first network delay value and the first clock deviation value, which can realize the time synchronization between the camera device, the controller, and the lidar device.

[0105] For example, Figure 3 An interactive schematic diagram of a time synchronization method is shown, such as... Figure 3 As shown, the controller internally includes an x86 module, a logic module (CPLD), a switch, and OrinA, OrinB, OrinC, and OrinD. The logic module (CPLD) sends a standard clock signal (PPS+GPRMC) to the x86 module and the LiDAR device. The x86 module synchronizes time with OrinA, OrinB, OrinC, and OrinD via Ethernet. Any Orin module, such as OrinB, synchronizes time with the camera device through the camera network.

[0106] Figure 4 This is a schematic diagram of the structure of a time synchronization device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, this device is applied to a vehicle, which includes a controller, a lidar device, and a camera device; specifically, the device may include:

[0107] The first determining module 701 is used to determine the target time source corresponding to the computing unit module based on the first time source corresponding to the processor module in the controller and the target point cloud data sent by the lidar device to the computing unit module in the controller.

[0108] The first time source is determined based on a standard clock signal used by the logic module in the controller to synchronize the processor module. The first time source is used to synchronize the time of the computing unit module, and the target point cloud data is used to calibrate the time of the synchronized computing unit module.

[0109] The first timing module 702 is used to synchronize the time of the camera device based on the target time source and the first network delay value and the first clock deviation value corresponding to the computing unit module and the camera device, so as to realize the time synchronization of each module inside the controller and the time synchronization between the controller, the lidar device and the camera device.

[0110] This invention provides a time synchronization device. Based on a first time source corresponding to the processor module in the controller and target point cloud data sent from the LiDAR device to the computing unit module, a target time source corresponding to the computing unit module is determined, enabling time synchronization for the computing unit module. Since the first time source is determined based on a standard clock signal, and the target time source is determined through time synchronization based on the first time source and calibration based on the target point cloud data, the calibrated target time source of the computing unit module remains consistent with the standard clock signal, ensuring the accuracy of the target time source of the computing unit module. Furthermore, by synchronizing the camera device with the target time source, a first network delay value, and a first clock deviation value, the clock error caused by network delay and clock deviation is considered, ensuring that the system time of the camera device after synchronization is consistent with the target time source of the computing unit module. In other words, the various modules within the controller, namely the processor module and the computing unit module, achieve time synchronization, and the controller also achieves time synchronization with the LiDAR device and the camera device, ensuring the time consistency and accuracy of the controller, LiDAR device, and camera device.

[0111] Optionally, the device may further include:

[0112] The first setting module is used to set the first time corresponding to the standard clock signal as the first time source of the processor module when the processor module receives the standard clock signal sent by the logic module.

[0113] The second setting module is used to set the second time corresponding to the first clock signal as the second time source of the processor module; the second time is the network time obtained by the processor module.

[0114] The first calibration module is used to calibrate the second time source based on a predetermined target time difference between the first time and the second time when the time mode of the processor module is the second time source.

[0115] Optionally, the first determining module 701 includes:

[0116] The second time synchronization module is used to synchronize the time of the computing unit module based on the first time source corresponding to the processor module, so as to obtain the first time to be calibrated;

[0117] The first acquisition module is used to acquire the timestamp information carried by the target point cloud data and determine the first time information of the target point cloud data received by the computing unit module when the computing unit module receives the target point cloud data sent by the lidar device to the computing unit module.

[0118] The second acquisition module is used to acquire the second time information carried in the first calculation message when the calculation unit module receives the first calculation message sent by the lidar device, and to determine the third time information of the first feedback message sent by the calculation unit module.

[0119] Wherein, the first calculation message is sent by the lidar device upon receiving the first feedback message, and the second time information is the reception time information of the lidar device upon receiving the first feedback message;

[0120] The second calibration module is used to calibrate the first time to be calibrated based on the timestamp information, the first time information, the second time information, and the third time information, and to determine the target time source.

[0121] Optionally, the second calibration module includes:

[0122] The second determining module is used to determine a second network delay value and a second clock deviation value based on the timestamp information, the first difference between the first time information, the second difference between the second time information and the third time information.

[0123] The first calibration submodule is used to calibrate the first time to be calibrated based on the second network delay value and the second clock deviation value, so as to obtain the system time corresponding to the computing unit module;

[0124] The third determining module is used to determine the system time corresponding to the computing unit module as the target time source.

[0125] Optionally, the device may further include the following modules:

[0126] The third acquisition module is used to acquire the first transmission time and the first reception time of the first synchronization message; the first synchronization message is sent by the computing unit module to the camera device;

[0127] The fourth acquisition module is used to acquire the second transmission time and the second reception time of the second synchronization message; the second synchronization message is sent by the camera device to the computing unit module;

[0128] The fourth determining module is used to determine a first correspondence based on the first sending time and the first receiving time, and to determine a second correspondence based on the second sending time and the second receiving time;

[0129] The fifth determining module is used to determine the first network delay value and the first clock deviation value based on the first correspondence and the second correspondence.

[0130] Optionally, the first timing module 702 includes:

[0131] The first time synchronization submodule is used to synchronize the time of the camera device with the time of the calculation unit module based on the target time source to obtain the second time to be calibrated;

[0132] The second calibration time is used for the camera device to determine the system time of the camera device based on the first network delay value and the first clock deviation value, so as to achieve time synchronization between the camera device, the controller and the lidar device.

[0133] The present invention also provides an electronic device, see [link to relevant documentation]. Figure 5 It includes: a processor 801, a memory 802, and a computer program 8021 stored in the memory and executable on the processor, wherein the processor executes the program to implement the time synchronization method of the foregoing embodiments.

[0134] The present invention also provides a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the time synchronization method of the foregoing embodiments.

[0135] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0136] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0137] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0138] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0139] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0140] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the sorting device according to the present invention. The present invention can also be implemented as a device or apparatus program for performing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0141] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0142] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0143] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0145] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A time synchronization method, characterized in that, Applied to a vehicle, the vehicle including a controller, a lidar device, and a camera device; the method includes: Based on the first time source corresponding to the processor module in the controller and the target point cloud data sent by the lidar device to the computing unit module in the controller, the target time source corresponding to the computing unit module is determined; wherein, the target time source is the calibrated system time of the computing unit module; when the lidar device sends the target point cloud data to the computing unit module, it determines timestamp information based on its own system time and sends the timestamp information to the computing unit module, and the timestamp information is used to characterize the time corresponding to the lidar device sending the target point cloud data; The first time source is determined based on a standard clock signal used by the logic module in the controller to synchronize the processor module. The first time source is used to synchronize the time of the computing unit module, and the target point cloud data is used to calibrate the time of the synchronized computing unit module. Based on the target time source and the first network delay value and first clock deviation value corresponding to the computing unit module and the camera device, the camera device is timed to achieve time synchronization of each module inside the controller and time synchronization between the controller, the lidar device and the camera device; The logic module is used to distribute the received standard clock signal to the processor module and the lidar device to provide time synchronization for the processor module and the lidar device.

2. The method according to claim 1, characterized in that, The method further includes: When the processor module receives a standard clock signal sent by the logic module, the first time corresponding to the standard clock signal is set as the first time source of the processor module; The second time corresponding to the first clock signal is set as the second time source of the processor module; the second time is the network time obtained by the processor module; the first clock signal is the clock signal corresponding to the network time obtained by the processor module, and the first clock signal is sent by the communication module. When the time mode of the processor module is the second time source, the second time source is calibrated based on the predetermined target time difference between the first time and the second time.

3. The method according to claim 1, characterized in that, The step of determining the target time source corresponding to the computing unit module based on the first time source corresponding to the processor module in the controller and the target point cloud data sent by the lidar device to the computing unit module in the controller includes: Based on the first time source corresponding to the processor module, the computing unit module is timed to obtain the first time to be calibrated. When the computing unit module receives target point cloud data sent by the lidar device to the computing unit module, it obtains the timestamp information carried by the target point cloud data and determines the first time information of the computing unit module receiving the target point cloud data. When the computing unit module receives the first computing message sent by the lidar device, it obtains the second time information carried in the first computing message and determines the third time information of the first feedback message sent by the computing unit module. Wherein, the first calculation message is sent by the lidar device upon receiving the first feedback message, and the second time information is the reception time information of the lidar device upon receiving the first feedback message; Based on the timestamp information, the first time information, the second time information, and the third time information, the first time to be calibrated is calibrated, and the target time source is determined.

4. The method according to claim 3, characterized in that, The step of calibrating the first time to be calibrated based on the timestamp information, the first time information, the second time information, and the third time information, and determining the target time source, includes: A first difference is determined based on the timestamp information and the first time information; a second difference is determined based on the second time information and the third time information; a second network delay value and a second clock offset value are determined based on the first difference and the second difference. Based on the second network delay value and the second clock deviation value, the first time to be calibrated is calibrated to obtain the system time corresponding to the computing unit module; The system time corresponding to the computing unit module is determined as the target time source.

5. The method according to claim 1, characterized in that, Before timing the camera device based on the target time source, the first network delay value corresponding to the computing unit module and the camera device, and the first clock offset value, the method further includes: The first transmission time and the first reception time of the first synchronization message are obtained; the first synchronization message is sent by the computing unit module to the camera device; The second transmission time and the second reception time of the second synchronization message are obtained; the second synchronization message is sent by the camera device to the computing unit module. Based on the first transmission time and the first reception time, a first correspondence is determined, and based on the second transmission time and the second reception time, a second correspondence is determined; the first reception time is obtained by adding the first transmission time, the first network delay value, and the first clock deviation value; the second reception time is obtained by subtracting the first clock deviation value from the sum of the second transmission time and the first network delay value. Based on the first correspondence and the second correspondence, the first network delay value and the first clock offset value are determined.

6. The method according to claim 1, characterized in that, The step of providing time synchronization to the camera device based on the target time source, the first network delay value corresponding to the computing unit module and the camera device, and the first clock deviation value includes: Based on the target time source, the calculation unit module provides time synchronization to the camera device to obtain the second time to be calibrated; The second calibration time is used for the camera device to determine the system time of the camera device based on the first network delay value and the first clock deviation value, so as to achieve time synchronization between the camera device, the controller and the lidar device.

7. A time synchronization device, characterized in that, Applied to a vehicle, the vehicle including a controller, a lidar device, and a camera device; the device includes: The first determining module is used to determine the target time source corresponding to the computing unit module based on the first time source corresponding to the processor module in the controller and the target point cloud data sent by the lidar device to the computing unit module in the controller; wherein, the target time source is the calibrated system time of the computing unit module; when the lidar device sends the target point cloud data to the computing unit module, it determines timestamp information based on its own system time and sends the timestamp information to the computing unit module, and the timestamp information is used to characterize the time corresponding to the lidar device sending the target point cloud data; The first time source is determined based on a standard clock signal used by the logic module in the controller to synchronize the processor module. The first time source is used to synchronize the time of the computing unit module, and the target point cloud data is used to calibrate the time of the synchronized computing unit module. The first time synchronization module is used to synchronize the time of the camera device based on the target time source and the first network delay value and the first clock deviation value corresponding to the computing unit module and the camera device, so as to realize the time synchronization of each module inside the controller and the time synchronization between the controller, the lidar device and the camera device. The logic module is used to distribute the received standard clock signal to the processor module and the lidar device to provide time synchronization for the processor module and the lidar device.

8. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the time synchronization method as described in any one of claims 1-6.

9. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the time synchronization method according to any one of claims 1-6.

Images