Time synchronization method, multi-sensor embedded system and storage medium

Abstract

The embodiment of the application relates to the technical field of embedded systems, and discloses a time synchronization method, a multi-sensor embedded system and a storage medium. The method is applied to a multi-sensor embedded system, the multi-sensor embedded system comprises a master control module, a laser radar module and a camera module, and the method comprises the following steps: according to a communication connection between the master control module and an external device, controlling the master control module and the external device to complete time synchronization, wherein the master control module and the external device are connected in communication through a universal serial bus interface; and according to communication connections between the master control module and the laser radar module and the camera module, respectively, controlling the master control module to perform time synchronization on the laser radar module and the camera module. By taking the external device as a reference to calibrate the master control module first and then taking the master control module as a reference to calibrate the laser radar module and the camera module, error accumulation is reduced in phased calibration, the stability of time is enhanced, and the real-time performance and reliability of the overall system are improved.

Description

Technical Field

[0001] This application relates to the field of embedded system technology, and in particular to a time synchronization method, a multi-sensor embedded system, and a storage medium. Background Technology

[0002] In multi-sensor embedded systems, different sensors typically operate independently and have different clock sources. If the clock sources of these sensors are not synchronized, data errors or inconsistencies will occur during data fusion and analysis, thus affecting the overall stability and reliability of the system. Summary of the Invention

[0003] This application provides a time synchronization method, a multi-sensor embedded system, and a storage medium, aiming to solve the problem of clock source asynchrony between different sensors in a multi-sensor embedded system.

[0004] In a first aspect, embodiments of this application provide a time synchronization method applied to a multi-sensor embedded system, the multi-sensor embedded system including a main control module, a LiDAR module, and a camera module, the method comprising:

[0005] Based on the communication connection between the main control module and the external device, the main control module and the external device are controlled to complete time synchronization. The main control module and the external device communicate through a universal serial bus interface.

[0006] Based on the communication connections between the main control module and the LiDAR module and camera module respectively, the main control module synchronizes the time of the LiDAR module and camera module. By using the clock of an external device as a reference source, the main control module's time is first calibrated, and then the LiDAR module and camera module are calibrated using the main control module's time as a reference. This staged calibration reduces error accumulation, enhances the time stability of the multi-sensor embedded system, and also improves the real-time performance and reliability of the overall system.

[0007] In some embodiments, based on the communication connection between the main control module and the external device, the main control module is controlled to complete time synchronization with the external device. This includes determining the current local time of the main control module based on its local time before time synchronization, the clock offset between the main control module and the external device, the clock drift rate between the main control module and the external device, and the time interval between the last time the main control module completed time synchronization with the external device and the current time. By considering the initial local time and clock offset of the main control module, as well as the clock drift, and compensating for the clock drift in real time, the current local time of the main control module is accurately calculated, significantly improving the accuracy of time synchronization between the main control module and the external device.

[0008] In some embodiments, based on the communication connection between the main control module and the external device, controlling the main control module to complete time synchronization with the external device includes: configuring the clock of the main control module as a slave clock and the clock of the external device as the master clock to complete time synchronization; based on the communication connection between the main control module and the LiDAR module and the camera module respectively, controlling the main control module to perform time synchronization with the LiDAR module and the camera module includes: configuring the clock of the main control module as the master clock and the clocks of the LiDAR module and the camera module as slave clocks to perform time synchronization.

[0009] In some embodiments, based on the communication connection between the main control module and the external device, the main control module is controlled to complete time synchronization with the external device, including:

[0010] The main control module receives synchronization messages sent by external devices through a transmission channel and records the second local time of the received synchronization messages. The physical carrier of the transmission channel is a Universal Serial Bus (USB) interface, and the synchronization messages include the first local time of the external device sending the synchronization messages. The main control module sends delay request messages through the transmission channel and records the third local time of the delayed request messages. The main control module receives delay response messages sent by external devices, which include the fourth local time of the external device receiving the delayed request messages. Based on the first, second, third, and fourth local times, the path delay and clock offset between the main control module and the external devices are calculated. The current local time of the main control module is adjusted according to the clock offset. Time synchronization between the main control module and external devices is achieved through a bidirectional timestamp interaction mechanism based on the USB interface and the HMI (Human Machine Interface) device protocol, eliminating the need for dedicated high-precision clock synchronization hardware and reducing hardware costs. Furthermore, the HMI device protocol is widely supported by most operating systems, allowing the time synchronization method based on the HMI device protocol to be used in various types of devices and systems without the need to develop dedicated drivers for different operating systems, reducing development costs and adaptation difficulties.

[0011] In some embodiments, the transmission channel between the main control module and the external device is implemented via interrupt transmission. By supporting interrupt transmission based on the human-machine interface device protocol, the transmission of communication data between the main control module and the external device achieves high real-time performance, ensuring timely time synchronization between the main control module and the external device.

[0012] In some embodiments, based on the communication connection between the main control module and the external device, controlling the main control module and the external device to complete time synchronization further includes: performing structured encapsulation processing on the actual communication data transmitted between the main control module and the external device to obtain a structured communication data packet. The structured communication data packet includes a frame header start bit, a data payload length, a cyclic redundancy check, and communication data. The frame header start bit is used to identify the starting position of the structured communication data packet, the data payload length is the number of bytes of the actual transmitted communication data, the cyclic redundancy check is used to verify the integrity of the structured communication data packet, and the communication data includes synchronization messages, delay request messages, and delay response messages. By using a standard human-machine interface protocol and a customized encapsulation and verification mechanism for communication data to ensure correct data transmission, the reliability and robustness of time synchronization between the main control module and the external device are improved.

[0013] In some embodiments, based on the communication connections between the main control module and the lidar module and the camera module respectively, the main control module is controlled to perform time synchronization between the lidar module and the camera module, including: controlling the main control module to perform time synchronization between the lidar module and the camera module simultaneously, or controlling the main control module to perform time synchronization between the lidar module and the camera module sequentially.

[0014] In some embodiments, based on the communication connections between the main control module and the LiDAR module and the camera module respectively, the main control module is controlled to synchronize the time of the LiDAR module and the camera module. This includes: the main control module and the LiDAR module synchronizing time via a precise time protocol; and the main control module sending a time synchronization pulse to the camera module via a hardware interface for time synchronization. Using existing hardware interfaces and communication protocols for time synchronization allows for multi-module time synchronization without adding additional high-precision clock hardware or complex wiring, reducing the complexity of multi-module time synchronization and lowering hardware costs. Furthermore, both the precise time protocol and the synchronization pulse have a certain degree of anti-interference capability, ensuring the stability and reliability of the time synchronization between the main control module and the LiDAR module and camera module respectively.

[0015] Secondly, embodiments of this application provide a multi-sensor embedded system, which includes a main control module, a lidar module, a camera module, and a central processing unit;

[0016] Based on the communication connection between the main control module and the external device, the central processing unit is used to control the main control module to complete time synchronization with the external device. The main control module and the external device communicate through a universal serial bus interface. Based on the communication connection between the main control module and the LiDAR module and the camera module respectively, the central processing unit is used to control the main control module to synchronize the time of the LiDAR module and the camera module.

[0017] Thirdly, embodiments of this application provide a storage medium on which a computer program is stored, and when the computer program is executed by a processor, it is used to implement any of the time synchronization methods described above.

[0018] The technical solution provided in this application has the following beneficial effects:

[0019] This application discloses a time synchronization method, a multi-sensor embedded system, and a storage medium. Time synchronization is performed based on the universal serial bus interface between the external device and the main control module of the multi-sensor embedded system, as well as the Human Interface Device (HID) protocol. Then, the main control module of the calibrated multi-sensor embedded system performs time synchronization on the LiDAR module and the camera module. On the one hand, since the HID protocol is widely supported by most operating systems, it can be used in different types of devices and systems without the need to develop dedicated drivers for different operating systems, reducing development costs and improving adaptability. On the other hand, through staged calibration, error accumulation is reduced, enhancing system stability and improving the overall system's real-time performance and reliability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a time synchronization method provided in an embodiment of this application;

[0022] Figure 2 This is an architecture diagram of a multi-sensor embedded system for time synchronization with external devices, provided in an embodiment of this application.

[0023] Figure 3 This is a flowchart illustrating time synchronization between a main control module and an external device, as provided in an embodiment of this application.

[0024] Figure 4 This is a schematic diagram illustrating the time synchronization between a main control module and an external device, as provided in an embodiment of this application.

[0025] Figure 5 This is a schematic diagram of a multi-sensor embedded system structure provided in an embodiment of this application.

[0026] The accompanying figures are labeled as follows:

[0027] 1. External devices; 2. Multi-sensor embedded system; 21. Main control module; 22. LiDAR module; 23. Camera module; 24. Central processing unit; T1. First local time; T2. Second local time; T3. Third local time; T4. Fourth local time. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described clearly and in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0029] In multi-sensor embedded systems, different sensors typically operate independently with different clock sources. If these clock sources are out of sync, data errors or inconsistencies can occur during data fusion and analysis, affecting the overall stability and reliability of the system. For example, in autonomous vehicles equipped with both LiDAR and cameras, the LiDAR and cameras need precise time synchronization to accurately match and fuse LiDAR point cloud data and image data, thereby achieving accurate perception and recognition of the surrounding environment.

[0030] Existing time synchronization methods include those based on the Network Time Protocol (NTP) and those based on custom time synchronization protocols. NTP-based methods are highly dependent on the network; network latency and packet loss significantly impact synchronization accuracy. Poor network conditions can lead to substantial synchronization errors. Custom time synchronization protocols are typically designed for specific application scenarios, exhibiting poor protocol compatibility and versatility, limiting system scalability and interoperability. Furthermore, the lack of unified standards and specifications increases the difficulty of system maintenance and upgrades.

[0031] Based on this, embodiments of this application provide a time synchronization method, a multi-sensor embedded system, and a storage medium. This time synchronization method, based on the Human Interface Device Protocol (HID), introduces a time drift estimation and compensation mechanism, further improving the accuracy of time synchronization. Furthermore, this time synchronization method, also based on the standard HID protocol, uses a custom data encapsulation and verification mechanism to ensure correct data transmission, enhancing the system's reliability and robustness. The embodiments in this application are all described using a multi-sensor embedded system applied to automotive equipment; however, in other embodiments of this application, the multi-sensor embedded system can also be applied to other products, such as robots and drones.

[0032] like Figure 1 and Figure 2 As shown, Figure 1 A flowchart illustrating a time synchronization method provided in an embodiment of this application. Figure 2 This diagram illustrates an architecture for time synchronization between a multi-sensor embedded system and an external device, as provided in an embodiment of this application. This time synchronization method is applied to a multi-sensor embedded system 2. In some embodiments, the multi-sensor embedded system 2 is a high-precision environmental perception system that integrates multiple sensors and possesses real-time data processing capabilities, making it suitable for scenarios such as autonomous driving, robot navigation, industrial automation, and intelligent security.

[0033] In one embodiment, such as Figure 2 As shown, the multi-sensor embedded system 2 includes a main control module 21, a lidar module 22, and a camera module 23. The main control module 21 is responsible for the acquisition, processing, and transmission of data for the entire system. The main control module 21 interacts with the lidar module 22 and the camera module 23 via a communication interface, receiving large amounts of data from the sensors in real time and performing preprocessing, analysis, and fusion. Furthermore, the main control module 21 is also responsible for communication with other external devices, ensuring the stable operation and efficient collaboration of the entire system.

[0034] The lidar module 22, through the emission and reception of lasers, can acquire real-time three-dimensional spatial information of the surrounding environment, generating high-resolution point cloud data to provide the system with depth information of the environment. The camera module 23 captures visual information of the environment through a camera, acquiring color images of the environment and providing the system with rich image data.

[0035] In the multi-sensor embedded system 2, the main control module 21 fuses data from the lidar module 22 and the camera module 23 to achieve more accurate environmental perception and decision-making. Specifically, the data acquired by the lidar module 22 and the camera module 23 need to be time-aligned for effective data fusion. Therefore, the point cloud data from the lidar and the image data from the camera require precise time synchronization to improve the measurement accuracy of the multi-sensor embedded system 2.

[0036] like Figure 1 As shown, the time synchronization method includes steps S10 to S20:

[0037] In some embodiments, step S10 is first executed to align the clock of the main control module 21 with the clock of the external device 1. Then, step S20 is executed to align the clocks of the lidar module 22 and the camera module 23 with the clock of the main control module 21, thereby achieving time synchronization of the multi-sensor embedded system 2. Through this phased time synchronization method, the clocks of the external device, the main control module 21, the lidar module 22, and the camera module 23 are kept aligned, thus providing reliable support for accurate perception and intelligent decision-making in complex environments.

[0038] S110. Based on the communication connection between the main control module 21 and the external device 1, control the main control module 21 and the external device 1 to complete time synchronization, wherein the main control module 21 and the external device 1 communicate through a universal serial bus interface.

[0039] In some embodiments, the multi-sensor embedded system 2 is connected to the external device 1 via a Universal Serial Bus (USB) interface. The external device 1 is a USB host capable of generating accurate and stable time signals for use as a time reference source; the multi-sensor embedded system 2 is a USB slave device used as a clock to be calibrated. The multi-sensor embedded system 2 and the external device 1 communicate in real-time based on the HID protocol through a standardized interrupt transmission mechanism, thereby achieving time synchronization.

[0040] Specifically, HID time synchronization synchronizes the timestamp generated by the high-precision clock source of the external device 1 on the host side with the clock of the multi-sensor embedded system 2 on the slave side through a specific data format defined by the HID report descriptor. This mechanism enables the multi-sensor embedded system 2 to obtain the absolute time reference of the external device 1 and to achieve timestamp alignment of the multi-sensor nodes within the system based on this reference.

[0041] In one embodiment, the transmission channel between the main control module 21 and the external device 1 in the multi-sensor embedded system 2 is performed via interrupt transmission, enabling communication data transmission between the main control module 21 and the external device 1. The external device 1 can be a car, robot, drone, etc. By supporting interrupt transmission based on the HID protocol, the transmission of communication data between the main control module 21 and the external device 1 achieves high real-time performance, ensuring timely time synchronization between them.

[0042] like Figure 3 and Figure 4 As shown, Figure 3 A flowchart illustrating time synchronization between a main control module and an external device is provided as an embodiment of this application. Figure 4 This diagram illustrates the principle of time synchronization between the master control module and the external device provided in this embodiment. In some embodiments, the external device 1, acting as a Universal Serial Bus (USB) interface master, periodically sends synchronization commands to the master control module 21, which acts as a USB slave, via HID interrupt transmission. The master control module 21 calculates the clock offset and path delay with the external device 1 by recording bidirectional timestamps, continuously adjusting its clock to maintain consistency with the high-precision clock of the external device 1. Based on this clock reference, timestamp alignment of multiple sensor nodes within the system is achieved. The specific steps for time synchronization between the master control module 21 and the external device 1 based on the HID protocol are as follows:

[0043] In one embodiment, when the main control module 21 and the external device 1 synchronize their time based on the HID protocol, the clock of the main control module 21 is configured as the slave clock, and the clock of the external device 1 is configured as the master clock, thereby completing the time synchronization between the main control module 21 and the external device 1.

[0044] S111, the main control module 21 receives the synchronization message sent by the external device 1 through the transmission channel and records the second local time T2 of the main control module 21 receiving the synchronization message. The physical carrier of the transmission channel is a universal serial bus interface, and the synchronization message includes the first local time T1 of the synchronization message sent by the external device 1.

[0045] In one embodiment, when the master clock of external device 1 arrives at a preset synchronization period, it actively sends a synchronization message to the slave clock of master control module 21 through the transmission channel. The synchronization message includes the first local time T1 of the master clock of external device 1 at the time of sending. The slave clock of master control module 21 receives the synchronization message through the transmission channel and records the second local time T2 of the slave clock of master control module 21 at the time of receiving the synchronization message.

[0046] S112, the main control module 21 sends a delay request message through the transmission channel and records the third local time T3 when the main control module 21 sends the delay request message.

[0047] In one embodiment, after the master control module 21 slave clock records the second local time T2 when receiving the synchronization message, the master control module 21 slave clock immediately sends a delay request message to the master clock of the external device 1 through the transmission channel; simultaneously, it records the third local time T3 of the master control module 21 slave clock at the time of transmission. The delay request message is used to measure the path delay from the master control module 21 slave clock to the master clock of the external device 1.

[0048] S113, the main control module 21 receives a delay response message sent by the external device 1, wherein the delay response message includes the fourth local time T4 at which the external device 1 receives the delay request message.

[0049] In one embodiment, the master clock of external device 1 receives a delay request message through a transmission channel and records the fourth local time T4 of the master clock at the moment of reception. After recording the fourth local time T4 at the time of receiving the delay request message, the master clock of external device 1 immediately sends a delay response message to the slave clock of the master control module 21 through the transmission channel. The delay response message includes the precise value of the fourth local time T4 at the moment when the master clock of external device 1 receives the delay request message.

[0050] S114. Calculate the path delay and clock offset between the main control module 21 and the external device 1 based on the first local time T1, the second local time T2, the third local time T3 and the fourth local time T4.

[0051] In one embodiment, the master control module 21 receives a delay response message from the slave clock through the transmission channel, and calculates the path delay and clock offset between the slave clock of the master control module 21 and the master clock of the external device 1 based on the first local time T1, the second local time T2, the third local time T3 and the fourth local time T4.

[0052] Specifically, the path delay is: PathDelay=((T4-T3)+(T2-T1)) / 2, and the clock offset is: ClockOffset=((T3-T4)+(T2-T1)) / / 2.

[0053] S115. Adjust the current local time of the master control module 21 according to the clock offset. The slave clock of the master control module 21 adjusts the local time of the slave clock of the master control module 21 according to the ClockOffset.

[0054] By employing a bidirectional timestamp interaction mechanism based on the Universal Serial Bus interface and the HID protocol, time synchronization between the main control module 21 and external device 1 is achieved, eliminating the need for dedicated high-precision clock synchronization hardware and reducing hardware costs. Furthermore, the HID protocol is widely supported by most operating systems, allowing time synchronization methods based on it to be used in various types of devices and systems without requiring the development of dedicated drivers for different operating systems, thus reducing development costs and adaptation difficulties.

[0055] Although the master control module 21 and external device 1 have completed time synchronization based on the HID protocol through the above steps, achieving clock alignment between the slave clock of the master control module 21 and the master clock of external device 1, slight clock drift may still occur between the slave clock of the master control module 21 and the master clock of external device 1 due to hardware and environmental factors. To further improve the clock alignment accuracy between the slave clock of the master control module 21 and the master clock of external device 1, it is necessary to estimate the clock drift during the time synchronization process and compensate for this drift, thereby achieving high-precision clock alignment between the slave clock of the master control module 21 and the master clock of external device 1.

[0056] In some embodiments, the current local time of the main control module 21 is determined based on the local time of the main control module 21 before time synchronization, the clock offset between the main control module 21 and the external device 1, the clock drift rate between the main control module 21 and the external device 1, and the time interval between the last time the main control module 21 and the external device 1 completed time synchronization and the current time. By considering the initial local time and clock offset of the main control module 21, as well as the clock drift, and compensating for the clock drift in real time, the current local time of the main control module 21 is accurately calculated, significantly improving the accuracy of time synchronization between the main control module 21 and the external device 1.

[0057] In one embodiment, the master control module 21 sets the slave clock to t at each time synchronization is performed. i Furthermore, the clock offset between the main control module 21's slave clock and the external device 1's master clock is set to ClockOffset each time time synchronization is performed. i Assuming the clock drift is linear, the clock offset (ClockOffset) of the master control module 21 is analyzed at each time synchronization and the corresponding time. i The clock drift rate can be estimated using the least squares method. According to the least squares formula, the clock drift rate can be calculated using the following formula:

[0058]

[0059] in, It is the synchronization of all time moments. It is all clock offsets (ClockOffset) i The average value. Once the DriftRate is calculated, the current time (AdjustedTime) of the master control module 21's slave clock can be dynamically adjusted based on the drift rate to keep it aligned with the master clock of external device 1. The current time of the master control module 21's slave clock can be calculated using the following formula:

[0060] AdjustedTime=LocalTime+ClockOffset+DriftRate×Δt

[0061] Wherein, LocalTime is the local time of the main control module 21 before the clock was synchronized, and Δt is the time interval from the last time the main control module 21 completed time synchronization to the present.

[0062] This dynamic adjustment mechanism based on the least squares method effectively corrects the clock drift of the master control module 21's slave clock, ensuring that the slave clock of the master control module 21 maintains a high degree of consistency with the master clock of the external device 1 during multiple time synchronizations. This method not only improves the accuracy of time synchronization but also enhances the stability and reliability of the system.

[0063] In the bidirectional timestamp interaction mechanism between the main control module 21's slave clock and the external device 1's master clock based on the aforementioned HID protocol, the correctness and integrity of the communication data transmitted between the main control module 21 and the external device 1 are also crucial. To further improve the reliability and robustness of data transmission between the main control module 21 and the external device 1, a customized data encapsulation and verification mechanism is implemented based on the standard HID protocol to ensure the correct transmission of communication data between the main control module 21 and the external device 1.

[0064] In some embodiments, the communication data actually transmitted between the main control module 21 and the external device 1 is subjected to structured encapsulation to obtain structured communication data packets. Each structured communication data packet includes a frame header start bit, a data payload length, a Cyclic Redundancy Check (CRC), and communication data. The frame header start bit identifies the starting position of the structured communication data packet, the data payload length is the number of bytes of the actual transmitted communication data, the CRC is used to verify the integrity of the structured communication data packet, and the communication data includes synchronization messages, delay request messages, and delay response messages. By using a custom encapsulation and verification mechanism based on the standard HID protocol to ensure correct data transmission, the reliability and robustness of time synchronization between the main control module 21 and the external device 1 are improved.

[0065] In one embodiment, the main control module 21 communicates with the external device 1 based on the HID protocol, defining the communication data packet format through the HID report descriptor to obtain structured communication data packets. Specifically, each structured communication data packet includes fields such as the frame header start bit, data payload length, CRC, and communication data.

[0066] The start-of-frame (FOB) bit, typically a fixed two-byte hexadecimal value, serves as the start identifier of the data packet and allows the receiver to quickly locate valid data. It also provides initial transmission error detection, such as out-of-order delivery or packet loss due to noise. In one embodiment, it is defined as two consecutive static bytes in the HID report descriptor, for example, 0xFE and 0xAA. When parsing data, the receiver must first match this field; otherwise, the packet is considered invalid.

[0067] Data payload length, typically representing the actual number of bytes in the communication data, is used to define the boundaries of the communication data and prevent parsing errors, such as truncation or overflow. In one embodiment, the sender calculates the length based on the actual communication data and fills in the data payload length field defined in the HID report descriptor.

[0068] CRC, typically a checksum generated based on a preset polynomial, such as a 16-bit or 32-bit CRC, covers the start bit of the frame header, the length of the data payload, and all the communication data. CRC is used to detect data errors caused by interference during transmission, such as bit flips or data loss, thereby providing end-to-end data integrity verification. In one embodiment, it is defined as a fixed-length INPUT field in the HID report descriptor, for example, a 2-byte 16-bit CRC; the sender calculates and fills the CRC value before encapsulating the data packet; the receiver recalculates the CRC after parsing, and discards the data packet if it does not match the field value.

[0069] Communication data, typically variable-length application-layer data fields, contains specific protocol instructions or sensor data, such as time synchronization messages, delay request messages, and delay response messages.

[0070] S120. Based on the communication connection between the main control module 21 and the lidar module 22 and the camera module 23 respectively, control the main control module 21 to synchronize the lidar module 22 and the camera module 23 in time.

[0071] In one embodiment, after the clock of the main control module 21 is synchronized with the clock of the external device 1, the clock of the main control module 21 switches from slave clock mode to master clock mode. Correspondingly, at this time, the clocks of the lidar module 22 and the camera module 23 are both configured as slave clocks, thereby completing the time synchronization between the lidar module 22 and the camera module 23 and the main control module 21.

[0072] In some embodiments, the main control module 21 simultaneously performs time synchronization of the lidar module 22 and the camera module 23, or the main control module 21 sequentially performs time synchronization of the lidar module 22 and the camera module 23.

[0073] Specifically, the control module 21 sequentially performs time synchronization with the lidar module 22 and the camera module 23, including: the master clock of the master module 21 first completes time synchronization with the slave clock of the lidar module 22, and then completes time synchronization between the master clock of the master module 21 and the slave clock of the camera module 23; or, the master clock of the master module 21 first completes time synchronization with the slave clock of the camera module 23, and then completes time synchronization with the slave clock of the lidar module 22.

[0074] In one embodiment, the main control module 21 and the lidar module 22 synchronize their time using a Precision Time Protocol (PTP). The main control module 21 sends time synchronization pulses to the camera module 23 via a hardware interface for time synchronization. Specifically, the hardware interface through which the main control module 21 sends time synchronization pulses to the camera module 23 includes a General Purpose Input / Output (GPIO) interface and an Inter-Integrated Circuit (IPC) bus. I2C It includes dedicated synchronization signal interfaces, etc. Time synchronization via existing hardware interfaces and communication protocols allows for multi-module time synchronization without adding extra high-precision clock hardware or complex wiring, reducing the complexity of multi-module time synchronization and lowering hardware costs. Furthermore, the precise time protocol and synchronization pulses have a certain degree of anti-interference capability, ensuring the stability and reliability of the main control module 21 when synchronizing time with the lidar module 22 and the camera module 23, respectively.

[0075] The time synchronization method in the above embodiment first calibrates the time of the main control module 21 using the clock of the external device 1 as a reference source, and then calibrates the lidar module 22 and the camera module 23 using the time of the main control module 21 as a reference. This staged calibration reduces error accumulation, enhances the stability of the time of the multi-sensor embedded system 2, and also improves the real-time performance and reliability of the overall system.

[0076] like Figure 5As shown in the figure, this application provides a schematic diagram of a multi-sensor embedded system structure. In some embodiments, the multi-sensor embedded system 2 includes a main control module 21, a lidar module 22, a camera module 23, and a central processing unit 24; based on the communication connection between the main control module 21 and the external device 1, the central processing unit 24 is used to control the main control module 21 to perform time synchronization with the external device 1, wherein the main control module 21 and the external device 1 are connected via a universal serial bus interface; based on the communication connections between the main control module 21 and the lidar module 22 and the camera module 23 respectively, the central processing unit 24 is used to control the main control module 21 to perform time synchronization between the lidar module 22 and the camera module 23.

[0077] In one embodiment, this application provides a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor performs some or all of the steps S110 to S120 in the embodiments of this application.

[0078] In the description of this application, it should be understood that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. The terms “and / or” and “and / or” as used herein describe the relationship between related objects, indicating that three relationships may exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship. The singular forms “a” and “an” are also intended to include the plural forms unless the context clearly indicates otherwise. When the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof, i.e., including any and all combinations of one or more of the associated listed items. Ordinal numbers such as “first” and “second” referenced in the embodiments of this application are merely identifiers and do not imply any particular order or relative importance.

[0079] In this application, unless otherwise expressly specified and limited, "above" or "below" a second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of a second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" a second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances. The phrase "one or more embodiments" as used herein does not refer to the same embodiment, but rather to any suitable combination of specific features, structures, or characteristics. The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A time synchronization method applied to a multi-sensor embedded system, the multi-sensor embedded system comprising a main control module, a lidar module, and a camera module, characterized in that, The method includes: Based on the communication connection between the main control module and the external device, the main control module and the external device are controlled to complete time synchronization, wherein the main control module and the external device communicate through a universal serial bus interface; Based on the communication connections between the main control module and the lidar module and the camera module respectively, the main control module controls the lidar module to synchronize the time of the camera module and the camera module.

2. The method according to claim 1, characterized in that, Based on the communication connection between the main control module and the external device, control the main control module to complete time synchronization with the external device, including: The current local time of the main control module is determined based on the local time before the main control module performs time synchronization, the clock offset between the main control module and the external device, the clock drift rate between the main control module and the external device, and the time interval between the last time the main control module and the external device completed time synchronization to the current time.

3. The method according to claim 1, characterized in that, Based on the communication connection between the main control module and the external device, control the main control module to complete time synchronization with the external device, including: Time synchronization is achieved by configuring the clock of the master control module as the slave clock and the clock of the external device as the master clock. Based on the communication connections between the main control module and the lidar module and the camera module respectively, the main control module is controlled to perform time synchronization between the lidar module and the camera module, including: Time synchronization is achieved by configuring the clock of the main control module as the master clock and the clocks of the lidar module and the camera module as slave clocks.

4. The method according to claim 1, characterized in that, Based on the communication connection between the main control module and the external device, control the main control module to complete time synchronization with the external device, including: The main control module receives synchronization messages sent by the external device through a transmission channel and records the second local time of the main control module receiving the synchronization messages. The physical carrier of the transmission channel is the universal serial bus interface, and the synchronization message includes the first local time of the external device sending the synchronization message. The main control module sends a delay request message through the transmission channel and records the third local time when the main control module sends the delay request message; The main control module receives a delay response message sent by the external device, wherein the delay response message includes the fourth local time of the external device when it receives the delay request message; Calculate the path delay and clock offset between the main control module and the external device based on the first local time, the second local time, the third local time, and the fourth local time. The main control module adjusts its current local time according to the clock offset.

5. The method according to claim 4, characterized in that, The transmission channel between the main control module and the external device is a communication data transmission method that uses interrupt transmission.

6. The method according to claim 5, characterized in that, Based on the communication connection between the main control module and the external device, controlling the main control module to synchronize time with the external device also includes: The actual communication data transmitted between the main control module and the external device is encapsulated in a structured manner to obtain a structured communication data packet. The structured communication data packet includes a frame header start bit, data payload length, cyclic redundancy check, and communication data. The frame header start bit is used to identify the start position of the structured communication data packet, the data payload length is the number of bytes of the actual transmitted communication data, the cyclic redundancy check is used to verify the integrity of the structured communication data packet, and the communication data includes the synchronization message, the delay request message, and the delay response message.

7. The method according to claim 1, characterized in that, Based on the communication connections between the main control module and the lidar module and the camera module respectively, the main control module is controlled to perform time synchronization between the lidar module and the camera module, including: The main control module can simultaneously synchronize the time of the lidar module and the camera module, or it can sequentially synchronize the time of the lidar module and the camera module.

8. The method according to claim 1, characterized in that, Based on the communication connections between the main control module and the lidar module and the camera module respectively, the main control module is controlled to perform time synchronization between the lidar module and the camera module, including: The main control module and the lidar module synchronize their time using a precise time protocol. The main control module sends time synchronization pulses to the camera module through a hardware interface for time synchronization.

9. A multi-sensor embedded system, characterized in that, The multi-sensor embedded system includes a main control module, a lidar module, a camera module, and a central processing unit; Based on the communication connection between the main control module and the external device, the central processing unit is used to control the main control module and the external device to complete time synchronization, wherein the main control module and the external device communicate through a universal serial bus interface; Based on the communication connections between the main control module and the lidar module and the camera module respectively, the central processing unit is used to control the main control module to synchronize the lidar module and the camera module in time.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, is used to implement the time synchronization method as described in any one of claims 1 to 8.

Images