Time synchronization system and synchronization method of handheld multi-sensor fusion scanning platform

Abstract

The invention discloses a time synchronization system and synchronization method of a handheld multi-sensor fusion scanning platform. The system comprises an upper computer and a microcontroller configured as a system main clock source. The microcontroller is used for generating a serial time message which at least comprises a message synchronization control pulse signal, a hardware trigger pulse signal and defines a main time system; the upper computer runs a time synchronization device comprising an original data receiving module, a timestamp synchronization calibration module and a data storage and visualization module, and adopts different time synchronization strategies for different sensors, including external direct time service, time system offset calibration and trigger acquisition and low-delay neighbor matching strategies. According to the invention, the microcontroller is used as a synchronization core, and sub-millisecond-level hardware time synchronization precision is realized without any expensive external synchronization equipment; the time synchronization system and method provided by the invention have high flexibility and robustness, and can ensure high efficiency and low delay of data processing.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional space perception and data processing, and in particular to a time synchronization system and synchronization method for a handheld multi-sensor fusion scanning platform. Background Art

[0002] With the rapid development of robotics, autonomous driving, virtual reality (VR), and augmented reality (AR), the market demand for high-precision, high-density 3D environmental perception is growing. In this context, handheld 3D scanners, due to their unparalleled portability, operational flexibility, and relatively low cost, have found widespread application in indoor digital modeling, cultural heritage preservation, industrial reverse engineering, and quality inspection.

[0003] To obtain rich, multi-dimensional environmental information, existing handheld scanners typically integrate multiple sensors. Among these, LiDAR provides precise depth and 3D geometry; industrial cameras capture high-resolution texture and color information; and inertial measurement units (IMUs) provide real-time measurement of the device's posture and motion. Efficiently and accurately fusing this diverse data source and characteristics is the fundamental prerequisite and core challenge for achieving high-precision 3D reconstruction algorithms (such as Simultaneous Localization and Mapping (SLAM)).

[0004] In multi-sensor data fusion, precise time synchronization is the cornerstone of ensuring fusion quality. Even the slightest deviation or jitter in the timestamps of data from different sensors can lead to a series of serious problems, such as misalignment between the texture mapping of the 3D point cloud and the 2D image, inconsistencies in the construction of motion trajectories and environmental maps, and failure to capture dynamic targets. These problems ultimately lead to a significant reduction in the accuracy of the reconstructed 3D model, or even complete failure.

[0005] Currently, common time synchronization solutions in the industry can be divided into two categories:

[0006] Software synchronization: After data arrives at a host computer (such as a personal computer), the operating system assigns a timestamp based on the time the data packet is received. The main drawback of this solution is that uncontrollable delays in data transmission links (such as USB and Ethernet) and the non-deterministic jitter of the operating system's task scheduling introduce significant time errors, typically on the order of milliseconds (ms), making it difficult to meet the requirements of high-precision fusion applications.

[0007] Hardware synchronization: This solution uses a unified external synchronization signal (such as the pulse-per-second signal (PPS) output by a GPS module or a dedicated synchronization controller) to strictly trigger the acquisition actions of all sensors. While this solution can achieve high synchronization accuracy, its drawbacks are also significant. The system often relies on expensive external synchronization equipment, significantly increasing overall costs. Alternatively, the system hardware structure becomes complex, increasing the size, power consumption, and design complexity of the scanning platform. This poses an insurmountable obstacle for lightweight, low-cost handheld devices.

[0008] In addition, there is currently a lack of a unified, flexible and efficient synchronization solution for heterogeneous sensors with different interfaces and synchronization mechanisms; and existing solutions are often deeply bound to specific software platforms (such as the Robot Operating System ROS under the Linux operating system), limiting the system's portability and application scope. Summary of the Invention

[0009] In response to the shortcomings of the existing technology, the present invention provides a time synchronization system and synchronization method for a handheld multi-sensor fusion scanning platform. The system hardware includes various sensors, a microcontroller (MCU) as the system's main clock source, and a host computer for running the time synchronization device.

[0010] To achieve precise system-level synchronization, the present invention provides a message synchronization control pulse signal and a serial time message to sensors that support external timing functions, such as lidar, so that they can use this information to directly calibrate their internal clocks, thereby outputting data whose timestamps are aligned with the master time system. For sensors that do not support external timing but can support pulse event monitoring, such as inertial measurement units, the present invention adopts a time system offset calibration strategy. Specifically, by providing a message synchronization control pulse signal to this type of sensor, the sensor generates an event data packet with its own internal timestamp when detecting the pulse. The time synchronization device on the host computer continuously compares the event timestamp with the master time corresponding to the pulse, dynamically calculates and maintains the linear mapping relationship and time offset between the sensor's internal time domain and the master time system domain, and uses this offset to correct the timestamps of all sensor measurement data. In addition, for sensors that only support external hardware triggering, such as industrial cameras, the present invention adopts a trigger acquisition and low-latency neighbor matching strategy. The hardware pulse signal is used to strictly control the sensor's acquisition interval, so that the host computer's time synchronization device can use the nearest neighbor principle to reliably match the received sensor data packet with the hardware trigger pulse that is closest in time and has a known master time, thereby completing the timestamp calibration.

[0011] The specific technical solutions of the present invention are as follows:

[0012] A time synchronization system for a handheld multi-sensor fusion scanning platform, the system comprising a host computer and a microcontroller configured as a system master clock source;

[0013] The microcontroller and the host computer are both connected to the multiple sensors on the handheld multi-sensor fusion scanning platform;

[0014] The microcontroller is used to generate a composite synchronization signal with strict phase locking and send it to the host computer and various sensors; the composite synchronization signal includes at least a message synchronization control pulse signal, a hardware trigger pulse signal and a serial time message that defines the master time system, namely a pseudo GPRMC message;

[0015] A time synchronization device is run on the host computer, and the time synchronization device includes a raw data receiving module, a timestamp synchronization calibration module, and a data storage and visualization module;

[0016] The raw data receiving module asynchronously receives raw data from each sensor and microcontroller of the handheld multi-sensor fusion scanning platform by forming a thread group through multi-threaded parallel processing. After adding the local system timestamp of the host computer to the raw data, the raw data and the local system timestamp of the host computer are assembled into a custom structure and pushed into the input buffer corresponding to each sensor and microcontroller.

[0017] The timestamp synchronization and calibration module operates in a producer-consumer mode, monitors multiple input buffers simultaneously, extracts a pseudo GPRMC message from the input buffer as a reference frame, uses the local system timestamp of the corresponding host computer encapsulated in the same structure, searches for data packets in the other three buffers, performs nearest neighbor matching operations, and executes a heterogeneous composite time synchronization strategy for different sensors, assembles a synchronized data frame containing multiple heterogeneous sensor data, and pushes it to the alignment buffer;

[0018] The data storage and visualization module is used to realize continuous data storage, interruption and recovery at any time, and real-time visualization of the acquisition process.

[0019] Furthermore, the microcontroller includes an internal timer supporting interrupt processing and callback functions, as well as a pulse sending module and a message sending module.

[0020] Furthermore, the input buffer and the alignment buffer are both lock-free circular queue data structures implemented based on atomic variables.

[0021] Furthermore, the sensors on the handheld multi-sensor fusion scanning platform include the following three categories:

[0022] Sensors that support external timing using pseudo-GPRMC messages;

[0023] Sensors that support pulse event monitoring;

[0024] Low-latency sensors that do not support external timing of pseudo-GPRMC messages and support pulse event monitoring, but support hardware external triggering; the low latency specifically means that the transmission delay of the sensor is less than the period of the hardware trigger pulse signal.

[0025] Furthermore, the sensors on the handheld multi-sensor fusion scanning platform include a laser radar, an inertial measurement unit and an industrial camera.

[0026] A time synchronization method for a handheld multi-sensor fusion scanning platform is implemented based on a time synchronization system of the handheld multi-sensor fusion scanning platform. The method comprises the following steps:

[0027] Step 1: The microcontroller, serving as the master clock source, generates and distributes a set of composite synchronization signals to the host computer and each sensor. The composite synchronization signal includes at least a message synchronization control pulse signal, a hardware trigger pulse signal, and a serial time message that defines the master time system, i.e., a pseudo GPRMC message.

[0028] Step 2: For sensors that support external timing of pseudo GPRMC messages, the sensor simultaneously receives the message synchronization control pulse signal and the pseudo GPRMC message, performs timing, and directly aligns the timestamp of its output data with the main time system and sends it to the host computer;

[0029] For sensors that support pulse event monitoring, the sensor receives the message synchronization control pulse signal and sends an event data packet with an internal timestamp to the host computer;

[0030] For low-latency sensors that do not support external timing of pseudo-GPRMC messages and support pulse event monitoring but support hardware external triggering, the sensor receives the hardware trigger pulse signal to trigger its data collection and sends the obtained data packet to the host computer;

[0031] Step 3: The host computer asynchronously receives raw data from each sensor and the microcontroller, adds the host computer's local system timestamp to the raw data, assembles the raw data and the host computer's local system timestamp into a custom structure, and pushes it into the corresponding input buffers of each sensor and microcontroller;

[0032] Step 4: The host computer works in producer-consumer mode and monitors multiple input buffers at the same time. It takes out the pseudo GPRMC message from the input buffer as the reference frame, uses the local system timestamp of the corresponding host computer encapsulated in the same structure, searches for data packets in the other three input buffers, performs nearest neighbor matching operations, and executes heterogeneous composite time synchronization strategies for different sensors. It assembles a synchronized data frame containing multiple heterogeneous sensor data and pushes it to the alignment buffer for data storage and visualization.

[0033] Furthermore, for sensors that support external timing via pseudo GPRMC messages, the time synchronization strategy adopts external direct timing, i.e., simultaneously receiving the synchronization control pulse signal and pseudo GPRMC messages to calibrate their internal clocks to the master time system;

[0034] For sensors that support pulse event monitoring, the time synchronization strategy is: the host computer continuously compares the internal timestamp T of the event data packet. imu The main timestamp T in the pseudo GPRMC message mcu ,dynamically calculate and maintain the linear mapping relationship and time offset Δt between the internal time domain of the sensor and the main time system domain, and use this offset to correct the timestamps of all sensor measurement data;

[0035] For low-latency sensors that do not support external timing via pseudo-GPRMC messages and support pulse event monitoring but support hardware external triggering, the time synchronization strategy is that the host computer uses the nearest neighbor matching method to associate the collected data with the master time corresponding to the pulse.

[0036] Furthermore, the host computer uses a nearest neighbor matching method to associate the collected data with the main time corresponding to the pulse, specifically:

[0037] When the host computer receives a pseudo GPRMC message, it corrects the timestamp of the first data packet that arrives thereafter to T, and sets the timestamps of the subsequent N-1 data packets to T+1 / N, T+2 / N, ..., T+(N-1) / N seconds in sequence. This process is repeated after the next pseudo GPRMC message arrives, thereby achieving accurate timestamp calibration of the sensor; where T represents an integer second, and N represents the number of cycles of the hardware trigger pulse per second.

[0038] Furthermore, the time synchronization strategy adopts external direct timing, specifically:

[0039] When the rising edge of the pulse of the message synchronization control pulse signal is detected, the sensor records a precise moment and waits for the first pseudo GPRMC message to arrive. Once the pseudo GPRMC message is received, the absolute time parsed from the pseudo GPRMC message is used as the previously recorded moment for timing.

[0040] A computer-readable storage medium stores a program, which, when executed by a processor, implements a time synchronization method for a handheld multi-sensor fusion scanning platform.

[0041] The beneficial effects of the present invention are as follows:

[0042] 1. The time synchronization system for the handheld multi-sensor fusion scanning platform of the present invention utilizes a microcontroller as the synchronization core of the system. Without the need for any expensive external synchronization equipment, it achieves sub-millisecond hardware time synchronization accuracy, greatly lowering the entry threshold for high-performance handheld scanning platforms.

[0043] 2. The time synchronization system and method of the present invention are highly flexible and robust. It does not require all sensors on the scanning platform to have a unified high-level synchronization interface. The method of the present invention is applicable to sensors that support direct timing, event triggering only, or external triggering only.

[0044] 3. The time synchronization method of the present invention ensures high efficiency and low latency in the data processing process, and can provide high-quality, frame-by-frame aligned multimodal data for applications such as high-precision 3D reconstruction, robot SLAM, and autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 FIG. 1 is a schematic diagram of a time synchronization system for a handheld multi-sensor fusion scanning platform according to an embodiment of the present invention.

[0046] Figure 2 Schematic diagram of a host computer time synchronization device according to one embodiment of the present invention.

[0047] Figure 3 FIG. 1 is a schematic diagram of a hardware platform according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0049] On the one hand, an embodiment of the present invention provides a time synchronization system for a handheld multi-sensor fusion scanning platform, such as Figure 1As shown, it includes a host computer and a microcontroller configured as the system's main clock source.

[0050] 1. Microcontroller

[0051] The microcontroller is used to generate a composite synchronization signal with strict phase locking, which includes at least a message synchronization control pulse signal, a hardware trigger pulse signal and a serial time message defining a master time system (ie, a pseudo GPRMC message).

[0052] The microcontroller includes an internal timer supporting interrupt processing and callback functions, a pulse sending module and a message sending module, and at least three general-purpose input and output (GPIO) pins and at least one set of universal synchronous asynchronous receiver and transmitter (UART) pins.

[0053] The pulse transmission module accurately generates a periodic interrupt with a period equal to the hardware trigger period by setting the values ​​of the internal timer's prescaler and auto-reload register (ARR). A GPIO pin then outputs a level flip, forming a stable hardware trigger pulse signal with a period equal to the hardware trigger period. This signal is then sent to the external trigger pin of a low-latency sensor that supports hardware external triggering. Simultaneously, another set of GPIO pins is configured in the same manner to output a message synchronization control pulse signal with a period equal to the pseudo-GPRMC message transmission period. This signal is then sent to the synchronization control signal pins of sensors that support pseudo-GPRMC message external timing and sensors that support pseudo-GPRMC message external timing.

[0054] The message sending module is used to read the value of a 64-bit high-precision hardware counter inside it as the reference of the microcontroller system time, and format it into a serial message in the NMEA GPRMC format, that is, a pseudo-GPRMC message, and send it to the host computer through the UART pin. At the same time, it is sent to the sensor that supports external timing of pseudo-GPRMC messages through the third GPIO pin.

[0055] 2. Host Computer

[0056] like Figure 2 As shown, a time synchronization device is running on the host computer, and the time synchronization device includes a raw data receiving module, a timestamp synchronization calibration module, and a data storage and visualization module;

[0057] The raw data receiving module asynchronously receives raw data from each sensor and microcontroller of the handheld multi-sensor fusion scanning platform by forming a thread group through multi-threaded parallel processing. After adding the local system timestamp of the host computer to the raw data, the raw data and the local system timestamp of the host computer are assembled into a custom structure and pushed into the corresponding input buffer of each sensor and microcontroller for use by the timestamp synchronization calibration module.

[0058] The timestamp synchronization and calibration module works in producer-consumer mode, monitors multiple input buffers simultaneously, takes the pseudo GPRMC message from the input buffer as the reference frame, uses the local system timestamp of the corresponding host computer encapsulated in the same structure, searches for data packets in the other three input buffers, performs nearest neighbor matching operations, and executes heterogeneous composite time synchronization strategies for different sensors. It assembles a synchronized data frame containing multiple heterogeneous sensor data and pushes it to the alignment buffer for use by the data storage and visualization module.

[0059] The input buffer and the alignment buffer are both lock-free circular queue data structures implemented based on atomic variables. This data structure allows multiple producers and consumers to exchange data efficiently without generating data contention.

[0060] The data storage and visualization module supports continuous data storage and can be interrupted and resumed at any time. It supports real-time visualization of the acquisition process. Data storage is implemented through a console switch, which the user can trigger to continuously save the subsequent data stream to the hard disk at the press of a button. Storage can be interrupted and resumed at any time.

[0061] The time synchronization system and synchronization method of the handheld multi-sensor fusion scanning platform of the present invention can be applied to scanning platforms integrated with various types of sensors, including the following three categories:

[0062] Type 1: Sensors that support external timing using pseudo-GPRMC messages, such as lidar.

[0063] Type 2: Sensors that support pulse event monitoring, such as inertial measurement units;

[0064] Type 3: Low-latency sensors that do not belong to the previous two categories but support hardware external triggering. "Low latency" here means that the sensor's transmission delay is less than the hardware trigger pulse signal period, such as industrial cameras.

[0065] For sensors that support external timing via pseudo-GPRMC messages, the time synchronization adopts an external direct timing strategy, which is configured to simultaneously receive message synchronization control pulse signals and pseudo-GPRMC messages to calibrate its internal clock to the master time system, thereby outputting data whose timestamps are aligned with the master time system to the host computer. Specifically:

[0066] When the rising edge of the pulse of the message synchronization control pulse signal is detected, the sensor records a precise moment and waits for the first pseudo GPRMC message to arrive. Once the pseudo GPRMC message is received, the absolute time parsed from the pseudo GPRMC message is used as the previously recorded moment for timing.

[0067] In this way, the sensor's internal clock is fully calibrated to the microcontroller system time, and all data packets output by the sensor have precisely aligned timestamps.

[0068] For sensors that support pulse event monitoring, a time system offset calibration strategy is used for time synchronization. Specifically, the sensor receives the message synchronization control pulse signal and generates an event data packet containing its own internal timestamp upon receiving the signal. This data packet is then sent to the host computer for time calibration.

[0069] For low-latency sensors that do not belong to the previous two categories but support hardware external triggering, they receive the hardware trigger pulse signal sent by the microcontroller, trigger data acquisition, and send the collected data packets to the host computer, which performs time calibration.

[0070] On the other hand, another embodiment of the present invention provides a time synchronization method for a handheld multi-sensor fusion scanning platform, which is implemented based on the time system of the handheld multi-sensor fusion scanning platform described above and includes the following steps:

[0071] Step 1: The microcontroller, serving as the master clock source, generates and distributes a set of composite synchronization signals to the host computer and each sensor. The composite synchronization signal includes at least a message synchronization control pulse signal, a hardware trigger pulse signal, and a serial time message that defines the master time system, namely, a pseudo GPRMC message.

[0072] Step 2: For sensors that support external timing using pseudo-GPRMC messages, the sensor simultaneously receives the message synchronization control pulse signal and the pseudo-GPRMC message, performs timing, and aligns the timestamp of its output data directly with the main time system and sends it to the host computer;

[0073] For sensors that support pulse event monitoring, the sensor receives the message synchronization control pulse signal and sends the event data packet with internal timestamp to the host computer;

[0074] For low-latency sensors that do not support external timing of pseudo-GPRMC messages and support pulse event monitoring but support hardware external triggering, the sensor receives the hardware trigger pulse signal to trigger its data acquisition and sends the obtained data packet to the host computer;

[0075] Step 3: The host computer asynchronously receives raw data from each sensor and microcontroller, appends the host computer's local system timestamp to the raw data, assembles the raw data and the host computer's local system timestamp into a custom structure, and pushes it into the corresponding input buffer of each sensor and microcontroller;

[0076] Step 4: The host computer works in producer-consumer mode, monitoring multiple input buffers simultaneously. It extracts the pseudo GPRMC message from the input buffer as the reference frame, uses the local system timestamp of the corresponding host computer encapsulated in the same structure, searches for data packets in the other three input buffers, performs nearest neighbor matching, and executes a heterogeneous composite time synchronization strategy for different sensors. It assembles a synchronized data frame containing multiple heterogeneous sensor data and pushes it to the alignment buffer for data storage and visualization.

[0077] Among them, for sensors that support pulse event monitoring, the time synchronization strategy of the host computer is: the host computer continuously compares the internal timestamp T of the event data packet imu The main timestamp T in the pseudo GPRMC message mcu , dynamically calculate and maintain the linear mapping relationship and time offset Δt between the internal time domain of the sensor and the main time system domain, and use this offset to correct the timestamps of all sensor measurement data.

[0078] The time offset Δt is the time when the host computer receives the pulse event data packet of the sensor and a pseudo GPRMC message at the same time, and the main timestamp T in the pseudo GPRMC message is mcu The internal time stamp T of the inertial measurement unit imu The difference between

[0079] ∆t= T mcu -T imu

[0080] T mcu =f(T imu )

[0081] This offset Δt is continuously calculated and updated during device operation, and is smoothed by a sliding average or Kalman filter to compensate for the possible slight frequency drift between the two crystal oscillators. And f represents the main timestamp T in the pseudo GPRMC message. mcu The internal time stamp T of the sensor imu The host computer is the internal timestamp T of the event data packet. imu After adding a time offset Δt, it serves as the final synchronization timestamp of the event data packet;

[0082] For low-latency sensors that do not support external timing via pseudo-GPRMC messages and do support pulse event monitoring but do support external hardware triggering, the host computer uses a nearest neighbor matching method to associate the collected data with the master time corresponding to the pulse. Specifically, when the host computer receives a pseudo-GPRMC message (representing an integer second time T), it corrects the timestamp of the first subsequent data packet (reliably assumed to correspond to the pulse at time T) to T. It then sets the timestamps of the subsequent N-1 data packets to T+1 / N, T+2 / N, ..., T+(N-1) / N seconds, where N represents the number of hardware trigger pulse cycles per second. This process is repeated after the next pseudo-GPRMC message arrives, achieving accurate timestamp calibration for the sensor.

[0083] A specific embodiment is given below to further illustrate the present invention.

[0084] like Figure 3 As shown, a schematic diagram of the handheld multi-sensor fusion scanning platform of this embodiment is shown. The platform hardware includes a handheld frame, a laser radar integrated on the frame, an industrial camera and an inertial measurement unit, as well as a microcontroller and a host computer.

[0085] In this embodiment, the microcontroller (MCU) is STM32F103C8T6, which has a main frequency of 72MHz. Its internal timer (TIM) can provide nanosecond-level accuracy and has a rich set of peripheral interfaces such as USART and SPI. It can fully meet the requirements of the present invention for the main clock source at an extremely low cost and is connected to the host computer via a USB cable.

[0086] LiDAR: Use the Ouster OS0-128. This radar supports external synchronization input, can parse serial messages in a GPRMC-like format, and receive PPS (Pulse Per Second) signals. Connect to the host computer using the manufacturer's interface box and an Ethernet cable.

[0087] Industrial Camera: The Hikvision MV-CS050-60UC is a 4-megapixel industrial camera that supports multiple hardware trigger modes. It connects to a host computer via a USB 3.0 interface. By registering the "trigger signal callback" function in the development software, we measure the time it takes from the signal reaching the camera to the host computer receiving the complete image data. The camera's end-to-end latency is typically stable between a few milliseconds and tens of milliseconds, significantly less than the 100ms period corresponding to a 10Hz acquisition frequency, thus meeting low-latency requirements.

[0088] Inertial Measurement Unit (IMU): Use the Xsens MTi-630. This IMU outputs data via a serial port, and its supporting software or data protocol supports parsing external events (such as pulses). Connect to the host computer via a USB cable.

[0089] The microcontroller connects to each sensor via DuPont cables. The GPIO interface corresponding to the message synchronization control pulse signal is connected to the corresponding GPIO input pins of the lidar and inertial measurement unit via a split-to-two DuPont cable. The GPIO interface corresponding to the hardware trigger pulse signal on the microcontroller is connected to the trigger signal pin of the industrial camera via a DuPont cable. The GPIO pin on the microcontroller used to send pseudo GPRMC messages is connected to the GPRMC input pin of the lidar via a DuPont cable. The GND pins corresponding to all three sensors must be connected to the GND pins of the microsensors via DuPont cables. Furthermore, a dedicated UART-to-USB adapter is required to connect the pseudo GPRMC messages sent by the microcontroller via a set of UART pins to the USB port of the host computer. The lidar transmits data directly to the host computer via an Ethernet cable, while the industrial camera and inertial measurement unit transmit data via USB.

[0090] Furthermore, in this embodiment, to prevent obstruction of the LiDAR's scanning angle, the industrial camera and inertial measurement unit are mounted on the underside of the handheld scanning platform, while the LiDAR is mounted separately on the side of the platform and elevated to a certain degree. All sensors are hard-connected to the platform via bolts and nuts to ensure scanning stability.

[0091] Physical platform: 3D structural modeling was completed using SolidWorks software, and various assembly parts were produced using PLA material using an Allct330Pro printer. Ultimately, all the above hardware and power supply modules were integrated into a handheld frame with a total weight of approximately 2kg.

[0092] Software Development: Use STM32CubeIDE software to develop MCU firmware on Windows. Use C++ to develop the entire software on Ubuntu 22.04.

[0093] External parameter calibration: Use PVC to make a calibration plate with a thickness of about 2 cm, a width of 140 cm, and a height of 100 cm. The size should be as large as possible to reduce the deviation of edge detection and visual inspection.

[0094] The visualization function of the host computer in this embodiment uses the Pangolin graphics library to display the currently acquired camera image, the three-dimensional laser point cloud projected onto the image, and the line graph of the angular velocity and acceleration measured by the inertial measurement unit in real time in a window, providing the operator with intuitive feedback on the scanning process.

[0095] The handheld multi-sensor fusion scanning platform of this embodiment can stably collect, synchronize, save and visualize multi-sensor data at a frequency of 10 Hz.

[0096] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art will still be able to modify the technical solutions described in the foregoing examples or substitute equivalents for some of the technical features therein. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the invention shall be included within the scope of protection of the invention.

Claims

1. A time synchronization system for a handheld multi-sensor fusion scanning platform, characterized in that: The system includes a host computer and a microcontroller configured as the system's main clock source; The microcontroller and the host computer are both connected to the multiple sensors on the handheld multi-sensor fusion scanning platform; The microcontroller is used to generate a composite synchronization signal with strict phase locking and send it to the host computer and various sensors; the composite synchronization signal includes at least a message synchronization control pulse signal, a hardware trigger pulse signal and a serial time message that defines the master time system, namely a pseudo GPRMC message; A time synchronization device is run on the host computer, and the time synchronization device includes a raw data receiving module, a timestamp synchronization calibration module, and a data storage and visualization module; The raw data receiving module asynchronously receives raw data from each sensor and microcontroller of the handheld multi-sensor fusion scanning platform by forming a thread group through multi-threaded parallel processing. After adding the local system timestamp of the host computer to the raw data, the raw data and the local system timestamp of the host computer are assembled into a custom structure and pushed into the input buffer corresponding to each sensor and microcontroller. The timestamp synchronization and calibration module operates in a producer-consumer mode, monitors multiple input buffers simultaneously, extracts a pseudo GPRMC message from the input buffer as a reference frame, uses the local system timestamp of the corresponding host computer encapsulated in the same structure, searches for data packets in the other three buffers, performs nearest neighbor matching operations, and executes a heterogeneous composite time synchronization strategy for different sensors, assembles a synchronized data frame containing multiple heterogeneous sensor data, and pushes it to the alignment buffer; The data storage and visualization module is used to realize continuous data storage, interruption and recovery at any time, and real-time visualization of the acquisition process.

2. The time synchronization system of the handheld multi-sensor fusion scanning platform according to claim 1, characterized in that: The microcontroller includes an internal timer supporting interrupt processing and callback functions, as well as a pulse sending module and a message sending module.

3. The time synchronization system of the handheld multi-sensor fusion scanning platform according to claim 1, characterized in that: The input buffer and the alignment buffer are both lock-free circular queue data structures implemented based on atomic variables.

4. The time synchronization system of the handheld multi-sensor fusion scanning platform according to claim 1, characterized in that: The sensors on the handheld multi-sensor fusion scanning platform include the following three categories: Sensors that support external timing using pseudo-GPRMC messages; Sensors that support pulse event monitoring; Low-latency sensors that do not support external timing of pseudo-GPRMC messages and support pulse event monitoring, but support hardware external triggering; the low latency specifically means that the transmission delay of the sensor is less than the period of the hardware trigger pulse signal.

5. The time synchronization system of the handheld multi-sensor fusion scanning platform according to claim 1, characterized in that: The sensors on the handheld multi-sensor fusion scanning platform include laser radar, inertial measurement unit and industrial camera.

6. A time synchronization method for a handheld multi-sensor fusion scanning platform, characterized in that: The method is implemented based on the time synchronization system of the handheld multi-sensor fusion scanning platform according to claim 1, and comprises the following steps: Step 1: The microcontroller, serving as the master clock source, generates and distributes a set of composite synchronization signals to the host computer and each sensor. The composite synchronization signal includes at least a message synchronization control pulse signal, a hardware trigger pulse signal, and a serial time message that defines the master time system, i.e., a pseudo GPRMC message. Step 2: For sensors that support external timing of pseudo GPRMC messages, the sensor simultaneously receives the message synchronization control pulse signal and the pseudo GPRMC message, performs timing, and directly aligns the timestamp of its output data with the main time system and sends it to the host computer; For sensors that support pulse event monitoring, the sensor receives the message synchronization control pulse signal and sends an event data packet with an internal timestamp to the host computer; For low-latency sensors that do not support external timing of pseudo-GPRMC messages and support pulse event monitoring but support hardware external triggering, the sensor receives the hardware trigger pulse signal to trigger its data collection and sends the obtained data packet to the host computer; Step 3: The host computer asynchronously receives raw data from each sensor and the microcontroller, adds the host computer's local system timestamp to the raw data, assembles the raw data and the host computer's local system timestamp into a custom structure, and pushes it into the corresponding input buffers of each sensor and microcontroller; Step 4: The host computer works in producer-consumer mode and monitors multiple input buffers at the same time. It takes out the pseudo GPRMC message from the input buffer as the reference frame, uses the local system timestamp of the corresponding host computer encapsulated in the same structure, searches for data packets in the other three input buffers, performs nearest neighbor matching operations, and executes heterogeneous composite time synchronization strategies for different sensors. It assembles a synchronized data frame containing multiple heterogeneous sensor data and pushes it to the alignment buffer for data storage and visualization.

7. The time synchronization method of the handheld multi-sensor fusion scanning platform according to claim 6, characterized in that: For sensors that support external timing using pseudo GPRMC messages, the time synchronization strategy uses external direct timing, i.e., it receives the synchronization control pulse signal and pseudo GPRMC messages simultaneously to calibrate its internal clock to the master time system; For sensors that support pulse event monitoring, the time synchronization strategy is: the host computer continuously compares the internal timestamps of event data packets T imu The main timestamp in the pseudo GPRMC message T mcu , dynamically calculate and maintain the linear mapping relationship and time offset Δ between the sensor internal time domain and the main time system domain t , and use the offset to correct the timestamps of all sensor measurement data; For low-latency sensors that do not support external timing via pseudo-GPRMC messages and support pulse event monitoring but support hardware external triggering, the time synchronization strategy is that the host computer uses the nearest neighbor matching method to associate the collected data with the master time corresponding to the pulse.

8. The time synchronization method of the handheld multi-sensor fusion scanning platform according to claim 7, characterized in that: The host computer uses the nearest neighbor matching method to associate the collected data with the main time corresponding to the pulse, specifically: When the host computer receives a pseudo GPRMC message, it corrects the timestamp of the first data packet that arrives thereafter to T, and sets the timestamps of the subsequent N-1 data packets to T+1 / N, T+2 / N, ..., T+(N-1) / N seconds in sequence. This process is repeated after the next pseudo GPRMC message arrives, thereby achieving accurate timestamp calibration of the sensor; where T represents an integer second, and N represents the number of cycles of the hardware trigger pulse per second.

9. The time synchronization method of the handheld multi-sensor fusion scanning platform according to claim 7, characterized in that: The time synchronization strategy adopts external direct timing, specifically: When the rising edge of the pulse of the message synchronization control pulse signal is detected, the sensor records a precise moment and waits for the first pseudo GPRMC message to arrive. Once the pseudo GPRMC message is received, the absolute time parsed from the pseudo GPRMC message is used as the previously recorded moment for timing.

10. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by the processor, the time synchronization method of the handheld multi-sensor fusion scanning platform described in any one of claims 6 to 9 is implemented.

Images