Time synchronization system and synchronization method for handheld multi-sensor fusion scanning platform

Abstract

The application discloses a time synchronization system and method of a handheld multi-sensor fusion scanning platform, and the system comprises an upper computer and a microcontroller configured as a main clock source of the system; the microcontroller is used for generating at least a message synchronization control pulse signal, a hardware trigger pulse signal and a serial time message defining a main time system; the upper computer comprises a time synchronization device running a raw data receiving module, a time stamp synchronization calibration module and a data storage and visualization module, different time synchronization strategies are adopted for different sensors, including external direct time service, time system offset calibration and trigger acquisition and low-delay near-neighbor matching strategy. The application uses a microcontroller as a synchronization core, and realizes a hardware time synchronization precision of sub-millisecond level without any expensive external synchronization equipment; the time synchronization system and method have high flexibility and robustness, and can guarantee efficient and low-delay data processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional space perception and data processing, and in particular to a time synchronization system and method for a handheld multi-sensor fusion scanning platform. BACKGROUND

[0002] With the rapid development of robotics, autonomous driving, virtual reality (VR) and augmented reality (AR), the market demand for high-precision, high-density three-dimensional environment perception capability is growing. In this context, handheld three-dimensional scanners have been widely used in indoor digital modeling, cultural heritage protection, industrial reverse engineering and quality detection due to their unparalleled portability, flexible operation and relatively low cost.

[0003] To obtain rich, multi-dimensional environmental information, existing handheld scanners usually integrate multiple sensors. Among them, LiDAR is used to provide accurate depth and three-dimensional geometric structure information; industrial cameras are used to collect high-resolution texture and color information; and inertial measurement units (IMU) are used to measure the device's attitude and motion information in real time. Efficient and accurate fusion of these data with different sources and characteristics is the fundamental premise and core challenge of implementing high-precision three-dimensional reconstruction algorithms (such as Simultaneous Localization and Mapping, SLAM).

[0004] In multi-sensor data fusion, accurate time synchronization is the cornerstone of ensuring fusion quality. If there is even a small deviation or jitter in the timestamps of different sensor data, it will cause a series of serious problems, such as misalignment of three-dimensional point cloud and two-dimensional image texture mapping, inconsistency of motion trajectory and environment map construction, and failure to capture dynamic targets. These problems will ultimately lead to a significant reduction in the accuracy of the reconstructed three-dimensional model, or even complete failure.

[0005] Currently, the common time synchronization schemes in the industry can be divided into two categories:

[0006] Software synchronization scheme: This scheme assigns a timestamp to the data packet based on its reception time by the operating system after the data arrives at the host computer (such as a personal computer). The main drawback of this scheme is that the uncontrollable delay of data in the transmission link (such as USB, Ethernet) and the non-deterministic jitter of the operating system task scheduling will introduce a large time error, which is usually on the order of milliseconds (ms), making it difficult to meet the needs of high-precision fusion applications.

[0007] Hardware synchronization scheme: this scheme strictly triggers the acquisition actions of all sensors through a unified external synchronization signal (such as the second pulse signal PPS output by the GPS module, or a dedicated synchronization controller). Although such a scheme can achieve high synchronization accuracy, its disadvantages are also very obvious: the system usually relies on expensive external synchronization equipment, resulting in a significant increase in overall cost; or the system hardware structure becomes complex, increasing the volume, power consumption and design difficulty of the scanning platform, which constitutes an insurmountable obstacle for handheld devices that pursue portability and low cost.

[0008] In addition, there is currently a lack of a unified, flexible and efficient synchronization solution for heterogeneous sensors with different interfaces and synchronization mechanisms; and existing solutions are often deeply bound to specific software platforms (such as the Robot Operating System ROS under the Linux operating system), limiting the portability and application range of the system. SUMMARY

[0009] In view of the deficiencies of the prior art, the present application provides a time synchronization system and method for a handheld multi-sensor fusion scanning platform. The system includes, in hardware, various sensors, a microcontroller (MCU) as the main time clock source of the system, and a host computer for running a time synchronization device.

[0010] To achieve accurate synchronization at the system level, for sensors that support external time service, such as laser radars, the present application provides them with a message synchronization control pulse signal and a serial time message at the same time, so that they can directly calibrate their internal clocks using this information, thereby outputting data whose time stamp has been aligned with the main time system. For sensors that do not support external time service but can support pulse event listening, such as inertial measurement units, the present application adopts a time system offset calibration strategy. Specifically, by providing a message synchronization control pulse signal to this type of sensor, the sensor will generate an event data packet with its own internal time stamp when it detects the pulse. The time synchronization device on the host computer dynamically calculates and maintains the linear mapping relationship and time offset between the sensor's internal time domain and the main time system domain by continuously comparing the event time stamp with the main time corresponding to the pulse, and corrects the time stamp of all sensor measurement data using the offset. In addition, for sensors that only support external hardware triggering, such as industrial cameras, the present application adopts a trigger acquisition and low-delay nearest neighbor matching strategy. By strictly controlling the acquisition interval of the sensor using a hardware pulse signal, the time synchronization device on the host computer can reliably match the received sensor data packet with the hardware trigger pulse that is closest in time and has a known main time, thereby completing the time stamp calibration.

[0011] The specific technical solutions of the present application are as follows:

[0012] A time synchronization system of a handheld multi-sensor fusion scanning platform, the system comprising a host computer and a microcontroller configured as a system master clock source;

[0013] The microcontroller and the host computer are connected with a plurality of sensors on the handheld multi-sensor fusion scanning platform;

[0014] The microcontroller is configured to generate a phase-strictly locked complex synchronization signal and send it to the host computer and each sensor; the complex synchronization signal at least includes a message synchronization control pulse signal, a hardware trigger pulse signal, and a serial time message defining the master time system, i.e. a pseudo GPRMC message;

[0015] A time synchronization device is running on the host computer, the time synchronization device comprising an original data receiving module, a timestamp synchronization calibration module, and a data storage and visualization module;

[0016] The original data receiving module receives original data from each sensor and the microcontroller of the handheld multi-sensor fusion scanning platform asynchronously through a multi-thread parallel thread group, and assembles the original data and the local system timestamp of the host computer into a custom structure after attaching the local system timestamp of the host computer to the original data, and pushes them into the input buffer corresponding to each sensor and the microcontroller;

[0017] The timestamp synchronization calibration module works in a producer-consumer mode, while monitoring multiple input buffers, taking out the pseudo GPRMC message as a reference frame from the input buffer, using the local system timestamp of the host computer encapsulated in the same structure to find data packets in the other three buffers, performing nearest neighbor matching operation, and executing heterogeneous composite time synchronization strategy for different sensors, assembling a synchronization data frame containing data of multiple heterogeneous sensors, and pushing it to the alignment buffer;

[0018] The data storage and visualization module is configured to realize continuous data saving, interruption and recovery at any time, and real-time visualization of the acquisition process.

[0019] Further, the microcontroller comprises an internal timer supporting interrupt processing and callback functions, as well as a pulse sending module and a message sending module.

[0020] Further, the input buffer and the alignment buffer are both non-locking circular queue data structures based on atomic variables.

[0021] Further, the sensors on the handheld multi-sensor fusion scanning platform include the following three types:

[0022] Sensors supporting external timing of pseudo GPRMC messages;

[0023] a sensor supporting impulse event listening;

[0024] a low-delay sensor not supporting external time synchronization of pseudo GPRMC message and supporting impulse event listening but supporting hardware external trigger, wherein the low-delay is specifically that the transmission delay of the sensor is less than the period of the hardware trigger pulse signal.

[0025] Further, the sensors on the handheld multi-sensor fusion scanning platform include a laser radar, an inertial measurement unit and an industrial camera.

[0026] A time synchronization method of a handheld multi-sensor fusion scanning platform, which is implemented based on a time synchronization system of the handheld multi-sensor fusion scanning platform, and comprises the following steps:

[0027] Step 1: generating and distributing a set of synchronization signals to an upper computer and various sensors by a microcontroller as a master clock source, wherein the synchronization signals at least include a message synchronization control pulse signal, a hardware trigger pulse signal and a serial time message defining a master time system, i.e. a pseudo GPRMC message;

[0028] Step 2: for a sensor supporting external time synchronization of a pseudo GPRMC message, the sensor simultaneously receives the message synchronization control pulse signal and the pseudo GPRMC message, performs time synchronization, so that the time stamp of the output data is directly aligned with the master time system, and sends the data to the upper computer;

[0029] for a sensor supporting impulse event listening, the sensor receives the message synchronization control pulse signal, and sends an event data packet with an internal time stamp to the upper computer;

[0030] for a low-delay sensor not supporting external time synchronization of a pseudo GPRMC message and supporting impulse event listening but supporting hardware external trigger, the sensor receives the hardware trigger pulse signal to trigger data acquisition, and sends a data packet obtained to the upper computer;

[0031] Step 3: the upper computer asynchronously receives raw data from various sensors and the microcontroller, and assembles the raw data and a local system time stamp of the upper computer into a self-defined structure after attaching the local system time stamp of the upper computer to the raw data, and pushes the raw data and the local system time stamp of the upper computer into respective input buffer areas of the various sensors and the microcontroller;

[0032] Step four: the upper computer works in the producer-consumer mode, monitors multiple input buffers, takes the pseudo GPRMC message as the reference frame, uses the corresponding local system timestamp of the upper computer encapsulated in the same structure to search for data packets in the other three input buffers, performs nearest neighbor matching operation, and executes the heterogeneous composite time synchronization strategy for different sensors, assembles the synchronized data frame containing data of multiple heterogeneous sensors, and pushes it to the alignment buffer for data storage and visualization.

[0033] Further, for sensors supporting external timing of pseudo GPRMC messages, the time synchronization strategy adopts external direct timing, that is, simultaneously receives the synchronization control pulse signal and the pseudo GPRMC message to calibrate its internal clock to the master time system;

[0034] For sensors supporting pulse event listening, the time synchronization strategy is that the upper computer dynamically calculates and maintains the linear mapping relationship and time offset Δt between the internal time domain of the sensor and the domain of the master time system by continuously comparing the internal timestamp T imu of the event data packet with the master timestamp T mcu in the pseudo GPRMC message, and uses the offset to correct the timestamps of all sensor measurement data;

[0035] For low-delay sensors that do not support external timing of pseudo GPRMC messages and support pulse event listening but support hardware external triggering, the time synchronization strategy is that the upper computer uses the nearest neighbor matching method to associate the collected data with the master time corresponding to the pulse.

[0036] Further, the upper computer uses the nearest neighbor matching method to associate the collected data with the master time corresponding to the pulse, specifically:

[0037] When the upper computer receives a pseudo GPRMC message, the timestamp of the first data packet arriving after the pseudo GPRMC message is corrected to T, and the timestamps of the subsequent N-1 data packets are set to T+1 / N, T+2 / N,..., T+(N-1) / N seconds in turn. This process is repeated when the next pseudo GPRMC message arrives, thereby realizing accurate timestamp calibration of the sensor; where T represents an integer second, and N represents the number of cycles per second of the hardware trigger pulse.

[0038] Further, the time synchronization strategy adopts external direct timing, specifically:

[0039] When the rising edge of the pulse of the message synchronization control pulse signal is detected, the sensor records the next accurate time point, and waits for the first pseudo GPRMC message to come subsequently; once the pseudo GPRMC message is received, the absolute time parsed in the pseudo GPRMC message is used to time the previously recorded time point.

[0040] A computer readable storage medium, having stored thereon a program, which when executed by a processor, implements a time synchronization method of a handheld multi-sensor fusion scanning platform.

[0041] The beneficial effects of the present application are as follows:

[0042] 1. The time synchronization system of the handheld multi-sensor fusion scanning platform of the present application uses a microcontroller as the synchronization core of the system, and achieves sub-millisecond hardware time synchronization accuracy without any expensive external synchronization device, greatly reducing the access threshold of high-performance handheld scanning platforms.

[0043] 2. The time synchronization system and method of the present application have high flexibility and robustness, and do not require all sensors of the scanning platform to have a unified advanced synchronization interface, whether the sensors support direct time setting, only support event triggering or only support external triggering, the method of the present application can be applied.

[0044] 3. The time synchronization method of the present application ensures the efficiency and low delay of the data processing flow, and can provide high-quality, frame-aligned multi-modal data for high-precision three-dimensional reconstruction, robot SLAM, automatic driving and other applications. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The figure is a schematic diagram of the time synchronization system of the handheld multi-sensor fusion scanning platform of one embodiment of the present application.

[0046] Figure 2 The figure is a schematic diagram of the time synchronization device of the host computer of one embodiment of the present application.

[0047] Figure 3 The figure is a schematic diagram of the hardware platform of one embodiment of the present application. DETAILED DESCRIPTION

[0048] The present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings and preferred embodiments, it should be understood that the specific embodiments described herein are intended to be illustrative only and not to limit the present application.

[0049] In one aspect, an embodiment of the present application provides a time synchronization system of a handheld multi-sensor fusion scanning platform, as shown in Figure 1As shown, the system includes a host computer and a microcontroller configured as a system master clock source.

[0050] I. Microcontroller

[0051] The microcontroller is used to generate a phase-strictly locked complex synchronization signal, which includes at least a message synchronization control pulse signal, a hardware trigger pulse signal, and a serial time message (i.e., a pseudo-GPRMC message) that defines the master time system.

[0052] The microcontroller includes an internal timer that supports interrupt processing and callback functions, as well as a pulse sending module and a message sending module, at least three general-purpose input / output (GPIO) pins, and at least one set of universal asynchronous receiver / transmitter (UART) pins.

[0053] The pulse sending module accurately generates a periodic interrupt with a period equal to the hardware trigger period by setting the values of the prescaler and automatic reload register (ARR) of the internal timer, and outputs a level flip from a GPIO pin to form a stable hardware trigger pulse signal with a period equal to the hardware trigger period, which is sent to the external trigger pin of a low-delay sensor that supports hardware external triggering. Meanwhile, another set of GPIO pins is configured in the same way to output a message synchronization control pulse signal with a period equal to the pseudo-GPRMC message sending period, which is sent to the synchronization control signal pin of a sensor that supports external timing of the pseudo-GPRMC message.

[0054] The message sending module reads the value of a 64-bit high-precision hardware counter in its internal memory as the reference of the system time of the microcontroller, formats it into a serial message in NMEA GPRMC format, i.e., a pseudo-GPRMC message, and sends it to the host computer through the UART pin, as well as to a sensor that supports external timing of the pseudo-GPRMC message through a third GPIO pin.

[0055] II. Host Computer

[0056] As shown, a time synchronization device is running on the host computer, which includes an original data receiving module, a timestamp synchronization calibration module, and a data storage and visualization module. Figure 2

[0057] The original data receiving module receives original data from each sensor of the handheld multi-sensor fusion scanning platform and the microcontroller asynchronously through a multi-thread parallel thread group, and attaches a local system timestamp of the host computer to the original data. The original data and the local system timestamp of the host computer are assembled into a custom structure and pushed into the input buffer corresponding to each sensor and the microcontroller, for use by the timestamp synchronization calibration module.​

[0058] The time stamp synchronization calibration module works in a producer-consumer mode, monitors multiple input buffers, takes out pseudo GPRMC messages as reference frames from the input buffers, searches data packets in other three input buffers by using the local system time stamp of the corresponding host computer encapsulated in the same structure, performs nearest neighbor matching operation, and executes heterogeneous composite time synchronization strategies for different sensors, assembles into a synchronization data frame containing multiple heterogeneous sensor data, and pushes it to an alignment buffer for use by the data storage and visualization module.

[0059] The input buffer and the alignment buffer are both non-locking circular queue data structures based on atomic variables, which allows multiple producers and consumers to exchange data efficiently without data race.

[0060] The data storage and visualization module supports continuous data saving and interruption and recovery at any time. Real-time visualization of the acquisition process is supported, and the data saving function is realized through a console switch. The user can trigger continuous saving of subsequent data streams to the hard disk by pressing a key. Saving supports interruption and recovery at any time.

[0061] The time synchronization system and synchronization method of the handheld multi-sensor fusion scanning platform of the application can be applied to scanning platforms integrated with various types of sensors, including the following three types:

[0062] Type one: sensors supporting external timing of pseudo GPRMC messages, such as laser radars;

[0063] Type two: sensors supporting pulse event listening, such as inertial measurement units;

[0064] Type three: low-delay sensors that do not belong to the above two types but support hardware external triggering. Here, "low delay" means that the transmission delay of the sensor is less than the period of the hardware trigger pulse signal, such as an industrial camera.

[0065] For sensors supporting external timing of pseudo GPRMC messages, the time synchronization adopts an external direct timing strategy, which is configured to receive a message synchronization control pulse signal and a pseudo GPRMC message at the same time, to calibrate its internal clock to the master time system, and to output data with time stamp aligned to the master time system to the host computer. Specifically:

[0066] When the rising edge of the pulse of the message synchronization control pulse signal is detected, the sensor records the next accurate time point and waits for the first pseudo GPRMC message to come; once the pseudo GPRMC message is received, the absolute time parsed from the pseudo GPRMC message is used to time the previously recorded time point.

[0067] In this way, the internal clock of the sensor is completely calibrated to the microcontroller system time, and all data packets output by the sensor are self-contained with time stamps that have been accurately aligned.

[0068] For sensors supporting pulse event listening, a time system offset calibration strategy is therefore adopted for time synchronization. Specifically, the sensor receives a message synchronization control pulse signal and generates an event data packet containing its own internal time stamp when the signal is received, which is sent to the host computer for time calibration.

[0069] For low-delay sensors that do not belong to the above two categories but support hardware external triggering, a hardware trigger pulse signal is received from the microcontroller to trigger data acquisition, and the acquired data packet is sent to the host computer for time calibration.

[0070] On the other hand, another embodiment of the present application provides a time synchronization method for a handheld multi-sensor fusion scanning platform, which is implemented based on the time system of the handheld multi-sensor fusion scanning platform and includes the following steps:

[0071] Step 1: A microcontroller serving as a master clock source generates and distributes a set of complex synchronization signals to the host computer and each sensor, the complex synchronization signals including at least a message synchronization control pulse signal, a hardware trigger pulse signal, and a serial time message defining the master time system, i.e., a pseudo GPRMC message;

[0072] Step 2: For sensors supporting external time synchronization of the pseudo GPRMC message, the sensor simultaneously receives the message synchronization control pulse signal and the pseudo GPRMC message, performs time synchronization, and sends the time stamp of the output data directly to the host computer, which is aligned with the master time system;

[0073] For sensors supporting pulse event listening, the sensor receives the message synchronization control pulse signal and sends an event data packet with an internal time stamp to the host computer;

[0074] For low-delay sensors that do not support external time synchronization of the pseudo GPRMC message and support pulse event listening but support hardware external triggering, the sensor receives a hardware trigger pulse signal to trigger data acquisition, and sends the obtained data packet to the host computer;

[0075] Step 3: The host computer asynchronously receives raw data from each sensor and the microcontroller, and attaches a local system time stamp of the host computer to the raw data, assembles the raw data and the local system time stamp of the host computer into a custom structure, and pushes it into the input buffer corresponding to each sensor and the microcontroller;

[0076] Step four: the host computer works in the producer-consumer mode, monitors multiple input buffers, takes the pseudo GPRMC message as the reference frame, uses the corresponding local system timestamp of the host computer encapsulated in the same structure, searches for data packets in the other three input buffers, performs nearest neighbor matching operation, and executes the heterogeneous composite time synchronization strategy for different sensors, assembles the synchronized data frame containing multiple heterogeneous sensor data, and pushes it to the alignment buffer for data storage and visualization;

[0077] Among them, for the sensor supporting pulse event listening, the time synchronization strategy of the host computer is: the host computer continuously compares the internal timestamp T imu of the event data packet with the master timestamp T mcu in the pseudo GPRMC message, dynamically calculates and maintains the linear mapping relationship and time offset Δt between the sensor internal time domain and the master time system domain, and corrects the time stamp of all sensor measurement data using the offset.

[0078] Among them, the time offset Δt is the difference between the master timestamp T mcu in the pseudo GPRMC message and the internal timestamp T imu of the inertial measurement unit when the host computer receives the pulse event data packet and a pseudo GPRMC message at the same time, that is

[0079] Δt = T mcu -T imu

[0080] T mcu =f(T imu )

[0081] This offset Δt is continuously calculated and updated during the operation of the device, and is smoothed in the form of a sliding average or Kalman filter, so as to compensate for the possible small frequency drift between the two crystal oscillators. And f represents the accurate mapping relationship between the master timestamp T mcu in the pseudo GPRMC message and the internal timestamp T imu of the sensor. The host computer adds a time offset Δt to the internal timestamp T imu of the event data packet as the final synchronization timestamp of the event data packet;

[0082] For low-delay sensors that do not support external timing of pseudo-GPRMC messages and support pulse event listening but support hardware external triggering, the host computer associates the collected data with the main time corresponding to the pulse using a nearest neighbor matching method. Specifically, after the host computer receives a pseudo-GPRMC message (representing an integer second time T), the host computer corrects the timestamp of the first data packet that arrives thereafter (which is reliably considered to correspond to the pulse at time T) to T, and sets the timestamps of the subsequent N-1 data packets to T+1 / N, T+2 / N,..., T+(N-1) / N seconds in turn, where N represents the number of cycles per second of the hardware trigger pulse. This process is repeated after the next pseudo-GPRMC message arrives, thereby achieving accurate timestamp calibration of the sensor.

[0083] A specific embodiment is given below to further illustrate the present application.

[0084] As shown in Figure 3 , the schematic diagram of the handheld multi-sensor fusion scanning platform of the present embodiment, which includes a handheld frame, a laser radar, an industrial camera and an inertial measurement unit integrated on the frame, and a microcontroller and a host computer on the hardware.

[0085] In the present embodiment, the microcontroller (MCU) is selected as STM32F103C8T6, which has a main frequency of 72MHz, its internal timer (TIM) can provide nanosecond-level accuracy, and has rich USART, SPI and other peripheral interfaces, which can fully meet the requirements of the present application for the main clock source at a very low cost, and is connected to the host computer through a USB line.

[0086] Laser radar (LiDAR): Ouster OS0-128 is selected. This radar supports external synchronization input, can parse serial messages in GPRMC format and receive PPS (Pulse Per Second) pulse signals. It is connected to the host computer through the manufacturer's interface box and network cable.

[0087] Industrial camera (Camera): Hikvision MV-CS050-60UC is selected. It is a 4-megapixel industrial camera that supports multiple hardware trigger modes (Hardware Trigger). It is connected to the host computer through a USB3.0 interface. By registering the "trigger signal callback" function in the development software, the time required from the arrival of the signal at the camera end to the reception of the complete image data by the host computer is measured. The end-to-end delay of this camera is usually stable at a few milliseconds to a few tens of milliseconds, which is much smaller than the 100ms period corresponding to a 10Hz acquisition frequency, meeting the low-delay characteristic.

[0088] Inertial Measurement Unit (IMU): Xsens MTi-630 is chosen. The IMU outputs data through a serial port, and its supporting software or data protocol supports parsing the occurrence of external events (such as pulses). It is connected to the host computer through a USB cable.

[0089] The microcontroller is connected to each sensor through a Dupont line. The GPIO interface corresponding to the synchronization control pulse signal of the message is connected to the GPIO Input pin of the laser radar and the inertial measurement unit through a one-to-two Dupont line. The GPIO interface corresponding to the hardware trigger pulse signal in the microcontroller is connected to the trigger signal corresponding pin of the industrial camera through a Dupont line. The GPIO pin in the microcontroller for sending pseudo-GPRMC messages is connected to the GPRMC Input pin of the laser radar through a Dupont line. The GND pins of the three sensors need to be connected to the GND pin of the microsensor through a Dupont line. In addition, a UART-to-USB special adapter is needed to connect the pseudo-GPRMC messages sent by the microcontroller through a set of UART pins to the USB interface of the host computer. The laser radar transmits data directly to the host computer through an Ethernet cable, and the industrial camera and the inertial measurement unit transmit data to the host computer through a USB interface.

[0090] In addition, in this embodiment, in order to avoid the scanning angle of the laser radar being blocked, the industrial camera and the inertial measurement unit are both installed on the underside of the handheld scanning platform, and the laser radar is separately installed on the side of the platform and is raised to a certain extent. All sensors are hard-linked to the platform through a bolt-nut structure to ensure the stability of the scanning.

[0091] Physical platform: SolidWorks software is used to complete three-dimensional structure modeling, and PLA material is used to make various assembly parts through Allct330Pro printer. Finally, all the above hardware and power supply modules are integrated into a handheld frame with a total weight of about 2 kg.

[0092] Software development: STM32CubeIDE software is used in the Windows system for firmware development of the MCU. C++ is used on the Ubuntu 22.04 system to develop the entire software.

[0093] Extrinsic parameter calibration: A calibration board with a thickness of about 2 cm, a width of 140 cm, and a height of 100 cm is made of PVC. The size is as large as possible to reduce the deviation of edge detection and visual detection.

[0094] The visualization function of the host computer of this embodiment uses the Pangolin graphics library to display the currently collected camera image, the three-dimensional laser point cloud projected onto the image, and the polyline change graph of the angular velocity and acceleration measured by the inertial measurement unit in a window in real time, providing intuitive feedback on the scanning process for the operator.

[0095] The handheld multi-sensor fusion scanning platform of the present embodiment is capable of stably collecting, synchronizing, saving and visualizing multi-sensor data at a frequency of 10 Hz.

[0096] Those skilled in the art can understand that the above description is only a preferred example of the application and is not used to limit the application, although the application is described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions recorded in the foregoing examples or make equivalent replacement for part of the technical features. Any modification, equivalent replacement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A time synchronization system for a handheld multi-sensor fusion scanning platform, characterized in that, The system includes a host computer and a microcontroller configured as the system's master clock source; Both the microcontroller and the host computer are connected to multiple sensors on the handheld multi-sensor fusion scanning platform; The microcontroller is used to generate a phase-locked composite synchronization signal and send it to the host computer and various sensors; the composite synchronization signal includes at least a message synchronization control pulse signal, a hardware trigger pulse signal, and a serial time message that defines the master time system, i.e., a pseudo-GPRMC message. The host computer runs a time synchronization device, which includes a raw data receiving module, a timestamp synchronization calibration module, and a data storage and visualization module. The raw data receiving module asynchronously receives raw data from various sensors and microcontrollers of the handheld multi-sensor fusion scanning platform by forming a thread group in parallel through multiple threads. After adding the local system timestamp of the host computer to the raw data, the raw data and the local system timestamp of the host computer are assembled into a custom structure and pushed into the corresponding input buffer of each sensor and microcontroller. The timestamp synchronization calibration module operates in a producer-consumer mode, simultaneously monitoring multiple input buffers. It extracts pseudo-GPRMC messages from the input buffers as reference frames, uses the local system timestamp of the corresponding host computer encapsulated in the same structure to search for data packets in the other three buffers, performs nearest neighbor matching, and executes a heterogeneous composite time synchronization strategy for different sensors. It assembles the data into a synchronization data frame containing data from multiple heterogeneous sensors and pushes it to the alignment buffer. The data storage and visualization module is used to achieve continuous data saving, interruption and recovery at any time, and real-time visualization of the acquisition process.

2. The time synchronization system of the handheld multi-sensor fusion scanning platform according to claim 1, characterized in that, The microcontroller includes an internal timer that supports interrupt handling and callback functions, as well as a pulse transmission module and a message transmission module.

3. The time synchronization system of the handheld multi-sensor fusion scanning platform according to claim 1, characterized in that, Both the input buffer and the alignment buffer are lock-free circular queue data structures implemented based on atomic variables.

4. The time synchronization system of the handheld multi-sensor fusion scanning platform according to claim 1, characterized in that, The sensors on the handheld multi-sensor fusion scanning platform include the following three types: Sensors that support external timing synchronization via pseudo-GPRMC messages; Sensors that support pulse event monitoring; It does not support external time synchronization via pseudo-GPRMC messages and supports pulse event monitoring but supports low-latency sensors triggered externally by hardware; specifically, the low latency means that the transmission delay of the sensor is less than the period of the hardware trigger pulse signal.

5. The time synchronization system of the handheld multi-sensor fusion scanning platform according to claim 1, characterized in that, The sensors on the handheld multi-sensor fusion scanning platform include lidar, inertial measurement units, and industrial cameras.

6. A time synchronization method for a handheld multi-sensor fusion scanning platform, characterized in that, This method is based on the time synchronization system of the handheld multi-sensor fusion scanning platform described in claim 1, and includes the following steps: Step 1: The microcontroller, which serves as the master clock source, generates and distributes a set of composite synchronization signals to the host computer and various sensors. The composite synchronization signals include at least a message synchronization control pulse signal, a hardware trigger pulse signal, and a serial time message that defines the master time system, namely a pseudo-GPRMC message. Step 2: For sensors that support external time synchronization via pseudo-GPRMC messages, the sensor simultaneously receives the message synchronization control pulse signal and the pseudo-GPRMC message to synchronize the time, so that the timestamp of its output data is directly aligned with the master time system and sent to the host computer. For sensors that support pulse event monitoring, the sensor receives the message synchronization control pulse signal and sends an event data packet with an internal timestamp to the host computer. For low-latency sensors that do not support external time synchronization of pseudo-GPRMC messages and support pulse event listening but support external hardware triggering, the sensor receives the hardware trigger pulse signal to trigger its data acquisition and sends the obtained data packet to the host computer. Step 3: The host computer asynchronously receives raw data from each sensor and the microcontroller, adds the host computer's local system timestamp to the raw data, assembles the raw data and the host computer's local system timestamp into a custom structure, and pushes it into the corresponding input buffer of each sensor and the microcontroller. Step 4: The host computer operates in a producer-consumer mode, simultaneously monitoring multiple input buffers. It retrieves pseudo-GPRMC messages from the input buffers as reference frames, uses the local system timestamp of the host computer encapsulated in the same structure to search for data packets in the other three input buffers, performs nearest neighbor matching, and executes a heterogeneous composite time synchronization strategy for different sensors. It assembles the data into a synchronization data frame containing data from multiple heterogeneous sensors and pushes it to the alignment buffer for data storage and visualization.

7. The time synchronization method for the handheld multi-sensor fusion scanning platform according to claim 6, characterized in that, For sensors that support external time synchronization via pseudo-GPRMC messages, the time synchronization strategy adopts direct external time synchronization, that is, simultaneously receiving the synchronization control pulse signal and the pseudo-GPRMC message to calibrate its internal clock to the master time system. For sensors that support pulse event monitoring, the time synchronization strategy is as follows: the host computer continuously compares the internal timestamps of event data packets. T imu With the master timestamp in the pseudo GPRMC message T mcu Dynamically calculate and maintain the linear mapping relationship and time offset Δ between the sensor's internal time domain and the main time system domain. t And use this offset to correct the timestamps of all sensor measurement data; For low-latency sensors that do not support external time synchronization via pseudo-GPRMC messages and support pulse event monitoring but support external hardware triggering, the time synchronization strategy is as follows: the host computer uses the nearest neighbor matching method to associate the collected data with the master time corresponding to the pulse.

8. The time synchronization method for the handheld multi-sensor fusion scanning platform according to claim 7, characterized in that, The host computer uses a nearest neighbor matching method to correlate the collected data with the main time corresponding to the pulse, specifically: When the host computer receives a pseudo GPRMC message, it corrects the timestamp of the first data packet that arrives afterward to T, and sets the timestamps of the subsequent N-1 data packets to T+1 / N, T+2 / N, ..., T+(N-1) / N seconds in sequence. This process is repeated after the next pseudo GPRMC message arrives, thereby achieving accurate timestamp calibration of the sensor; where T represents an integer second and N represents the number of hardware trigger pulse cycles per second.

9. The time synchronization method for the handheld multi-sensor fusion scanning platform according to claim 7, characterized in that, The time synchronization strategy adopts external direct time synchronization, specifically: When the rising edge of the message synchronization control pulse signal is detected, the sensor records a precise time point and waits for the first pseudo GPRMC message to arrive. Once the pseudo GPRMC message is received, the absolute time parsed from the pseudo GPRMC message is used to synchronize the previously recorded time point.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the time synchronization method of the handheld multi-sensor fusion scanning platform as described in any one of claims 6 to 9.

Images