Perception device and method based on integration and synchronization of multi-modal sensor

Abstract

The invention discloses sensing equipment and method based on multi-modal sensor integration and synchronization, and relates to the technical field of robot sensing. The processing unit and the power supply unit are arranged in the main support, the processing unit is installed above the power supply unit, one end of the main support is provided with the display unit, the top of the main support is provided with the sensor unit, one side of the sensor unit is provided with the sensor synchronization module, and the power supply unit is electrically connected with the processing unit. The system has the advantages that a hardware trigger circuit is designed to trigger the laser radar, the visible light camera and the infrared camera to collect data at the same time, and then timestamp alignment of data of each frame of the laser radar, the visible light camera and the infrared camera is achieved in a memory sharing mode; through integration of a laser radar, a visible light camera and an infrared camera and a high-precision time synchronization method, comprehensive and high-quality data input is provided for an SLAM system.

Description

Technical Field

[0001] The present invention relates to the field of robot perception technology, and in particular to a perception device and method based on multi-modal sensor integration and synchronization. Background Art

[0002] The full name of simultaneous localization and mapping is Simultaneous Localization and Mapping, or SLAM for short. This technology usually uses vision, inertial, lidar and other sensors to build an environment map in an unknown scene, and uses this map to determine the movement trajectory of sensors such as cameras. Currently, there are mainly the following SLAM technologies: visual SLAM, lidar SLAM, and vision plus lidar SLAM.

[0003] The visual SLAM system only requires a visible light camera or an inertial measurement unit in terms of hardware. The visible light camera is used to perceive the texture information in the environment, and the inertial measurement unit provides acceleration and angular velocity information to estimate its own position and establish an environmental map to complete positioning in unknown environments. However, the existing visual SLAM system has poor environmental interference capabilities. Under non-ideal conditions such as light changes, light loss, or full of smoke, the visible light camera cannot perceive effective texture information, resulting in the failure of positioning and mapping algorithms.

[0004] The LiDAR SLAM system requires a LiDAR and an inertial measurement unit in hardware. It uses the LiDAR to scan the environment and generate a series of distance data point clouds and estimates its own motion through the inertial measurement unit. It updates and builds the environment map based on the point cloud data and motion estimation results. Although LiDAR is not sensitive to lighting conditions and can work in darkness, strong light and other conditions, it lacks the rich texture information provided by visible light cameras. In addition, in some complex environments, such as narrow corridors or dense urban environments, the multipath effect of LiDAR may cause data distortion.

[0005] The vision plus lidar SLAM system is a synchronous positioning and mapping technology that combines cameras and lidars. This system uses the rich texture information provided by visible light cameras and the three-dimensional point cloud data of lidars to work together to improve the accuracy and robustness of the SLAM system. However, under extreme lighting conditions (such as direct strong light, dark environment) and bad weather (fog, heavy rain), visible light cameras cannot work, lidar performance is affected, and point cloud data noise increases and quality decreases.

[0006] The existing simultaneous localization and mapping systems have performed well in many applications, but there are still some scenarios and problems that have not been fully considered or solved. For example, in extreme environments such as strong light, darkness, smoke, heavy rain, and no or low texture, the vision subsystem or lidar subsystem will fail and cannot produce satisfactory results. Summary of the invention

[0007] The purpose of the present invention is to provide a sensing device and method based on multimodal sensor integration and synchronization in view of the shortcomings and defects in the prior art. By using infrared camera imaging, which is only related to the thermal radiation of the scene, the device is less affected in harsh environments and has advantages such as night vision and anti-fog. It provides more one-dimensional information to the existing SLAM system and improves the robustness of the SLAM system in non-ideal scenes. The infrared camera can be used as an auxiliary sensor to provide additional scene information for the SLAM system. At the same time, in order to ensure the effect of the mapping algorithm of the multimodal sensor SLAM system, it is necessary to design a high-precision time synchronization method to ensure the alignment of multi-sensor data. A hardware trigger circuit is also designed to simultaneously trigger the laser radar, visible light camera and infrared camera to collect data. Then, the memory sharing method is used to achieve the alignment of the timestamps of each frame of data of the three. Through the integration of laser radar, visible light camera and infrared camera and the high-precision time synchronization method, the SLAM system is provided with comprehensive and high-quality data input.

[0008] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a perception device and method based on multimodal sensor integration and synchronization, which includes a sensor unit 1, a display unit 2 for displaying its related data, a processing unit 3 for processing the information of the sensor unit 1, a power supply unit 4 for powering the entire device, and a sensor synchronization module 5 and a main bracket 6 for synchronizing the sensor unit 1, wherein the processing unit 3 and the power supply unit 4 are arranged inside the main bracket 6, and the processing unit 3 is installed above the power supply unit 4, one end of the main bracket 6 is provided with a display unit 2, the top of the main bracket 6 is installed with a sensor unit 1, and one side of the sensor unit 1 is provided with a sensor synchronization module 5, the power supply unit 4 is electrically connected to its processing unit 3 and the sensor unit 1, and the processing unit 3 is electrically connected to its sensor unit 1, the display unit 2 and the sensor synchronization module 5.

[0009] Furthermore, the sensor unit 1 includes an infrared camera 1-1, a visible light camera 1-3, and a laser radar 1-2: the laser radar 1-2 can generate tens of thousands of points of three-dimensional point cloud data per frame by emitting laser pulses. These point cloud data provide high-precision three-dimensional information of the environment and accurately determine the shape and position of surrounding objects; the visible light camera 1-3 captures light signals through a CMOS sensor and converts them into digital image data, which is used to obtain two-dimensional image data of the environment and provide rich texture and color information; the infrared camera 1-1 uses an infrared detector to capture infrared radiation in the scene. Any object with a temperature above absolute zero will radiate infrared rays. According to the temperature and surface characteristics of the object, the intensity and wavelength of infrared rays radiated by different objects are different, and the infrared radiation energy is converted into electrical signals through the internal infrared detector, and the electrical signals are converted into image data through the processing unit.

[0010] Furthermore, the sensor synchronization module 5 is composed of a signal trigger circuit, and realizes the timestamp alignment of the three sensor data of the infrared camera 1-1, the visible light camera 1-3 and the laser radar 1-2 by means of shared memory. Specifically, the signal trigger circuit is composed of a single-chip microcomputer, a TTL to RS485 module and a TTL to USB module: the single-chip microcomputer is responsible for generating a synchronization trigger signal PWM wave and virtual GPRMC message data; the TTL to USB module is responsible for transmitting the virtual GPRMC message data from the single-chip microcomputer to the onboard computer; the TTL to RS485 module is responsible for converting the 0-5VPWM wave generated by the single-chip microcomputer, that is, the TTL signal, into an RS485 differential signal for triggering the laser radar to enter the synchronization state and divide the frequency.

[0011] Furthermore, the process of the sensor synchronization module 5 is specifically as follows: when the single-chip microcomputer generates a synchronization trigger signal PWM wave and a virtual GPRMC message data, two other 0-5V PWM waves, namely TTL signals, are also sent out at the same time to trigger the visible light camera 1-3 and the infrared camera 1-1 to collect data respectively. After receiving the differential signal and parsing the time information from the GPRMC data, the laser radar 1-2 will set the point cloud time of this frame to the GPS time parsed by the GPRMC, write the message time information of this frame of point cloud data into a shared memory, and then read the message time of this frame of point cloud data from the shared memory by the visible light image data and the infrared image data that are triggered at the same time and assign it to the current image frame time. The above operations realize the synchronization of each frame of data of the three sensors, providing an accurate time reference for the back-end SLAM algorithm.

[0012] A perception device and method based on multimodal sensor integration and synchronization, characterized in that it includes the following specific processes: under the processing unit 3 ROS platform, the driver of the laser radar 1-2 initializes a specific memory area through a smart pointer to store the laser radar 1-2 data timestamp, and uses smart pointer management to avoid memory leakage. After the laser radar 1-2 driver receives a hardware trigger signal, it directly accesses the shared memory address and writes the timestamp of this frame of laser radar 1-2 data. At the same time, the visible light camera 1-3 driver and the infrared camera 1-1 driver obtain the timestamp of this frame of laser radar 1-2 by reading the pointer and apply it to their respective image data. The same operation method is repeated for each frame of data to ensure that the timestamps of each frame of data are aligned.

[0013] After adopting the above technical solution, the beneficial effects of the present invention are as follows: by using infrared camera imaging, which is only related to the thermal radiation of the scene, the impact in harsh environments is small, and it has advantages such as night vision and anti-fog, providing more one-dimensional information to the existing SLAM system, improving the robustness of the SLAM system in non-ideal scenes, and the infrared camera can be used as an auxiliary sensor to provide additional scene information for the SLAM system. At the same time, in order to ensure the effect of the mapping algorithm of the multi-modal sensor SLAM system, it is necessary to design a high-precision time synchronization method to ensure the alignment of multi-sensor data. A hardware trigger circuit is also designed to simultaneously trigger the laser radar, visible light camera and infrared camera to collect data, and then use the memory sharing method to achieve the alignment of the timestamp of each frame of data of the three. Through the integration of laser radar, visible light camera and infrared camera and the high-precision time synchronization method, the SLAM system is provided with comprehensive and high-quality data input. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0015] Figure 1 It is a structural schematic diagram of the present invention.

[0016] Figure 2 It is a schematic diagram of the multi-sensor fusion hardware synchronization triggering solution in the present invention.

[0017] Figure 3 This is a schematic diagram of the multi-sensor synchronization principle in the present invention.

[0018] Explanation of the reference numerals: sensor unit 1, display unit 2, processing unit 3, power supply unit 4, sensor synchronization module 5, main bracket 6, infrared camera 1-1, visible light camera 1-3, laser radar 1-2. DETAILED DESCRIPTION

[0019] The technical solution adopted in this specific implementation is: Figure 1 As shown, it includes a sensor unit 1, a display unit 2 for displaying its related data, a processing unit 3 for processing the information of the sensor unit 1, a power supply unit 4 for powering the entire device, and a sensor synchronization module 5 and a main bracket 6 for synchronizing the sensor unit 1. The processing unit 3 and the power supply unit 4 are arranged inside the main bracket 6, and the processing unit 3 is installed above the power supply unit 4. The display unit 2 is provided at one end of the main bracket 6, the sensor unit 1 is installed on the top of the main bracket 6, and the sensor unit 1 is provided on one side of the sensor unit 1. The power supply unit 4 electrically connects its processing unit 3 and the sensor unit 1, and the processing unit 3 electrically connects its sensor unit 1, the display unit 2 and the sensor synchronization module 5.

[0020] More specifically, the sensor unit 1 includes an infrared camera 1-1, a visible light camera 1-3, and a laser radar 1-2:

[0021] LiDAR 1-2 can generate tens of thousands of points of 3D point cloud data per frame by emitting laser pulses. These point cloud data provide high-precision 3D information of the environment and accurately determine the shape and position of surrounding objects.

[0022] Visible light cameras 1-3 capture light signals through CMOS sensors and convert them into digital image data, which are used to obtain two-dimensional image data of the environment and provide rich texture and color information;

[0023] Infrared camera 1-1 uses an infrared detector to capture infrared radiation in a scene. Any object with a temperature above absolute zero will radiate infrared rays. The intensity and wavelength of infrared rays radiated by different objects vary according to the temperature and surface characteristics of the object. The infrared radiation energy is converted into an electrical signal through the internal infrared detector, and the electrical signal is converted into image data through the processing unit.

[0024] More specifically, the sensor synchronization module 5 is composed of a signal trigger circuit, and realizes the timestamp alignment of the three sensor data of the infrared camera 1-1, the visible light camera 1-3 and the laser radar 1-2 by sharing the memory, specifically:

[0025] The signal trigger circuit consists of a single chip microcomputer, a TTL to RS485 module and a TTL to USB module: see the overall hardware synchronization trigger solution Figure 2 shown.

[0026] The microcontroller is responsible for generating the synchronous trigger signal PWM wave and the virtual GPRMC message data;

[0027] The TTL to USB module is responsible for transmitting the virtual GPRMC message data from the microcontroller to the onboard computer;

[0028] The TTL to RS485 module is responsible for converting the 0-5V PWM wave generated by the microcontroller, namely the TTL signal, into an RS485 differential signal to trigger the laser radar to enter the synchronization state and divide the frequency.

[0029] To be more specific, the process of the sensor synchronization module 5 is as follows: when the single-chip microcomputer generates a synchronization trigger signal PWM wave and a virtual GPRMC message data, it also simultaneously sends out two other 0-5V PWM waves, namely TTL signals, which are used to trigger the visible light camera 1-3 and the infrared camera 1-1 to collect data respectively. After receiving the differential signal and parsing the time information from the GPRMC data, the laser radar 1-2 will set the point cloud time of this frame to the GPS time parsed by the GPRMC, write the message time information of this frame of point cloud data into a shared memory, and then read the message time of this frame of point cloud data from the shared memory by the visible light image data and the infrared image data that are triggered at the same time and assign it to the current image frame time. The above operations realize the synchronization of each frame of data of the three sensors, providing an accurate time reference for the back-end SLAM algorithm.

[0030] See also Figure 3 As shown, a sensing device and method based on multimodal sensor integration and synchronization is characterized in that it includes the following specific processes:

[0031] Under the ROS platform of processing unit 3, the driver of LiDAR 1-2 initializes a specific memory area through smart pointers to store the timestamp of LiDAR 1-2 data, and uses smart pointer management to avoid memory leaks. After the LiDAR 1-2 driver receives the hardware trigger signal, it directly accesses the shared memory address and writes the timestamp of this frame of LiDAR 1-2 data. At the same time, the driver of visible light camera 1-3 and the driver of infrared camera 1-1 obtain the timestamp of this frame of LiDAR 1-2 by reading the pointer and apply it to their respective image data. The same operation is repeated for each frame of data to ensure that the timestamps of each frame of data are aligned.

[0032] The above is only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A sensing device based on multimodal sensor integration and synchronization, characterized in that: It comprises a sensor unit (1), a display unit (2) for displaying relevant data thereof, a processing unit (3) for processing information of the sensor unit (1), a power supply unit (4) for supplying power to the entire device, a sensor synchronization module (5) for synchronizing the sensor unit (1), and a main bracket (6); the processing unit (3) and the power supply unit (4) are arranged inside the main bracket (6), and the processing unit (3) is installed above the power supply unit (4); one end of the main bracket (6) is provided with a display unit (2); the top of the main bracket (6) is provided with a sensor unit (1), and one side of the sensor unit (1) is provided with a sensor synchronization module (5); the power supply unit (4) is electrically connected to the processing unit (3) and the sensor unit (1), and the processing unit (3) is electrically connected to the sensor unit (1), the display unit (2) and the sensor synchronization module (5).

2. A sensing device based on multimodal sensor integration and synchronization according to claim 1, characterized in that: The sensor unit (1) comprises an infrared camera (1-1), a visible light camera (1-3), and a laser radar (1-2): LiDAR (1-2) can generate tens of thousands of points of three-dimensional point cloud data per frame by emitting laser pulses. These point cloud data provide high-precision three-dimensional information of the environment and accurately determine the shape and position of surrounding objects; Visible light cameras (1-3) capture light signals through CMOS sensors and convert them into digital image data, which are used to obtain two-dimensional image data of the environment and provide rich texture and color information; Infrared cameras (1-1) use infrared detectors to capture infrared radiation in a scene. Any object with a temperature above absolute zero will radiate infrared rays. The intensity and wavelength of infrared rays radiated by different objects vary depending on the temperature and surface characteristics of the object. The infrared radiation energy is converted into electrical signals through the internal infrared detector, and the electrical signals are converted into image data through the processing unit.

3. The sensing device based on multimodal sensor integration and synchronization according to claim 1, characterized in that: The sensor synchronization module (5) is composed of a signal trigger circuit and realizes the time stamp alignment of the three sensor data of the infrared camera (1-1), the visible light camera (1-3) and the laser radar (1-2) by means of shared memory, specifically: The signal trigger circuit consists of a single-chip microcomputer, a TTL to RS485 module and a TTL to USB module: The microcontroller is responsible for generating the synchronous trigger signal PWM wave and the virtual GPRMC message data; The TTL to USB module is responsible for transmitting the virtual GPRMC message data from the microcontroller to the onboard computer; The TTL to RS485 module is responsible for converting the 0-5V PWM wave generated by the microcontroller, namely the TTL signal, into an RS485 differential signal to trigger the laser radar to enter the synchronization state and divide the frequency.

4. The sensing device based on multimodal sensor integration and synchronization according to claim 1, characterized in that: The process of the sensor synchronization module (5) is specifically as follows: when the single chip microcomputer generates a synchronization trigger signal PWM wave and virtual GPRMC message data, it also simultaneously sends out two other 0-5V PWM waves, namely TTL signals, which are used to trigger the visible light camera (1-3) and the infrared camera (1-1) to collect data respectively. After receiving the differential signal and parsing the time information from the GPRMC data, the laser radar (1-2) sets the time of this frame of point cloud to the GPS time parsed by the GPRMC, writes the message time information of this frame of point cloud data into a shared memory, and then reads the message time of this frame of point cloud data from the shared memory by the visible light image data and the infrared image data obtained by the simultaneous triggering and assigns it to the current image frame time. The above operations realize the synchronization of each frame of data of the three sensors, providing an accurate time reference for the back-end SLAM algorithm.

5. A sensing device and method based on multimodal sensor integration and synchronization, characterized in that: It includes the following specific processes: Under the ROS platform of the processing unit (3), the driver of the laser radar (1-2) initializes a specific memory area through a smart pointer to store the timestamp of the laser radar (1-2) data, and uses smart pointer management to avoid memory leaks. After the laser radar (1-2) driver receives the hardware trigger signal, it directly accesses the shared memory address and writes the timestamp of this frame of laser radar (1-2) data. At the same time, the visible light camera (1-3) driver and the infrared camera (1-1) driver obtain the timestamp of this frame of laser radar (1-2) by reading the pointer and apply it to their respective image data. The same operation is repeated for each frame of data to ensure that the timestamps of each frame of data are aligned.

Images