Map construction method and device based on multi-source sensor fusion and storage medium

The multi-source sensor fusion approach using laser radar, inertial navigation, and industrial camera addresses the limitations of single-sensor mapping by integrating calibrated and synchronized data for accurate and comprehensive map construction.

CN119063710APending Publication Date: 2024-12-03SGCC SMART ENERGY & ELECTRIC TRANSPORTATION TECH INNOVATION CENT (SUZHOU) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310626257.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing map construction methods using single sensors like binocular cameras, depth cameras, and radars suffer from limitations such as light interference, limited ranging, poor anti-interference ability, high cost, and sparse data, leading to inaccurate and incomplete scene representation.

Method used

A map construction method utilizing multi-source sensor fusion involving laser radar, inertial navigation device, and industrial camera, including calibration of internal and external parameters, time synchronization, and data fusion to integrate point cloud, GPS, and image data for comprehensive and accurate mapping.

Benefits of technology

Ensures high-precision map construction by leveraging the strengths of multiple sensors, ensuring data accuracy, consistency, and completeness through calibration and synchronization, enabling real-time mapping.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119063710A_ABST
    Figure CN119063710A_ABST
Patent Text Reader

Abstract

The invention discloses a map construction method based on multi-source sensor fusion, sensors comprise a laser radar, inertial navigation equipment and an industrial camera, and the map construction method comprises the following steps: calibrating the internal reference of the industrial camera; calibrating first external parameters of the laser radar and the industrial camera; calibrating second external parameters of the inertial navigation equipment and the laser radar; performing time synchronization processing on the acquisition time of the laser radar, the inertial navigation equipment and the industrial camera; and based on the internal reference, the first external reference and the second external reference, constructing a map according to the point cloud data and odometer information acquired by the laser radar, the GPS positioning data acquired by the inertial navigation equipment and the image data acquired by the industrial camera. The advantages of image data collected by the industrial camera, point cloud data and odometer information collected by the laser radar and GPS positioning data collected by the inertial navigation equipment are fully exerted, the comprehensiveness, uniformity and precision of mapping data are ensured, and the construction of a high-precision map is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of map construction technology, and in particular to a map construction method, device and storage medium based on multi-source sensor fusion. Background Art

[0002] In recent years, with the development of artificial intelligence, mobile robots and autonomous driving have attracted more and more attention, and one of the main problems that needs to be solved is map construction. How to build fast and accurate maps has become a very important issue in mobile robots and unmanned vehicles.

[0003] At present, the mainstream methods generally use three types of sensors: binocular cameras, depth cameras and radars. The three sensors have inevitable shortcomings due to their own characteristics. The binocular camera is greatly affected by light and texture, and the depth camera has a limited range. The anti-interference ability is poor, and the lidar or millimeter-wave radar is expensive and the data is sparse. It can be seen that the construction of a map by a single sensor cannot fully and accurately reflect the structure of the scene, and its construction data is single, and the accuracy of the map cannot be guaranteed. Summary of the invention

[0004] In order to overcome the above-mentioned background technology, the current mainstream methods generally use three types of sensors, binocular cameras, depth cameras and radars. The three sensors have inevitable shortcomings due to their own characteristics. The binocular camera is greatly affected by light and texture, and the depth camera has a limited ranging range. The anti-interference ability is poor, and the lidar or millimeter-wave radar is expensive and the data is sparse. It can be seen that a single sensor constructs a map that cannot fully and accurately reflect the structure of the scene, and its construction data is single and cannot guarantee the shortcomings of map accuracy. The purpose of the present invention is to provide a map construction method, device and storage medium based on multi-source sensor fusion.

[0005] In order to achieve the above purpose, the present invention adopts the following technical solution:

[0006] In a first aspect of the present invention, a map construction method based on multi-source sensor fusion is provided, wherein the sensor includes a laser radar, an inertial navigation device and an industrial camera, and the map construction method includes:

[0007] Calibrate the internal parameters of the industrial camera;

[0008] Calibrate the first external parameters of the laser radar and the industrial camera;

[0009] Calibrate the second external parameters of the inertial navigation device and the laser radar;

[0010] Performing time synchronization processing on the acquisition time of the laser radar, inertial navigation device and industrial camera;

[0011] Based on the internal parameters, the first external parameters and the second external parameters, a map is constructed according to the point cloud data and odometer information collected by the laser radar, the GPS positioning data collected by the inertial navigation device, and the image data collected by the industrial camera.

[0012] The beneficial effects of the present invention are as follows: through the external parameter calibration of multi-source sensors - lidar, inertial navigation equipment and industrial cameras, as well as the internal parameter calibration of industrial cameras, the accuracy of their collected data is guaranteed; through the synchronous processing of collection time, the consistency of the actual collected data of each sensor is guaranteed, so that it is convenient to build a map according to the collected data in real time; through the image data collected by the industrial camera, the point cloud data and odometer information collected by the lidar, and the GPS positioning data collected by the inertial navigation equipment, their respective advantages are fully utilized to ensure the comprehensiveness, uniformity and accuracy of the mapping data, and realize the construction of high-precision maps.

[0013] In some possible implementations, the intrinsic parameters of the industrial camera are calibrated using a camera calibration toolkit provided by the ROS system.

[0014] In some possible implementations, arucotag is used to calibrate the first external parameters of the laser radar and the industrial camera.

[0015] In some possible implementations, the second external parameters of the inertial navigation device and the laser radar are calibrated by a hand-eye calibration method.

[0016] In some possible implementations, by giving the laser radar and the industrial camera a 10HZ frequency current pulse, the laser radar and the industrial camera can start collecting data at the same time.

[0017] In some possible implementations, the inertial navigation device obtains collected data that is time synchronized with the industrial camera or lidar by means of a time nearest neighbor query.

[0018] In a second aspect of the present invention, a map construction device based on multi-source sensor fusion is provided, wherein the sensors include a laser radar, an inertial navigation device and an industrial camera, and the device includes

[0019] The first calibration module is used to calibrate the internal parameters of the industrial camera;

[0020] A second calibration module: calibrating the first external parameters of the laser radar and the industrial camera;

[0021] The third calibration module is used to calibrate the second external parameters of the inertial navigation device and the laser radar;

[0022] Time synchronization module: performs time synchronization processing on the acquisition time of the laser radar, inertial navigation device and industrial camera;

[0023] Map construction module: Based on the internal parameters, the first external parameters and the second external parameters, a map is constructed according to the point cloud data and odometer information collected by the laser radar, the GPS positioning data collected by the inertial navigation device and the image data collected by the industrial camera.

[0024] In some possible implementations, the intrinsic parameters of the industrial camera are calibrated using a camera calibration toolkit provided by the ROS system.

[0025] In some possible implementations, arucotag is used to calibrate the first external parameters of the laser radar and the industrial camera.

[0026] In some possible implementations, the second external parameters of the inertial navigation device and the laser radar are calibrated by a hand-eye calibration method.

[0027] In some possible implementations, by giving the laser radar and the industrial camera a 10HZ frequency current pulse, the laser radar and the industrial camera can start collecting data at the same time.

[0028] In some possible implementations, the inertial navigation device obtains collected data that is time synchronized with the industrial camera or lidar by means of a time nearest neighbor query.

[0029] A third aspect of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned map construction method based on multi-source sensor fusion are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flowchart of the steps of a map construction method based on multi-source sensor fusion according to an embodiment of the present invention;

[0031] Figure 2 It is a structural schematic diagram of a map building device based on multi-source sensor fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0033] See attached Figure 1 As shown, this embodiment provides a map construction method based on multi-source sensor fusion, the sensors include lidar, inertial navigation equipment and industrial cameras, and the specific parameters are shown in Table 1.

[0034] Table 1: Parameter information of LiDAR, inertial navigation equipment and industrial camera

[0035] Sensor Name Equipment voltage / power Sensor Model LiDAR 24V / 20W OS1-64-gen1 Industrial Cameras 5V / 2.5W BFS-PGE-31S4C-C Inertial Navigation Equipment-Daishi Inertial Navigation 12V / 15W IFS-2000

[0036] The map construction method comprises the following steps:

[0037] S1: Calibrate the internal parameters of the industrial camera using the camera calibration toolkit provided by the ROS system.

[0038] S2: Use arucotag to calibrate the first external parameters of the laser radar and industrial camera.

[0039] Specifically, the lidar and industrial camera can be calibrated by identifying tags. The industrial camera identifies arucotag to determine the tag's position and corner points. The lidar obtains the tag's position and corner point coordinates by matching it with the prior world map, and then uses the ICP algorithm to get an initial value and then uses the SolvePnP algorithm to get the first external parameter.

[0040] S3: Calibrate the second external parameters of the inertial navigation device and the laser radar through the hand-eye calibration method.

[0041] Specifically, the motion trajectory of the laser radar around the figure 8 is obtained through the lego-loam algorithm, and the GPS output data obtained by the inertial navigation device is recorded at the same time, and then the hand-eye calibration algorithm is used to obtain the second external parameter.

[0042] S4: Perform time synchronization processing on the acquisition time of the laser radar, inertial navigation device and industrial camera.

[0043] Since the startup time of each sensor is inconsistent when collecting data, the timestamps of data collected by different sensors will also have some differences. At the same time, the collection frequency of different sensors is also inconsistent. In order to avoid these problems in subsequent processing, time synchronization processing is required.

[0044] Specifically, by giving the laser radar and the industrial camera a 10HZ frequency current pulse, the laser radar and the industrial camera can start collecting data at the same time. The inertial navigation device obtains the collected data synchronized with the industrial camera or the laser radar by means of a time nearest neighbor query.

[0045] S5: Based on the internal parameter, the first external parameter and the second external parameter, a map is constructed according to the point cloud data and odometer information collected by the laser radar, the GPS positioning data collected by the inertial navigation device and the image data collected by the industrial camera.

[0046] Specifically, the idea of ​​fusing the point cloud data of the LiDAR and the GPS positioning data output by the inertial navigation device is used to build the map. The more accurate GPS positioning data is used to stitch the point cloud of the LiDAR. The confidence of the LiDAR odometer is improved in the scenario where the GPS signal is weak. The loop detection and image data optimization methods are used to further improve the map. Figure 1 Consistency.

[0047] The beneficial effects of the present invention are as follows: through the external parameter calibration of multi-source sensors - lidar, inertial navigation equipment and industrial cameras, as well as the internal parameter calibration of industrial cameras, the accuracy of their collected data is guaranteed; through the synchronous processing of collection time, the consistency of the actual collected data of each sensor is guaranteed, so that it is convenient to build a map according to the collected data in real time; through the image data collected by the industrial camera, the point cloud data and odometer information collected by the lidar, and the GPS positioning data collected by the inertial navigation equipment, their respective advantages are fully utilized to ensure the comprehensiveness, uniformity and accuracy of the mapping data, and realize the construction of high-precision maps.

[0048] See attached Figure 2 As shown, this embodiment also provides a map construction device based on multi-source sensor fusion, wherein the sensor includes a laser radar, an inertial navigation device and an industrial camera, and the device includes

[0049] The first calibration module is used to calibrate the internal parameters of the industrial camera;

[0050] A second calibration module: calibrating the first external parameters of the laser radar and the industrial camera;

[0051] The third calibration module is used to calibrate the second external parameters of the inertial navigation device and the laser radar;

[0052] Time synchronization module: performs time synchronization processing on the acquisition time of the laser radar, inertial navigation device and industrial camera;

[0053] Map construction module: Based on the internal parameters, the first external parameters and the second external parameters, a map is constructed according to the point cloud data and odometer information collected by the laser radar, the GPS positioning data collected by the inertial navigation device and the image data collected by the industrial camera.

[0054] The first calibration module, the second calibration module and the third calibration module are used to complete the external parameter calibration of the multi-source sensors - lidar, inertial navigation equipment and industrial cameras, as well as the internal parameter calibration of the industrial camera, to ensure the accuracy of the collected data; the collection time is synchronized by the time synchronization module to ensure the consistency of the actual collected data of each sensor, so as to facilitate the real-time construction of the map based on the collected data; the map construction module fully utilizes the image data collected by the industrial camera, the point cloud data and odometer information collected by the lidar, and the GPS positioning data collected by the inertial navigation equipment to ensure the comprehensiveness, uniformity and accuracy of the mapping data and realize the construction of high-precision maps.

[0055] In some implementations, the internal parameters of the industrial camera are calibrated using a camera calibration toolkit provided by the ROS system.

[0056] The open source camera calibration toolkit (camera_calibration package) in the ROS system is used to obtain the camera intrinsic parameters and camera distortion parameters. The camera intrinsic parameters specifically include the camera intrinsic parameters fx, fy, cx, cy; the camera distortion parameters specifically include the radial distortion parameters k1, k2, k3 and the tangential distortion coefficients p1, p2.

[0057] In some embodiments, arucotag is used to calibrate the first external parameters of the laser radar and the industrial camera.

[0058] In some embodiments, the second external parameters of the inertial navigation device and the lidar are calibrated by a hand-eye calibration method.

[0059] In some embodiments, by giving the laser radar and the industrial camera a 10HZ frequency current pulse, the laser radar and the industrial camera can start collecting data at the same time.

[0060] In some implementations, the inertial navigation device obtains collected data that is time synchronized with an industrial camera or a laser radar by means of a time nearest neighbor query.

[0061] This embodiment provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned map construction method based on multi-source sensor fusion are implemented.

[0062] The processor may also be referred to as a CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor may be:

[0063] DSP (Digital Signal Processor, digital signal processor, digital signal processor is a processor composed of large-scale or ultra-large-scale integrated circuit chips to complete certain signal processing tasks. It has gradually developed to meet the needs of high-speed real-time signal processing tasks. With the development of integrated circuit technology and digital signal processing algorithms, the implementation methods of digital signal processors are constantly changing, and the processing functions are constantly improving and expanding.)

[0064] ASIC (Application Specific Integrated Circuit) is an integrated circuit designed and manufactured to meet the needs of specific users and specific electronic devices.

[0065] FPGA (Field Programmable Gate Array) is a product further developed on the basis of programmable devices such as PAL (Programmable Array Logic) and GAL (generic array logic). It emerged as a semi-custom circuit in the field of application-specific integrated circuits (ASICs), which not only solves the shortcomings of custom circuits, but also overcomes the shortcomings of the limited number of gate circuits of the original programmable devices.

[0066] A general-purpose processor may be a microprocessor or any conventional processor.

[0067] Other programmable logic devices, discrete gate or transistor logic devices and discrete hardware components, etc.

[0068] The storage medium stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, or a tablet.

[0069] The above implementation modes are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with this technology to understand the content of the present invention and implement it, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. Map construction method based on multi-source sensor fusion, Features: The sensor includes a laser radar, an inertial navigation device and an industrial camera, and the map construction method includes: Calibrate the internal parameters of the industrial camera; Calibrate the first external parameters of the laser radar and the industrial camera; Calibrate the second external parameters of the inertial navigation device and the laser radar; Performing time synchronization processing on the acquisition time of the laser radar, inertial navigation device and industrial camera; Based on the internal parameters, the first external parameters and the second external parameters, a map is constructed according to the point cloud data and odometer information collected by the laser radar, the GPS positioning data collected by the inertial navigation device, and the image data collected by the industrial camera.

2. The map construction method based on multi-source sensor fusion according to claim 1, Features: The internal parameters of the industrial camera are calibrated using the camera calibration toolkit provided by the ROS system.

3. The map construction method based on multi-source sensor fusion according to claim 1, Features: Arucotag is used to calibrate the first external parameters of the laser radar and the industrial camera.

4. The map construction method based on multi-source sensor fusion according to claim 1, Features: The second external parameters of the inertial navigation device and the laser radar are calibrated by a hand-eye calibration method.

5. The map construction method based on multi-source sensor fusion according to claim 1, Features: By giving the laser radar and industrial camera a 10HZ frequency current pulse, the laser radar and the industrial camera can start collecting data at the same time.

6. The map construction method based on multi-source sensor fusion according to claim 5, Features: The inertial navigation device obtains the collected data synchronized with the time of the industrial camera or the laser radar by means of a time nearest neighbor query.

7. Map building device based on multi-source sensor fusion, Features: The sensor includes a laser radar, an inertial navigation device and an industrial camera, and the device includes The first calibration module is used to calibrate the internal parameters of the industrial camera; A second calibration module: calibrating the first external parameters of the laser radar and the industrial camera; The third calibration module is used to calibrate the second external parameters of the inertial navigation device and the laser radar; Time synchronization module: performs time synchronization processing on the acquisition time of the laser radar, inertial navigation device and industrial camera; Map construction module: Based on the internal parameters, the first external parameters and the second external parameters, a map is constructed according to the point cloud data and odometer information collected by the laser radar, the GPS positioning data collected by the inertial navigation device and the image data collected by the industrial camera.

8. The map construction device based on multi-source sensor fusion according to claim 7, Features: The internal parameters of the industrial camera are calibrated using the camera calibration toolkit provided by the ROS system.

9. The map construction device based on multi-source sensor fusion according to claim 7, Features: Arucotag is used to calibrate the first external parameters of the laser radar and the industrial camera.

10. The map construction device based on multi-source sensor fusion according to claim 7, Features: The second external parameters of the inertial navigation device and the laser radar are calibrated by a hand-eye calibration method.

11. The map construction device based on multi-source sensor fusion according to claim 7, Features: By giving the laser radar and industrial camera a 10HZ frequency current pulse, the laser radar and the industrial camera can start collecting data at the same time.

12. The map construction device based on multi-source sensor fusion according to claim 11, Features: The inertial navigation device obtains the collected data synchronized with the time of the industrial camera or the laser radar by means of a time nearest neighbor query.

13. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the map construction method based on multi-source sensor fusion according to any one of claims 1 to 6 are implemented.

Citation Information

Cited By

  • Space scanning, modeling and positioning equipment based on multi-modal visual fusion

    CN120252708A