Mapping method, device, equipment and product based on laser point cloud and image data

Abstract

The invention discloses a mapping method, device, equipment and product based on laser point cloud and image data, and relates to the technical field of multi-sensor data fusion. Acquiring laser point cloud data, image data, angular velocity data and acceleration data set equipment at each sampling time point; fusing the laser point cloud data and the image data to obtain point cloud map data; the pose of each Marker is calculated; the poses of the Markers with the same ID numbers are fused, and the average pose of the Markers with the ID numbers is obtained; calculating a coordinate system transformation matrix of the Marker relative to the reference ID number Marker, converting the pose of the Marker into the coordinate system of the reference ID number Marker to obtain the converted pose of each Marker, and generating Marker map data according to the converted pose; performing coordinate system transformation on the point cloud map data; and fusing the point cloud map data after coordinate system transformation and alignment with Marker map data to obtain scene point cloud map data. According to the invention, unified fusion of the Marker map and the laser point cloud map can be realized, and the navigation precision and stability of the robot are improved.

Description

Technical Field

[0001] This invention belongs to the field of multi-sensor data fusion technology, specifically relating to a mapping method, apparatus, equipment, and product based on laser point cloud and image data. Background Technology

[0002] With the rapid development of robotics technology, environmental perception has become an important indicator of a robot's intelligence level. Currently, robot environmental perception mainly relies on various sensors, such as LiDAR, cameras, and inertial measurement units (IMUs). Each of these sensors has its advantages and disadvantages. LiDAR can provide high-precision three-dimensional spatial information, but it is expensive and easily affected by lighting conditions. Cameras are less expensive and can provide rich semantic information, but their positioning accuracy is difficult to guarantee due to limitations in texture features and lighting conditions. IMUs can provide continuous pose estimation, but they are easily affected by accumulated errors.

[0003] To overcome the limitations of single sensors, multi-sensor fusion technologies have emerged, such as laser-vision-inertial fusion methods and marker-based mapping methods. However, existing laser-vision-inertial fusion methods cannot unify multi-source information such as LiDAR point clouds, inertial measurement units, and visual markers into the same coordinate system. This results in data from different sensors being processed in different coordinate systems, failing to fully utilize the complementarity of multi-source information and affecting the robot's localization and navigation accuracy and stability. Marker-based mapping methods, on the other hand, completely lack a fusion mechanism with laser point cloud maps, cannot calculate the precise transformation matrix between the marker map and the point cloud map, and cannot construct a unified scene map containing geometric information and semantic tags. This affects the robot's localization and navigation accuracy and stability, limiting its application in complex environments. Furthermore, marker-based mapping methods require that a marker must appear in the same image as other markers already in the map to be added to the map; this co-occurrence constraint limits the flexibility of marker map construction.

[0004] Therefore, how to provide an effective solution to improve the flexibility of map construction and achieve high-precision map fusion has become a pressing problem to be solved by existing technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a mapping method, apparatus, device, and product based on laser point cloud and image data to solve the above-mentioned problems existing in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a mapping method based on laser point cloud and image data, comprising: The system acquires laser point cloud data collected by the lidar at each sampling time point within the sampling period, image data collected by the camera at each sampling time point, and angular velocity and acceleration data collected by the inertial measurement unit at each sampling time point, wherein the lidar, the camera, and the inertial measurement unit are all mounted on the same mobile acquisition device. By fusing laser point cloud data and image data at the same sampling time point, point cloud map data at each sampling time point can be obtained. Based on the pose of the lidar at each sampling time point and the extrinsic data of the lidar and the camera, the pose of the camera at each sampling time point is determined. Detect markers in each image data and calculate the pose of each marker in each image data; The poses of markers with the same ID number in each image data are fused using the Markley method to obtain the average pose of each ID number marker. Based on the average pose of each ID number Marker and the average pose of the reference ID number Marker, calculate the coordinate system transformation matrix of the initial coordinate system of each ID number Marker relative to the coordinate system of the reference ID number Marker, and based on the coordinate system transformation matrix, transform the pose of all Markers in each image data to the coordinate system of the reference ID number Marker to obtain the transformed pose of each Marker in each image data. Based on the transformed pose of each marker in each image data, generate marker map data corresponding to each image data and containing the marker pose; Based on the coordinate system transformation matrix, the point cloud map data corresponding to the same sampling time point of each image data is transformed to obtain the point cloud map data after coordinate system transformation and alignment of each sampling time point. The point cloud map data after coordinate system transformation and alignment at each sampling time point is fused with the corresponding Marker map data to obtain the final scene point cloud map data at each sampling time point.

[0007] In one possible design, after calculating the pose of each marker in each image data, the method further includes: Transform the pose of each marker in each image data to the global coordinate system at the initial time; Remove markers with abnormal pose fluctuations from each image data to obtain the filtered image data; Correspondingly, the poses of markers with the same ID number in each image data are fused using the Markley method to obtain the average pose of each ID number marker, including: The Markley method is used to fuse the poses of markers with the same ID number in each filtered image data to obtain the average pose of each ID number marker.

[0008] In one possible design, markers with abnormal pose fluctuations are removed from each image data set, resulting in filtered image data, including: For markers with the same ID number, calculate the median absolute deviation of the pose of markers with the same ID number in each image data. Markers with the same ID number whose median absolute deviation of pose in each image data exceeds a preset absolute deviation threshold are designated as abnormal markers. Remove all abnormal markers from each image data to obtain the filtered image data.

[0009] In one possible design, the poses of markers with the same ID number in each of the filtered image data are fused using the Markley method to obtain the average pose of each ID number marker, including: Calculate the average quaternion and average translation vector of the same ID number marker in each filtered image data; A weighted matrix is ​​constructed based on the average quaternion of all identical ID markers in each filtered image data. Perform eigenvalue decomposition on the weighted matrix to determine the largest feature in the weighted matrix; The average of the quaternions of the Markers with the same ID number corresponding to the largest feature is taken as the target quaternion average. The average value of the target quaternion is combined with the average value of the translation vector of each ID number marker in each filtered image data to obtain the average pose of each ID number marker.

[0010] In one possible design, before calculating the coordinate system transformation matrix of each ID number marker relative to the coordinate system of the reference ID number marker based on the average pose of each ID number marker and the average pose of the reference ID number marker, the method further includes: Determine whether a reference ID number (Marker) has been specified; If no reference ID number Marker is specified, a random ID number Marker with the same ID number will be assigned as the reference ID number Marker.

[0011] In one possible design, the pose of each marker in each image data is calculated, including: The pose of each marker in each image data relative to the camera at the corresponding sampling time point is calculated using the PnP algorithm. Based on the camera pose at each sampling time point and the pose of each marker in each image data relative to the camera at the corresponding sampling time point, the pose of each marker in each image data is calculated.

[0012] In one possible design, laser point cloud data and image data at the same sampling time point are fused to obtain point cloud map data for each sampling time point, including: The laser visual SLAM algorithm is used to fuse laser point cloud data and image data at the same sampling time point to obtain point cloud map data at each sampling time point.

[0013] Secondly, the present invention provides a mapping device based on laser point cloud and image data, comprising: The acquisition unit is used to acquire laser point cloud data collected by the lidar at each sampling time point within the sampling period, image data collected by the camera at each sampling time point, and angular velocity data and acceleration data collected by the inertial measurement unit at each sampling time point, wherein the lidar, the camera, and the inertial measurement unit are all installed on the same mobile acquisition device. The first fusion unit is used to fuse laser point cloud data and image data at the same sampling time point to obtain point cloud map data at each sampling time point; The determination unit is used to determine the pose of the camera at each sampling time point based on the pose of the lidar at each sampling time point and the extrinsic data of the lidar and the camera. The detection and calculation unit is used to detect markers in each image data and calculate the pose of each marker in each image data. The second fusion unit is used to fuse the poses of markers with the same ID number in each image data using the Markley method to obtain the average pose of each ID number marker. The calculation and transformation unit is used to calculate the coordinate system transformation matrix of the initial coordinate system of each ID number Marker relative to the coordinate system of the reference ID number Marker based on the average pose of each ID number Marker and the average pose of the reference ID number Marker, and transform the pose of all Markers in each image data to the coordinate system of the reference ID number Marker based on the coordinate system transformation matrix, so as to obtain the transformed pose of each Marker in each image data. The generation unit is used to generate Marker map data corresponding to each image data and containing the Marker pose based on the transformed pose of each Marker in each image data. The transformation unit is used to perform coordinate system transformation on the point cloud map data corresponding to the same sampling time point of each image data based on the coordinate system transformation matrix, so as to obtain the point cloud map data after coordinate system transformation and alignment of each sampling time point. The third fusion unit is used to fuse the point cloud map data after coordinate system transformation and alignment at each sampling time point with the corresponding Marker map data to obtain the final scene point cloud map data at each sampling time point.

[0014] Thirdly, the present invention provides an electronic device comprising a memory, a processor, and a transceiver connected in sequence and communication, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the mapping method based on laser point cloud and image data as described in the first aspect or any possible design of the first aspect.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the mapping method based on laser point cloud and image data as described in the first aspect or any possible design of the first aspect.

[0016] Fifthly, the present invention provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform a mapping method based on laser point cloud and image data as described in the first aspect or any possible design of the first aspect.

[0017] Beneficial effects: This invention provides a multi-sensor data fusion scheme that enables unified fusion of marker maps and laser point cloud maps, independent of marker co-occurrence constraints, resulting in more flexible map construction. Specifically, it acquires laser point cloud data collected by the LiDAR at each sampling time point within the sampling period, image data collected by the camera at each sampling time point, and angular velocity and acceleration data collected by the inertial measurement unit at each sampling time point. The laser point cloud data and image data at the same sampling time point are fused to obtain point cloud map data for each sampling time point. Based on the camera's initial pose, angular velocity data at each sampling time point, and acceleration data at each sampling time point, the camera's pose at each sampling time point is determined. Markers in each image data are detected, and the pose of each marker in each image data is calculated. The poses of markers with the same ID number in each image data are fused using the Markley method to obtain the average pose of each ID number marker. Based on the average pose of each ID number marker and a reference ID number marker... The average pose is calculated, and the coordinate system transformation matrix of the initial coordinate system of each ID number Marker relative to the coordinate system of the reference ID number Marker is calculated. Based on the coordinate system transformation matrix, the poses of all Markers in each image data are transformed to the coordinate system of the reference ID number Marker to obtain the transformed pose of each Marker in each image data. Based on the transformed poses of each Marker in each image data, Marker map data corresponding to each image data and containing Marker poses are generated. Based on the coordinate system transformation matrix, the point cloud map data corresponding to the same sampling time point of each image data is transformed to obtain the point cloud map data after coordinate system transformation and alignment at each sampling time point. The point cloud map data after coordinate system transformation and alignment at each sampling time point is fused with the corresponding Marker map data to obtain the final scene point cloud map data at each sampling time point. This invention, through precise coordinate system alignment transformation and fusion modules, achieves for the first time the unified fusion of visual marker maps and laser point cloud maps. It can construct a unified scene map containing geometric information and semantic tags, realizing true multimodal map fusion. This solves the core problem of existing technologies that cannot simultaneously achieve marker map construction and unified fusion with point cloud maps, unifying multi-source information into the same coordinate system and significantly improving the accuracy and stability of robot localization and navigation. Furthermore, this invention does not require markers to appear in the same image as other markers already in the map, making map construction more flexible. It can construct complete maps even in scenes with sparse marker distribution, improving the system's adaptability and practicality.

[0018] Furthermore, the Markley method is used to fuse multiple pose data of each marker into an average pose. By constructing a quaternion weighted matrix and performing eigenvalue decomposition, the characteristics of quaternions on the manifold are fully considered. Compared with the simple averaging or weighted averaging of existing methods, the fusion accuracy is significantly improved, and multiple observations can be fully utilized to improve the accuracy of marker pose estimation.

[0019] Furthermore, outlier detection is performed on all observed marker poses using the median absolute deviation, which can effectively identify and remove abnormal markers caused by detection errors, occlusion, etc. Compared with existing methods that lack outlier detection mechanisms, this significantly improves mapping accuracy and robustness. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the mapping method based on laser point cloud and image data provided in this application embodiment; Figure 2 A block diagram of a mapping device based on laser point cloud and image data provided in an embodiment of this application; Figure 3 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0022] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0023] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0024] It should be understood that specific details are provided in the following description to facilitate a complete understanding of the exemplary embodiments. However, those skilled in the art will understand that the exemplary embodiments can be implemented without these specific details. For example, the system may be shown in block diagrams to avoid obscuring the example with unnecessary details. In other instances, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the exemplary embodiments.

[0025] To achieve the unified fusion of marker maps and laser point cloud maps, this application provides a mapping method, apparatus, device, and product based on laser point cloud and image data. The mapping method, apparatus, device, and product based on laser point cloud and image data can achieve the unified fusion of visual marker maps and laser point cloud maps, unifying multi-source information into the same coordinate system, significantly improving the accuracy and stability of robot positioning and navigation, while making map construction more flexible and improving the adaptability and practicality of the system.

[0026] The mapping method based on laser point cloud and image data provided in this application embodiment can be executed, but is not limited to, by a computer device or virtual machine with certain computing resources, such as by an electronic device such as a positioning and navigation robot, a personal computer, a smartphone or a personal digital assistant, or by a virtual machine.

[0027] It is understood that the execution entity described does not constitute a limitation on the embodiments of this application.

[0028] The mapping method based on laser point cloud and image data provided in the embodiments of this application will be described in detail below.

[0029] like Figure 1 The diagram shown is a flowchart of a mapping method based on laser point cloud and image data provided in the first aspect of the present application. The mapping method based on laser point cloud and image data may include, but is not limited to, the following steps S101-S109.

[0030] Step S101. Acquire the laser point cloud data collected by the lidar at each sampling time point within the sampling period, the image data collected by the camera at each sampling time point, and the angular velocity and acceleration data collected by the inertial measurement unit at each sampling time point.

[0031] The lidar, the camera, and the inertial measurement unit are all mounted on the same mobile acquisition device, and the coverage area of ​​the lidar is the same as the field of view of the camera. The mobile acquisition device may be, but is not limited to, a mobile robot, a drone, etc.

[0032] Before mapping, the mobile acquisition device can be controlled to move in the scene where mapping is required. During this process, laser point cloud data can be collected by LiDAR at each sampling time point within the sampling period, image data can be collected by camera at each sampling time point within the sampling period, and angular velocity and acceleration data can be collected by inertial measurement unit at each sampling time point within the sampling period.

[0033] Step S102. Fuse the laser point cloud data and image data at the same sampling time point to obtain point cloud map data at each sampling time point.

[0034] In one or more embodiments, existing LiDAR point cloud map generation methods (such as LiDAR visual SLAM algorithms, LiDAR visual SLAM2 algorithms, etc.) can be used to fuse LiDAR point cloud data and image data at the same sampling time point to obtain point cloud map data for each sampling time point. In this way, LiDAR point clouds and image pixels can be directly registered into a unified voxel map, realizing the construction of LiDAR point cloud maps.

[0035] Step S103. Based on the pose of the lidar at each sampling time point and the extrinsic data of the lidar and the camera, determine the pose of the camera at each sampling time point.

[0036] The pose includes position and attitude.

[0037] Step S104. Detect the markers in each image data and calculate the pose of each marker in each image data.

[0038] For each image data set, markers can be automatically detected, and the pose of each marker in each image data set can be calculated by combining the camera's pose. The markers in the image data can be manually labeled by the user or automatically labeled by the system.

[0039] Specifically, when calculating the pose of each marker in each image data set, the PnP algorithm can be used to first calculate the pose (position and orientation) of each marker relative to the camera at the corresponding sampling time point. Then, based on the camera pose at each sampling time point and the pose of each marker relative to the camera at the corresponding sampling time point, the pose of each marker in each image data set can be calculated. Since the camera pose at each sampling time point is known, the pose of each marker in each image data set can be calculated.

[0040] In one or more embodiments, after calculating the pose of each marker in each image data, the pose of each marker in each image data can also be transformed to the global coordinate system at the initial time for subsequent use.

[0041] In one or more embodiments, after calculating the pose of each marker in each image data, markers with abnormal pose fluctuations in each image data can be removed to obtain filtered image data.

[0042] Specifically, removing markers with abnormal pose fluctuations from each image data may include, but is not limited to, the following steps S201-S203.

[0043] Step S201. For the same ID number Marker, calculate the median absolute deviation of the pose of the same ID number Marker in each image data.

[0044] For example, 100 image data points are acquired within a sampling period. The ID numbers of the markers in these 100 image data points are divided into Marker1, Marker2, Marker3...Marker20, with each image data point containing 20 markers with different ID numbers (some image data points may have missing markers due to detection errors or occlusion). For Marker1, the pose center positions of all Marker1 markers in the 100 image data points can be calculated. Then, the absolute value of the difference between the pose of Marker1 in each image data point and its pose center position can be calculated to obtain the median absolute deviation of the pose of Marker1 in each image data point.

[0045] Step S202. Markers with the same ID number whose median absolute deviation of pose in each image data exceeds a preset absolute deviation threshold are designated as anomaly markers.

[0046] The preset absolute deviation threshold can be set according to the actual situation.

[0047] Step S203. Remove all abnormal markers from each image data to obtain the filtered image data.

[0048] Based on the example in step S201 above, if the median absolute deviation of the pose of Marker1 in the first image data is higher than a preset absolute deviation threshold, then Marker1 in the first image data is determined to be abnormal and is removed. The same operation is performed on Marker2-Marker20 as on Marker1, thereby removing all Markers with abnormal pose fluctuations from all image data, resulting in filtered image data.

[0049] By removing markers with abnormal pose fluctuations from each image data, it is possible to effectively identify and remove abnormal markers caused by detection errors, occlusion, etc., while retaining high-quality valid markers to improve mapping accuracy during subsequent fusion.

[0050] Step S105. The poses of markers with the same ID number in each image data are fused using the Markley method to obtain the average pose of each ID number marker.

[0051] In one or more embodiments, the poses of markers with the same ID number in each image data are fused using the Markley method, which may include, but is not limited to, the following steps S1051-S1055.

[0052] Step S1051. Calculate the average quaternion and average translation vector of the same ID number Marker in each filtered image data.

[0053] Based on the example in step S201 above, 100 image data points are collected during the sampling period. The markers in these 100 image data points are divided into 20 ID numbers: Marker1, Marker2, Marker3...Marker20. The quaternion average and translation vector average of all markers with the same ID number in these 100 image data points can then be calculated to obtain the quaternion average and translation vector average of the 20 ID number markers (i.e., Marker1, Marker2, Marker3...Marker20).

[0054] In this context, the quaternion of the Marker represents the pose in the Marker's pose, and the translation vector of the Marker represents the position in the Marker's pose.

[0055] Step S1052. Construct a weighted matrix based on the average quaternion of all identical ID markers in each filtered image data.

[0056] Specifically, the quaternion averages of all identical ID markers in the filtered image data can be combined into a 1×N matrix as a weighting matrix.

[0057] Step S1053. Perform eigenvalue decomposition on the weighted matrix to determine the largest feature in the weighted matrix.

[0058] Step S1054. Take the average of the quaternions of the same ID number Marker corresponding to the largest feature as the target quaternion average.

[0059] Based on the example in step S201 above, assuming that the same ID number Marker corresponding to the largest feature is Marker2, the average value of all Marker2 quaternions in the image data can be used as the target quaternion average value.

[0060] Step S1055. Combine the average value of the target quaternion with the average value of the translation vector of each ID number marker in each filtered image data to obtain the average pose of each ID number marker.

[0061] In one or more embodiments, if after calculating the pose of each marker in each image data, markers with abnormal pose fluctuations in each image data are removed, then when fusing the poses of markers with the same ID number in each image data using the Markley method, the poses of corresponding markers with the same ID number in each filtered image data can be fused using the Markley method to obtain the average pose of each ID number marker.

[0062] Step S106. Based on the average pose of each ID number Marker and the average pose of the reference ID number Marker, calculate the coordinate system transformation matrix of the initial coordinate system of each ID number Marker relative to the coordinate system of the reference ID number Marker, and based on the coordinate system transformation matrix, transform the pose of all Markers in each image data to the coordinate system of the reference ID number Marker to obtain the transformed pose of each Marker in each image data.

[0063] In one or more embodiments, before calculating the coordinate system transformation matrix of the initial coordinate system of each ID number marker relative to the coordinate system of the reference ID number marker based on the average pose of each ID number marker and the average pose of the reference ID number marker, it can be determined whether a reference ID number marker has been specified. If a reference ID number marker has been specified, the coordinate system transformation matrix of the initial coordinate system of each ID number marker relative to the coordinate system of the reference ID number marker is calculated based on the average pose of each ID number marker and the average pose of the reference ID number marker. If no reference ID number marker has been specified, an identical ID number marker can be randomly specified as the reference ID number marker, and then the coordinate system transformation matrix of the initial coordinate system of each ID number marker relative to the coordinate system of the reference ID number marker is calculated based on the average pose of each ID number marker and the average pose of the reference ID number marker.

[0064] After calculating the coordinate transformation matrix of each ID marker relative to the coordinate system of the reference ID marker, the poses of all markers in each image data can be transformed to the coordinate system of the reference ID marker based on this transformation matrix, thus obtaining the transformed pose of each marker in each image data. In this way, the reference marker is located at the origin of the coordinate system, and the poses of other markers are relative to the reference marker.

[0065] Step S107. Based on the transformed pose of each marker in each image data, generate marker map data corresponding to each image data and containing the marker pose.

[0066] Step S108. Based on the coordinate system transformation matrix, perform coordinate system transformation on the point cloud map data corresponding to the same sampling time point of each image data to obtain the point cloud map data after coordinate system transformation and alignment of each sampling time point.

[0067] Step S109. Merge the point cloud map data after coordinate system transformation and alignment at each sampling time point with the corresponding Marker map data to obtain the final scene point cloud map data at each sampling time point.

[0068] Specifically, when fusing the point cloud map data after coordinate system transformation and alignment at each sampling time point with the corresponding Marker map data, the point cloud map data after coordinate system transformation and alignment at each sampling time point can be directly linearly superimposed with the corresponding Marker map data to obtain the final scene point cloud map data at each sampling time point.

[0069] This invention acquires laser point cloud data collected by a lidar at each sampling time point within a sampling period, image data collected by a camera at each sampling time point, and angular velocity and acceleration data collected by an inertial measurement unit at each sampling time point. It then fuses the laser point cloud data and image data at the same sampling time point to obtain point cloud map data for each sampling time point. Based on the camera's initial pose, angular velocity data at each sampling time point, and acceleration data at each sampling time point, it determines the camera's pose at each sampling time point. It detects markers in each image data and calculates the pose of each marker in each image data. Using the Markley method, it fuses the poses of markers with the same ID number in each image data to obtain the average pose of each ID number marker. Finally, it calculates the average pose of each ID number marker and a reference ID number marker. The average pose is calculated, and the coordinate system transformation matrix of the initial coordinate system of each ID number Marker relative to the coordinate system of the reference ID number Marker is calculated. Based on the coordinate system transformation matrix, the poses of all Markers in each image data are transformed to the coordinate system of the reference ID number Marker to obtain the transformed pose of each Marker in each image data. Based on the transformed poses of each Marker in each image data, Marker map data corresponding to each image data and containing Marker poses are generated. Based on the coordinate system transformation matrix, the point cloud map data corresponding to the same sampling time point of each image data is transformed to obtain the point cloud map data after coordinate system transformation and alignment at each sampling time point. The point cloud map data after coordinate system transformation and alignment at each sampling time point is fused with the corresponding Marker map data to obtain the final scene point cloud map data at each sampling time point. This invention, through precise coordinate system alignment transformation and fusion modules, achieves for the first time the unified fusion of visual marker maps and laser point cloud maps. It can construct a unified scene map containing geometric information and semantic tags, realizing true multimodal map fusion. This solves the core problem of existing technologies being unable to simultaneously achieve marker map construction and unified fusion with point cloud maps, unifying multi-source information into the same coordinate system and significantly improving the accuracy and stability of robot localization and navigation. Furthermore, this invention does not require markers to appear in the same image as other markers already in the map, making map construction more flexible. It can construct complete maps even in scenes with sparse marker distribution, improving the system's adaptability and practicality. Secondly, the Markley method is used to fuse multiple pose data of each marker into an average pose. By constructing a quaternion weighted matrix and performing eigenvalue decomposition, the characteristics of quaternions on the manifold are fully considered. Compared with the simple averaging or weighted averaging of existing methods, the fusion accuracy is significantly improved, and multiple observations can be fully utilized to improve the accuracy of marker pose estimation.In addition, outlier detection is performed on all observed marker poses using the median absolute deviation, which can effectively identify and remove abnormal markers caused by detection errors, occlusion, etc. Compared with existing methods that lack outlier detection mechanisms, this method significantly improves mapping accuracy and robustness.

[0070] Please see Figure 2 The second aspect of this application provides a mapping apparatus based on laser point cloud and image data, the mapping apparatus comprising: The acquisition unit is used to acquire laser point cloud data collected by the lidar at each sampling time point within the sampling period, image data collected by the camera at each sampling time point, and angular velocity data and acceleration data collected by the inertial measurement unit at each sampling time point, wherein the lidar, the camera, and the inertial measurement unit are all installed on the same mobile acquisition device. The first fusion unit is used to fuse laser point cloud data and image data at the same sampling time point to obtain point cloud map data at each sampling time point; The determination unit is used to determine the pose of the camera at each sampling time point based on the pose of the lidar at each sampling time point and the extrinsic data of the lidar and the camera. The detection and calculation unit is used to detect markers in each image data and calculate the pose of each marker in each image data. The second fusion unit is used to fuse the poses of markers with the same ID number in each image data using the Markley method to obtain the average pose of each ID number marker. The calculation and transformation unit is used to calculate the coordinate system transformation matrix of the initial coordinate system of each ID number Marker relative to the coordinate system of the reference ID number Marker based on the average pose of each ID number Marker and the average pose of the reference ID number Marker, and transform the pose of all Markers in each image data to the coordinate system of the reference ID number Marker based on the coordinate system transformation matrix, so as to obtain the transformed pose of each Marker in each image data. The generation unit is used to generate Marker map data corresponding to each image data and containing the Marker pose based on the transformed pose of each Marker in each image data. The transformation unit is used to perform coordinate system transformation on the point cloud map data corresponding to the same sampling time point of each image data based on the coordinate system transformation matrix, so as to obtain the point cloud map data after coordinate system transformation and alignment of each sampling time point. The third fusion unit is used to fuse the point cloud map data after coordinate system transformation and alignment at each sampling time point with the corresponding Marker map data to obtain the final scene point cloud map data at each sampling time point.

[0071] The working process, working details and technical effects of the mapping device based on laser point cloud and image data provided in the second aspect of this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0072] Please see Figure 3 The third aspect of this application provides an electronic device, including a memory, a processor, and a transceiver that are sequentially and communicatively connected, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the mapping method based on laser point cloud and image data as described in the first aspect of the application.

[0073] Specifically, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or last-in-first-out (FILO) memory, etc.; the processor may not be limited to microprocessors of the STM32F105 series, ARM (Advanced RISC Machines), x86 architecture processors, or processors with integrated NPU (neural-network processing units); the transceiver may be, but is not limited to, WiFi (Wireless Fidelity) wireless transceivers, Bluetooth wireless transceivers, General Packet Radio Service (GPRS) wireless transceivers, ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard), 3G transceivers, 4G transceivers, and / or 5G transceivers, etc.

[0074] This fourth aspect of the embodiment provides a computer-readable storage medium storing instructions for a mapping method based on laser point clouds and image data as described in the first aspect of the embodiment. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, perform the mapping method based on laser point clouds and image data as described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0075] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the mapping method based on laser point cloud and image data as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0076] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A mapping method based on laser point cloud and image data, characterized in that, include: The system acquires laser point cloud data collected by the lidar at each sampling time point within the sampling period, image data collected by the camera at each sampling time point, and angular velocity and acceleration data collected by the inertial measurement unit at each sampling time point, wherein the lidar, the camera, and the inertial measurement unit are all mounted on the same mobile acquisition device. By fusing laser point cloud data and image data at the same sampling time point, point cloud map data at each sampling time point can be obtained. Based on the pose of the lidar at each sampling time point and the extrinsic data of the lidar and the camera, the pose of the camera at each sampling time point is determined. Detect markers in each image data and calculate the pose of each marker in each image data; The poses of markers with the same ID number in each image data are fused using the Markley method to obtain the average pose of each ID number marker. Based on the average pose of each ID number Marker and the average pose of the reference ID number Marker, calculate the coordinate system transformation matrix of the initial coordinate system of each ID number Marker relative to the coordinate system of the reference ID number Marker, and based on the coordinate system transformation matrix, transform the pose of all Markers in each image data to the coordinate system of the reference ID number Marker to obtain the transformed pose of each Marker in each image data. Based on the transformed pose of each marker in each image data, generate marker map data corresponding to each image data and containing the marker pose; Based on the coordinate system transformation matrix, the point cloud map data corresponding to the same sampling time point of each image data is transformed to obtain the point cloud map data after coordinate system transformation and alignment of each sampling time point. The point cloud map data after coordinate system transformation and alignment at each sampling time point is fused with the corresponding Marker map data to obtain the final scene point cloud map data at each sampling time point.

2. The mapping method based on laser point cloud and image data according to claim 1, characterized in that, After calculating the pose of each marker in each image data, the method further includes: Transform the pose of each marker in each image data to the global coordinate system at the initial time; Remove markers with abnormal pose fluctuations from each image data to obtain the filtered image data; Correspondingly, the poses of markers with the same ID number in each image data are fused using the Markley method to obtain the average pose of each ID number marker, including: The Markley method is used to fuse the poses of markers with the same ID number in each filtered image data to obtain the average pose of each ID number marker.

3. The mapping method based on laser point cloud and image data according to claim 2, characterized in that, After removing markers with abnormal pose fluctuations from each image dataset, the filtered image dataset is obtained, including: For markers with the same ID number, calculate the median absolute deviation of the pose of markers with the same ID number in each image data. Markers with the same ID number whose median absolute deviation of pose in each image data exceeds a preset absolute deviation threshold are designated as abnormal markers. Remove all abnormal markers from each image data to obtain the filtered image data.

4. The mapping method based on laser point cloud and image data according to claim 2, characterized in that, The Markley method is used to fuse the poses of markers with the same ID number in each filtered image data to obtain the average pose of each ID number marker, including: Calculate the average quaternion and average translation vector of the same ID number marker in each filtered image data; A weighted matrix is ​​constructed based on the average quaternion of all identical ID markers in each filtered image data. Perform eigenvalue decomposition on the weighted matrix to determine the largest feature in the weighted matrix; The average of the quaternions of the Markers with the same ID number corresponding to the largest feature is taken as the target quaternion average. The average value of the target quaternion is combined with the average value of the translation vector of each ID number marker in each filtered image data to obtain the average pose of each ID number marker.

5. The mapping method based on laser point cloud and image data according to claim 1, characterized in that, Before calculating the coordinate system transformation matrix of each ID number marker relative to the coordinate system of the reference ID number marker based on the average pose of each ID number marker and the average pose of the reference ID number marker, the method further includes: Determine whether a reference ID number (Marker) has been specified; If no reference ID number Marker is specified, an ID number Marker will be randomly assigned as the reference ID number Marker.

6. The mapping method based on laser point cloud and image data according to claim 1, characterized in that, Calculate the pose of each marker in each image data, including: The pose of each marker in each image data relative to the camera at the corresponding sampling time point is calculated using the PnP algorithm. Based on the camera pose at each sampling time point and the pose of each marker in each image data relative to the camera at the corresponding sampling time point, the pose of each marker in each image data is calculated.

7. The mapping method based on laser point cloud and image data according to claim 1, characterized in that, By fusing laser point cloud data and image data at the same sampling time point, point cloud map data for each sampling time point are obtained, including: The laser visual SLAM algorithm is used to fuse laser point cloud data and image data at the same sampling time point to obtain point cloud map data at each sampling time point.

8. A mapping device based on laser point cloud and image data, characterized in that, include: The acquisition unit is used to acquire laser point cloud data collected by the lidar at each sampling time point within the sampling period, image data collected by the camera at each sampling time point, and angular velocity data and acceleration data collected by the inertial measurement unit at each sampling time point, wherein the lidar, the camera, and the inertial measurement unit are all installed on the same mobile acquisition device. The first fusion unit is used to fuse laser point cloud data and image data at the same sampling time point to obtain point cloud map data at each sampling time point; The determination unit is used to determine the pose of the camera at each sampling time point based on the pose of the lidar at each sampling time point and the extrinsic data of the lidar and the camera. The detection and calculation unit is used to detect markers in each image data and calculate the pose of each marker in each image data. The second fusion unit is used to fuse the poses of markers with the same ID number in each image data using the Markley method to obtain the average pose of each ID number marker. The calculation and transformation unit is used to calculate the coordinate system transformation matrix of the initial coordinate system of each ID number Marker relative to the coordinate system of the reference ID number Marker based on the average pose of each ID number Marker and the average pose of the reference ID number Marker, and transform the pose of all Markers in each image data to the coordinate system of the reference ID number Marker based on the coordinate system transformation matrix, so as to obtain the transformed pose of each Marker in each image data. The generation unit is used to generate Marker map data corresponding to each image data and containing the Marker pose based on the transformed pose of each Marker in each image data. The transformation unit is used to perform coordinate system transformation on the point cloud map data corresponding to the same sampling time point of each image data based on the coordinate system transformation matrix, so as to obtain the point cloud map data after coordinate system transformation and alignment of each sampling time point. The third fusion unit is used to fuse the point cloud map data after coordinate system transformation and alignment at each sampling time point with the corresponding Marker map data to obtain the final scene point cloud map data at each sampling time point.

9. An electronic device, characterized in that, The system includes a memory, a processor, and a transceiver that are sequentially and communicatively connected. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the mapping method based on laser point cloud and image data as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the mapping method based on laser point cloud and image data as described in any one of claims 1 to 7.

Images