Vision-aided laser-imu misalignment calibration method, mapping method and system

Abstract

The application relates to the field of surveying and mapping, in particular to a laser-IMU installation deviation calibration method, a surveying and mapping method and a system based on visual assistance, wherein a plurality of images under a plurality of laser radar calibration conditions are processed according to camera parameters and camera-IMU correlation parameters, laser radar-IMU installation deviation is obtained, manual operation is avoided, high-precision and high-efficiency deviation calibration is realized, the relative spatial relationship of the laser radar-IMU is adjusted according to the laser radar-IMU installation deviation, the accurate matching and installation of the laser radar and the IMU are ensured, the frame corresponding relationship between the IMU data and the point cloud data is constructed according to the dynamic characteristics of a surveying and mapping path, the most matched point cloud data points of each IMU data position point on the surveying and mapping path are screened, a local point cloud map corresponding to the position point is constructed, and therefore the spatial map of a target region where the surveying and mapping path is located is integrated and constructed. The application can reduce data omission and improve the accuracy and reliability of spatial surveying and positioning through local point cloud data processing and map construction.

Description

Technical Field

[0001] This invention relates to the field of surveying and mapping, and in particular to a visually assisted laser-IMU installation deviation calibration method, surveying method, and system. Background Technology

[0002] With the development of automation technology, there is a significant demand for spatial mapping and positioning. To ensure the accurate and safe operation of mobile bodies such as robots in complex 3D environments, inertial measurement units (IMUs) are combined with lidar to form simultaneous laser localization and mapping (SLAM) technology for real-time 3D spatial mapping and positioning. To meet the requirements of integrated recognition, the mobile body simultaneously collects panoramic point cloud data of the target area while driving the IMU and lidar to construct a panoramic map. While the above methods can ensure the breadth and timeliness of the panoramic map, collecting panoramic point cloud data at the same time can easily lead to the omission of some point cloud data. This results in the global map not being able to accurately represent the state of objects within the spatial area, reducing the accuracy and reliability of spatial mapping and positioning.

[0003] Furthermore, the spatial installation layout of IMU and LiDAR directly affects the accuracy of point cloud data. Installation deviations in IMU and LiDAR can interfere with the data, and this interference accumulates as the mapping process progresses. Therefore, manual operation and calibration of the IMU and LiDAR are usually required before spatial mapping and positioning. However, manual operation is inefficient, inaccurate, and unreliable, failing to meet high-precision calibration requirements. Therefore, implementing automatic IMU and LiDAR installation and calibration, as well as local mapping and global integration, in IMU and LiDAR spatial mapping operations is crucial for improving the efficiency, accuracy, and level of detail in spatial mapping. Summary of the Invention

[0004] Considering that installation deviations between the IMU and the lidar can interfere with point cloud data, manual installation correction alone cannot completely correct these deviations and effectively eliminate interference. Furthermore, the global unified point cloud data acquisition and map construction of the target area is prone to missing some point cloud data, failing to accurately represent the three-dimensional situation within the spatial area and reducing the accuracy and reliability of spatial mapping and positioning. This invention provides a vision-assisted lidar-IMU installation deviation calibration method, which includes the following steps:

[0005] S1: When the camera associated with the lidar is a first distance away from the reference object, acquire a first image without laser illumination and a first image with laser illumination taken of the reference object;

[0006] S2: When the camera is separated from the reference object by a second distance, acquire a second image without laser illumination and a second image with laser illumination taken of the reference object;

[0007] S3: Process the first image without laser illumination, the first image with laser illumination, the second image without laser illumination, and the second image with laser illumination based on the camera's internal parameters and the camera-IMU associated external parameters to obtain the lidar-IMU installation deviation; wherein, the lidar-IMU installation deviation includes the pitch installation deviation angle and the heading installation deviation angle between the lidar and the IMU.

[0008] On the other hand, the present invention provides a surveying method, the method comprising the following steps:

[0009] S100: Adjust the relative spatial relationship of the lidar-IMU according to the installation deviation of the lidar-IMU; wherein, the installation deviation of the lidar-IMU is obtained by the above-mentioned vision-assisted lidar-IMU installation deviation calibration method.

[0010] S200: Based on the dynamic characteristics of the mapping path, construct a one-to-many correspondence between IMU data and point cloud data frames from the lidar; filter the point cloud data frames that best match the corresponding location points of the mapping path from all point cloud data corresponding to the same IMU data, thereby constructing a local point cloud map corresponding to the location points.

[0011] S300: Based on the spatial layout of all location points within the surveying path, integrate the local point cloud maps corresponding to all location points to construct a spatial map of the target area where the surveying path is located.

[0012] Preferably, in S100, the relative spatial relationship between the lidar and the IMU is adjusted according to the installation deviation of the lidar-IMU, specifically as follows:

[0013] The installation correction path of the lidar-IMU is determined based on the adjustable parameters of the single relative motion between the lidar and the IMU, the pitch installation deviation angle and the heading installation deviation angle between the lidar and the IMU.

[0014] According to the lidar-IMU installation correction path, adjust the spatial position of at least one of the lidar and IMU to change the relative spatial relationship between the lidar and IMU until the pitch installation deviation angle and the heading installation deviation angle are eliminated.

[0015] Preferably, in step S200, based on the dynamic characteristics of the mapping path, a one-to-many correspondence is constructed between IMU data and point cloud data frames from the lidar; the point cloud data frames that best match the corresponding location points on the mapping path are selected from all point cloud data corresponding to the same IMU data, thereby constructing a local point cloud map corresponding to the location points, specifically:

[0016] The motion dynamics of the moving body along the surveying path in the target area are obtained; wherein, the motion dynamics include motion speed variation characteristics and motion trajectory curvature variation characteristics; based on the motion dynamics, several position points are marked on the surveying path;

[0017] Based on the timing of the arrival of the mobile body at the several location points, a one-to-many correspondence between IMU data and point cloud data frames from the lidar is constructed; wherein, whenever the mobile body arrives at a location point, the IMU generates one IMU data and the lidar forms multiple point cloud data frames in different azimuth ranges relative to the location point.

[0018] The layout features of face points and corner points of all point cloud data frames corresponding to the same IMU data are obtained. Based on the layout features, the spatial positioning error of each point cloud data frame is estimated. The point cloud data frame with the smallest spatial positioning error is selected as the point cloud data frame that best matches the location point where the IMU data is generated, thereby constructing a local point cloud map corresponding to the location point.

[0019] Preferably, in step S300, based on the spatial layout of all location points within the surveying path, the local point cloud maps corresponding to all location points are integrated to construct a spatial map of the target area where the surveying path is located, specifically as follows:

[0020] Based on the spatial layout and positional relationship of all location points within the surveying path, the local point cloud maps corresponding to all location points are spatially arranged to determine the overlapping point cloud data between any two spatially adjacent local point cloud maps.

[0021] Redundant point cloud data is removed from the overlapping point cloud data and adjacent point cloud data are stitched together to obtain a global point cloud map formed by all local point cloud maps, thereby constructing a spatial map of the target area where the surveying path is located.

[0022] On the other hand, the present invention provides a surveying system, the system comprising the following modules:

[0023] The spatial adjustment module is used to adjust the relative spatial relationship of the lidar-IMU according to the installation deviation of the lidar-IMU; wherein, the installation deviation of the lidar-IMU is obtained by the above-mentioned vision-assisted lidar-IMU installation deviation calibration method.

[0024] The point cloud data preprocessing module is used to construct a one-to-many correspondence between IMU data and point cloud data frames from lidar based on the dynamic characteristics of the mapping path.

[0025] The local point cloud map construction module is used to filter the point cloud data frames that best match the corresponding location points of the mapping path from all point cloud data corresponding to the same IMU data, so as to construct the local point cloud map corresponding to the location points.

[0026] The spatial map construction module is used to integrate the local point cloud maps corresponding to all location points according to the spatial layout of all location points within the surveying path, and construct a spatial map of the target area where the surveying path is located.

[0027] Preferably, the spatial adjustment module is used to adjust the relative spatial relationship of the lidar-IMU according to the installation deviation of the lidar-IMU, specifically as follows:

[0028] The installation correction path of the lidar-IMU is determined based on the adjustable parameters of the single relative motion between the lidar and the IMU, the pitch installation deviation angle and the heading installation deviation angle between the lidar and the IMU.

[0029] According to the lidar-IMU installation correction path, adjust the spatial position of at least one of the lidar and IMU to change the relative spatial relationship between the lidar and IMU until the pitch installation deviation angle and the heading installation deviation angle are eliminated.

[0030] Preferably, the point cloud data preprocessing module is used to construct a one-to-many correspondence between IMU data and point cloud data frames from the lidar based on the dynamic characteristics of the mapping path, specifically:

[0031] The motion dynamics of the moving body along the surveying path in the target area are obtained; wherein, the motion dynamics include motion speed variation characteristics and motion trajectory curvature variation characteristics; based on the motion dynamics, several position points are marked on the surveying path;

[0032] Based on the timing of the movement reaching the several location points, a one-to-many correspondence is constructed between IMU data and point cloud data frames from the lidar; wherein, whenever the movement reaches a location point, the IMU generates IMU data and the lidar forms multiple point cloud data frames in different azimuth ranges relative to the location point.

[0033] Preferably, the local point cloud map construction module is used to filter the point cloud data frames that best match the corresponding location points of the mapping path from all point cloud data corresponding to the same IMU data, thereby constructing a local point cloud map corresponding to the location points, specifically:

[0034] The layout features of face points and corner points of all point cloud data frames corresponding to the same IMU data are obtained. Based on the layout features, the spatial positioning error of each point cloud data frame is estimated. The point cloud data frame with the smallest spatial positioning error is selected as the point cloud data frame that best matches the location point where the IMU data is generated, thereby constructing a local point cloud map corresponding to the location point.

[0035] Preferably, the spatial map construction module is used to integrate the local point cloud maps corresponding to all location points according to the spatial layout of all location points within the surveying path, and construct a spatial map of the target area where the surveying path is located, specifically:

[0036] Based on the spatial layout and positional relationship of all location points within the surveying path, the local point cloud maps corresponding to all location points are spatially arranged to determine the overlapping point cloud data between any two spatially adjacent local point cloud maps.

[0037] Redundant point cloud data is removed from the overlapping point cloud data and adjacent point cloud data are stitched together to obtain a global point cloud map formed by all local point cloud maps, thereby constructing a spatial map of the target area where the surveying path is located.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] This invention presents a vision-assisted laser-IMU installation deviation calibration method. It processes several images under various laser radar calibration conditions based on camera parameters and camera-IMU correlation parameters to obtain the laser radar-IMU installation deviation, avoiding manual operation and achieving high-precision and efficient deviation calibration. Furthermore, the mapping method and system within this invention adjust the relative spatial relationship between the laser radar and IMU based on the installation deviation, ensuring precise matching and installation between the laser radar and IMU. Based on the dynamic characteristics of the mapping path, it constructs a correspondence between IMU data and point cloud data frames, and selects the most matching point cloud data points for each IMU data location on the mapping path, thereby constructing a local point cloud map corresponding to each location point. Finally, it integrates all local point cloud maps corresponding to all location points to construct a spatial map of the target area within the mapping path. By correcting the installation deviation, the accuracy of the point cloud data at the acquisition level is ensured; furthermore, by processing local point cloud data and constructing maps, data omissions are reduced, improving the accuracy and reliability of spatial mapping and positioning. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0041] Figure 1 This is a flowchart of the vision-assisted laser-IMU installation deviation calibration method provided by the present invention.

[0042] Figure 2 It is a device used for laser-IMU installation deviation calibration.

[0043] Figure 3 This is a flowchart of the surveying method provided by the present invention.

[0044] Figure 4 It refers to the device layout of the moving parts.

[0045] Figure 5 This is the installation and calibration path for the lidar-IMU.

[0046] Figure 6 It refers to the location points distribution of the survey path.

[0047] Figure 7 It is the layout of face points and corner points in a point cloud data frame.

[0048] Figure 8 It is a local point cloud map corresponding to a location point.

[0049] Figure 9 It is an integrated global point cloud map.

[0050] Figure 10 This is a structural diagram of the surveying system provided by the present invention. Detailed Implementation

[0051] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and not for limiting the present invention. Furthermore, it should be noted that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, not all structures. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present invention.

[0052] The terms "comprising" and "having," and any variations thereof, used in this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0053] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0054] Please see Figure 1 As shown, this invention provides a vision-assisted laser-IMU installation deviation calibration method, which includes the following steps:

[0055] S1: When the camera associated with the lidar is a first distance away from the reference object, acquire a first image without laser illumination and a first image with laser illumination taken of the reference object;

[0056] S2: When the camera is separated from the reference object by a second distance, acquire a second image without laser illumination and a second image with laser illumination taken of the reference object;

[0057] S3: Process the first image without laser illumination, the first image with laser illumination, the second image without laser illumination, and the second image with laser illumination based on the camera's internal parameters and the camera-IMU associated external parameters to obtain the lidar-IMU installation deviation; wherein, the lidar-IMU installation deviation includes the pitch installation deviation angle and the heading installation deviation angle between the lidar and the IMU.

[0058] Please see Figure 2 A reference object and a robotic arm are placed on the base. A receiver and a camera are mounted on the free end of the robotic arm. The receiver is a combination of a lidar and an IMU (Integrated Device Unit), and the lidar and IMU are initially installed together in a relative spatial relationship before being positioned on the free end of the robotic arm. The camera is mounted on the receiver and provides visual assistance for correcting installation deviations of the lidar and IMU. The reference object can be, but is not limited to, a reference object with a grid pattern or a black-and-white checkerboard pattern, serving as a reference marker for camera visual recognition. The lidar emits invisible light during actual point cloud data acquisition and visible light during installation deviation correction (e.g., ...). Figure 2(The laser shown). After the base is leveled and corrected, the receiver is moved by a robotic arm to change the distance between the receiver and the reference object. To ensure effective fusion of the IMU data generated by the IMU and the point cloud data collected by the lidar, and to ensure the attitude representation of the point cloud data by the IMU data, the IMU and lidar need to be installed according to a preset relative spatial relationship. In actual installation, due to various factors, there will inevitably be installation deviations in pitch and yaw angles between the IMU and lidar. If these installation deviations are not corrected before the surveying operation, interference errors will accumulate continuously during the fusion of IMU data and point cloud data, affecting the accuracy of the surveying operation.

[0059] The specific steps for laser-assisted installation deviation correction of the IMU and lidar can be as follows:

[0060] After leveling the reference object and aiming the camera vertically at it, first set the camera to a first distance (e.g., 0.5m) from the reference object, and take the first image without laser illumination and the first image with laser illumination when the lidar is not emitting visible light and when it is emitting visible light, respectively; then adjust the camera to a second distance (e.g., 1.5m) from the reference object, and take the second image without laser illumination and the second image with laser illumination when the lidar is not emitting visible light and when it is emitting visible light, respectively.

[0061] A world coordinate system is established using a corner point of a reference object as the origin, with the X-axis positive to the right, the Y-axis positive downwards, and the Z-axis positive forwards. Given the grid size of the reference object's mesh pattern or checkerboard pattern, the world coordinates of all corner points of the reference object are calculated. The pose of the camera at a first distance from the reference object in the world coordinate system is then calculated based on the first image without laser illumination. ,in For rotation matrix, Let be the translation vector. Based on the first image without laser illumination and the first image with laser illumination, determine the light spot formed on the reference object by the emitted visible light rays. Calculate the above light spots Corresponding pixel coordinates The above light spots Corresponding pixel coordinates Transform camera parameters (such as focal length, principal point coordinates, distortion coefficients, etc.) into normalized planar coordinates. Using the camera's pose and the above normalized plane coordinates Calculate the light spot Coordinates in the world coordinate system The camera ray vector in the world coordinate system is constructed according to the following formula:

[0062] , c

[0063] In the above formula, Indicates the optical center and spot of the camera The projection of the direction vector formed by the connection into the world coordinate system.

[0064] Calculate the world coordinates of the light point using the following formula. :

[0065]

[0066] In the above formula, express The third row of data, express The third row of data.

[0067] Similarly, based on the above process, the pose of the camera in the world coordinate system at a second distance from the reference object is calculated for the second image without laser illumination and the second image with laser illumination. ,in For rotation matrix, The translation vector, and the spot formed by the emitted visible light rays on the reference object. Calculate the above light spots Coordinates in the world coordinate system .

[0068] Calculate the light spot according to the following formula. and coordinates in the camera coordinate system and :

[0069]

[0070]

[0071] In the above formula, and They represent and The inverse matrix.

[0072] The direction vector of the light ray in the camera coordinate system is calculated using the following formula:

[0073]

[0074] In the above formula, This represents the direction vector of the light ray in the camera coordinate system.

[0075] The direction vector of the ray in the IMU coordinate system is calculated using the following formula:

[0076]

[0077] In the above formula, This represents the direction vector of the light ray in the IMU coordinate system. This represents the rotation matrix from the position in the camera coordinate system to the position in the world coordinate system (i.e., the camera-IMU associated extrinsic parameter).

[0078] The pitch and heading installation deviation angles between the lidar and the IMU are calculated using the following formulas:

[0079]

[0080]

[0081] In the above formula, Δθ and ΔΨ represent the pitch installation deviation angle and heading installation deviation angle between the lidar and the IMU, respectively. , , They represent direction vectors respectively. Components in the X, Y, and Z directions.

[0082] Through the above process, the camera provides visual assistance to automatically and accurately calculate the pitch and heading installation deviation angles between the lidar and the IMU. This is to facilitate subsequent calibration of the installation spatial position relationship between the lidar and the IMU without manual operation, resulting in high calibration efficiency, effectively avoiding human error, and achieving high calibration accuracy.

[0083] Please see Figure 3 As shown, the present invention provides a surveying method, which includes the following steps:

[0084] S100: Adjust the relative spatial relationship of the lidar-IMU according to the installation deviation of the lidar-IMU; wherein, the installation deviation of the lidar-IMU is obtained by the above-mentioned vision-assisted lidar-IMU installation deviation calibration method.

[0085] Furthermore, in S100, the relative spatial relationship between the lidar and the IMU is adjusted according to the installation deviation of the lidar-IMU, specifically as follows:

[0086] The installation correction path of the lidar-IMU is determined based on the adjustable parameters of the single relative motion between the lidar and the IMU, the pitch installation deviation angle and the heading installation deviation angle between the lidar and the IMU.

[0087] According to the lidar-IMU installation calibration path, adjust the spatial position of at least one of the lidar and IMU to change the relative spatial relationship between the lidar and IMU until the pitch installation deviation angle and the heading installation deviation angle are eliminated.

[0088] Please see Figure 4 Spatial mapping of the target area is performed using a mobile vehicle such as a car or robot that moves along a mapping path within the target area, conducting laser-IMU mapping during the movement. The mobile vehicle is equipped with measuring devices such as lidar and inertial measurement unit (IMU).

[0089] Before the mobile unit begins mapping operations, the lidar and IMU are first installed in their respective initial positions. At this point, top-view assistance is used to determine the pitch and yaw installation deviation angles between the lidar and IMU, calibrating the installation angle deviations of the lidar and IMU in the pitch and yaw directions at their current initial positions. In reality, the lidar and IMU are in an adjustable position on the mobile unit; either the lidar or the IMU can change its position, thus achieving the relative pitch and yaw angle deviations between them. In practice, to prevent the lidar and / or IMU from moving too much during a single adjustment and failing to accurately move to the target position, both the lidar and / or IMU are set with a maximum displacement and a single changeable spatial angle. Thus, during the relative movement of the lidar and IMU, there are adjustable parameters for the single relative movement. These adjustable parameters may include, but are not limited to, the maximum displacement change and the maximum spatial angle change during a single relative movement change. Combined with the pitch and heading installation deviation angles between the lidar and IMU, a lidar-IMU installation correction path is determined, so that the pitch and heading installation deviation angles are gradually eliminated during the relative movement of the lidar and IMU.

[0090] Please refer to Figure 5 For cases where only the IMU moves, a correction path is set for the IMU. As the IMU moves from its initial position to the target position along the correction path, the pitch and yaw installation deviation angles between the lidar and the IMU gradually decrease. Alternatively, the lidar can be controlled to move along the correction path only, or both the lidar and the IMU can be controlled to move along their respective correction paths simultaneously. The process and principle of these two motion control methods are the same as those described above, which will not be repeated here. By setting correction paths for the lidar and / or IMU, a path reference benchmark can be provided to eliminate installation deviations between them, ensuring the accurate and effective fusion of subsequent IMU data and point cloud data.

[0091] S200: Based on the dynamic characteristics of the mapping path, construct a one-to-many correspondence between IMU data and point cloud data frames from LiDAR; select the point cloud data frames that best match the corresponding location points of the mapping path from all point cloud data corresponding to the same IMU data, and construct a local point cloud map corresponding to the location points.

[0092] Furthermore, in S200, based on the dynamic characteristics of the mapping path, a one-to-many correspondence is constructed between IMU data and point cloud data frames from the lidar; from all point cloud data corresponding to the same IMU data, the point cloud data frame that best matches the corresponding location point on the mapping path is selected, thereby constructing a local point cloud map corresponding to the location point, specifically:

[0093] Acquire the dynamic characteristics of the moving object's motion along the surveying path in the target area; the dynamic characteristics include the characteristics of changes in motion speed and the characteristics of changes in the curvature of the motion trajectory; based on the dynamic characteristics, mark several position points on the surveying path;

[0094] Based on the arrival time sequence of the moving body at several locations, a one-to-many correspondence between IMU data and point cloud data frames from LiDAR is constructed; wherein, whenever the moving body arrives at a location, the IMU generates one IMU data and the LiDAR forms multiple point cloud data frames with different azimuth ranges relative to the location.

[0095] The layout features of face points and corner points of all point cloud data frames corresponding to the same IMU data are obtained. Based on the layout features, the spatial positioning error of each point cloud data frame is estimated. The point cloud data frame with the smallest spatial positioning error is selected as the point cloud data frame that best matches the location of the generated IMU data, and a local point cloud map corresponding to the location point is constructed in this way.

[0096] The moving object collects point cloud data using its own lidar while moving within the target area. Considering that lidar operates in a scanning mode—meaning that whenever the moving object reaches a point within the target area, the lidar scans and detects the surrounding space using that point as a rotation base to generate corresponding point cloud data—if the lidar scans and generates point cloud data simultaneously with each point the moving object reaches a point, there may be significant overlap between the point cloud data of adjacent points. To avoid this significant overlap and reduce the lidar's scanning workload, the characteristics of the moving object's velocity and trajectory curvature changes along the mapping path within the target area are obtained. Based on these characteristics, the change in the lidar's scanning detection range along the mapping path is determined, and several points are then marked on the mapping path. Please refer to [link to relevant documentation]. Figure 6 There are several discrete location points along the surveying path. When the moving body moves along the surveying path and the surveying direction, the lidar will scan and detect the surrounding space and generate multiple point cloud data frames whenever it reaches a location point.

[0097] As previously described, when the moving body moves along the mapping path and the mapping direction, the LiDAR generates multiple point cloud data frames for each location point, and the IMU also generates one IMU data frame at each location point. Based on the time sequence of the moving body reaching each location point during its movement along the mapping path, a one-to-many correspondence is established between the one IMU data frame generated at each location point and the multiple point cloud data frames. This allows for centralized filtering and processing of all point cloud data frames generated at each location point. Please refer to [link to relevant documentation]. Figure 7 The shapes of objects in the space surrounding each location point are different. For example, there are planar wall parts and angled wall corner parts. The corresponding point cloud data frames contain two different point cloud sets: surface points and corner points. By spatially distinguishing and locating the surface points and corner points in the point cloud data frames, an accurate point cloud data foundation is provided for the subsequent construction of the local point cloud map corresponding to the location point.

[0098] Each IMU data point corresponds to multiple point cloud data frames. The number and location of facets and corners in different point cloud data frames vary, resulting in different accuracy in representing the actual object distribution in the area surrounding the location point for each point cloud data frame; that is, the spatial positioning error of each point cloud data point also differs. For example, based on the spatial distribution of facets and corners in each of the point cloud data frames corresponding to the same IMU data, the average spatial positioning deviation of each point cloud data frame for all objects in the surrounding area is determined. The point cloud data frame with the smallest average spatial positioning deviation is then used as the moving IMU to generate the best-matching point cloud data frame for the location point corresponding to the aforementioned IMU data. This best-matching point cloud data frame is then transformed to obtain the local point cloud map corresponding to the location point. Please refer to [link to relevant documentation]. Figure 8 This is a local point cloud map obtained by converting the most matching point cloud data frame for a given location point. The local point cloud map accurately locates the distribution of object planes and object surfaces in the space surrounding the location point, providing a basis for subsequent integration of the spatial map of the entire target area.

[0099] S300: Based on the spatial layout of all location points within the surveying path, integrate the local point cloud maps corresponding to all location points to construct a spatial map of the target area where the surveying path is located.

[0100] Furthermore, in S300, based on the spatial layout of all location points within the surveying path, the local point cloud maps corresponding to all location points are integrated to construct a spatial map of the target area where the surveying path is located, specifically:

[0101] Based on the spatial layout and positional relationship of all locations within the survey path, the local point cloud maps corresponding to all locations are spatially arranged to determine the overlapping point cloud data between any two spatially adjacent local point cloud maps.

[0102] Redundant point cloud data is removed from overlapping point cloud data and adjacent point cloud data are stitched together to obtain a global point cloud map formed by all local point cloud maps, thereby constructing a spatial map of the target area where the survey path is located.

[0103] As described above, all locations along the mapping path within the target area correspond to local point cloud maps, which reflect the spatial positioning data of the area surrounding each location. There is spatial overlap between the local point cloud maps of adjacent locations, resulting in overlapping point cloud data between them. This overlapping point cloud data may represent the same object in a similar way. To avoid this overlapping point cloud data affecting the overall positioning accuracy of the target area, the local point cloud maps corresponding to all locations are first spatially arranged to determine the overlapping point cloud data between any two spatially adjacent local point cloud maps. Then, redundant point cloud data is removed (e.g., duplicate point cloud data is removed) and adjacent point cloud data is stitched together (e.g., two adjacent point clouds with an actual distance less than a threshold are merged into one point cloud), thereby constructing a spatial map of the target area where the mapping path is located. Please refer to [link to relevant documentation]. Figure 9 The global point cloud map obtained by integrating multiple local point cloud maps completely and accurately represents the global spatial positioning of the entire target area.

[0104] Please see Figure 10 As shown, the present invention provides a surveying system, which includes the following modules:

[0105] The spatial adjustment module is used to adjust the relative spatial relationship of the lidar-IMU according to the installation deviation of the lidar-IMU; wherein, the installation deviation of the lidar-IMU is obtained by the above-mentioned vision-assisted lidar-IMU installation deviation calibration method.

[0106] The point cloud data preprocessing module is used to construct a one-to-many correspondence between IMU data and point cloud data frames from lidar based on the dynamic characteristics of the mapping path.

[0107] The local point cloud map construction module is used to filter the point cloud data frames that best match the corresponding location points of the mapping path from all point cloud data corresponding to the same IMU data, so as to construct the local point cloud map corresponding to the location points.

[0108] The spatial map construction module is used to integrate the local point cloud maps corresponding to all location points according to the spatial layout of all location points within the surveying path, and construct a spatial map of the target area where the surveying path is located.

[0109] Furthermore, the spatial adjustment module is used to adjust the relative spatial relationship of the lidar-IMU based on the installation deviation of the lidar-IMU, specifically as follows:

[0110] The installation correction path of the lidar-IMU is determined based on the adjustable parameters of the single relative motion between the lidar and the IMU, the pitch installation deviation angle and the heading installation deviation angle between the lidar and the IMU.

[0111] According to the lidar-IMU installation calibration path, adjust the spatial position of at least one of the lidar and IMU to change the relative spatial relationship between the lidar and IMU until the pitch installation deviation angle and the heading installation deviation angle are eliminated.

[0112] Furthermore, the point cloud data preprocessing module is used to construct a one-to-many correspondence between IMU data and point cloud data frames from the lidar based on the dynamic characteristics of the mapping path, specifically:

[0113] Acquire the dynamic characteristics of the moving object's motion along the surveying path in the target area; the dynamic characteristics include the characteristics of changes in motion speed and the characteristics of changes in the curvature of the motion trajectory; based on the dynamic characteristics, mark several position points on the surveying path;

[0114] Based on the arrival time sequence of the moving body at several locations, a one-to-many correspondence is constructed between IMU data and point cloud data frames from LiDAR; wherein, whenever the moving body arrives at a location, the IMU generates one IMU data and the LiDAR forms multiple point cloud data frames in different azimuth ranges relative to the location.

[0115] Furthermore, the local point cloud map construction module is used to filter point cloud data frames that best match the corresponding location points on the mapping path from all point cloud data corresponding to the same IMU data, thereby constructing a local point cloud map corresponding to the location points, specifically:

[0116] The layout features of face points and corner points of all point cloud data frames corresponding to the same IMU data are obtained. Based on the layout features, the spatial positioning error of each point cloud data frame is estimated. The point cloud data frame with the smallest spatial positioning error is selected as the point cloud data frame that best matches the location of the generated IMU data, and a local point cloud map corresponding to the location point is constructed in this way.

[0117] Furthermore, the spatial map construction module is used to integrate the local point cloud maps corresponding to all location points based on the spatial layout of all location points within the surveying path, and construct a spatial map of the target area where the surveying path is located, specifically:

[0118] Based on the spatial layout and positional relationship of all locations within the survey path, the local point cloud maps corresponding to all locations are spatially arranged to determine the overlapping point cloud data between any two spatially adjacent local point cloud maps.

[0119] Redundant point cloud data is removed from overlapping point cloud data and adjacent point cloud data are stitched together to obtain a global point cloud map formed by all local point cloud maps, thereby constructing a spatial map of the target area where the survey path is located.

[0120] The operation and effects of the surveying system of the present invention are consistent with those of the surveying method described above, and the surveying system will not be described again here.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Other embodiments may also be used. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vision-assisted laser-IMU installation deviation calibration method, characterized in that, The method includes the following steps: S1: When the camera associated with the lidar is a first distance away from the reference object, acquire a first image without laser illumination and a first image with laser illumination taken of the reference object; S2: When the camera is separated from the reference object by a second distance, acquire a second image without laser illumination and a second image with laser illumination taken of the reference object; S3: Process the first image without laser illumination, the first image with laser illumination, the second image without laser illumination, and the second image with laser illumination based on the camera's intrinsic parameters and the camera-IMU associated extrinsic parameters to obtain the lidar-IMU installation deviation; wherein, the lidar-IMU installation deviation includes the pitch installation deviation angle and the heading installation deviation angle between the lidar and the IMU, specifically: The pose of the camera in the world coordinate system, which is separated from the reference object by a first distance, is calculated based on the first laser-free image. c1 , t c1 ], where R c1 For rotation matrix, t c1 It is a translation vector; Based on the first image without laser illumination and the first image with laser illumination, determine the light spot l1 formed by the emitted visible light on the reference object, and calculate the pixel coordinates p1 corresponding to the light spot l1. The pixel coordinates p1 corresponding to the light spot l1 are transformed into normalized planar coordinates P using camera intrinsic parameters. 1, c Using the pose of the camera [R] c1 , t c1 ] and the normalized plane coordinates P 1, c Calculate the coordinates P of the light spot l1 in the world coordinate system. 1, w ; Construct the camera ray vector in the world coordinate system according to the following formula: l w1 = R c1 P 1, c Among them, l w1 Indicates the optical center and spot of the camera The projection of the direction vector formed by the connection into the world coordinate system; Calculate the world coordinates of light point l1 using the following formula. : ;in, express The third row of data, express The third row of data; Accordingly, the pose of the camera in the world coordinate system at a second distance from the reference object is calculated for the second unilluminated image and the second laser-illuminated image. c2 , t c2 ], where R c2 For rotation matrix, t c2 Given the translation vector and the light spot l2 formed by the emitted visible light ray on the reference object, calculate the coordinates P of the light spot l2 in the world coordinate system. 2, w ; Calculate the coordinates of light spots l1 and l2 in the camera coordinate system using the following formula. and : in, and They represent and The inverse matrix; The direction vector of the light ray in the camera coordinate system is calculated using the following formula: ; in, This represents the direction vector of the light ray in the camera coordinate system; The direction vector of the ray in the IMU coordinate system is calculated using the following formula: ; in, This represents the direction vector of the light ray in the IMU coordinate system. This represents the rotation matrix from the position in the camera coordinate system to the position in the world coordinate system, i.e., the camera-IMU associated extrinsic parameter; The pitch and heading installation deviation angles between the lidar and the IMU are calculated using the following formulas: Where Δθ and ΔΨ represent the pitch installation deviation angle and heading installation deviation angle between the lidar and the IMU, respectively. , , They represent direction vectors respectively. Components in the X, Y, and Z directions.

2. A surveying method, characterized in that, The method includes the following steps: S100: Adjust the relative spatial relationship of the lidar-IMU according to the lidar-IMU installation deviation; wherein, the lidar-IMU installation deviation is obtained by the vision-assisted lidar-IMU installation deviation calibration method as described in claim 1; S200: Based on the dynamic characteristics of the mapping path, construct a one-to-many correspondence between IMU data and point cloud data frames from the lidar; filter the point cloud data frames that best match the corresponding location points of the mapping path from all point cloud data corresponding to the same IMU data, thereby constructing a local point cloud map corresponding to the location points. S300: Based on the spatial layout of all location points within the surveying path, integrate the local point cloud maps corresponding to all location points to construct a spatial map of the target area where the surveying path is located.

3. The surveying method according to claim 2, characterized in that, In S100, the relative spatial relationship between the lidar and IMU is adjusted according to the installation deviation of the lidar-IMU, specifically as follows: The installation correction path of the lidar-IMU is determined based on the adjustable parameters of the single relative motion between the lidar and the IMU, the pitch installation deviation angle and the heading installation deviation angle between the lidar and the IMU. According to the lidar-IMU installation correction path, adjust the spatial position of at least one of the lidar and IMU to change the relative spatial relationship between the lidar and IMU until the pitch installation deviation angle and the heading installation deviation angle are eliminated.

4. The surveying method according to claim 2, characterized in that, In S200, based on the dynamic characteristics of the mapping path, a one-to-many correspondence is constructed between IMU data and point cloud data frames from the lidar; from all point cloud data corresponding to the same IMU data, the point cloud data frame that best matches the corresponding location point of the mapping path is selected, thereby constructing a local point cloud map corresponding to the location point, specifically: The motion dynamics of the moving body along the surveying path in the target area are obtained; wherein, the motion dynamics include motion speed variation characteristics and motion trajectory curvature variation characteristics; based on the motion dynamics, several position points are marked on the surveying path; Based on the timing of the arrival of the mobile body at the several location points, a one-to-many correspondence between IMU data and point cloud data frames from the lidar is constructed; wherein, whenever the mobile body arrives at a location point, the IMU generates one IMU data and the lidar forms multiple point cloud data frames in different azimuth ranges relative to the location point. The layout features of face points and corner points of all point cloud data frames corresponding to the same IMU data are obtained. Based on the layout features, the spatial positioning error of each point cloud data frame is estimated. The point cloud data frame with the smallest spatial positioning error is selected as the point cloud data frame that best matches the location point where the IMU data is generated, thereby constructing a local point cloud map corresponding to the location point.

5. The surveying method according to claim 2, characterized in that, In S300, based on the spatial layout of all location points within the surveying path, the local point cloud maps corresponding to all location points are integrated to construct a spatial map of the target area where the surveying path is located, specifically: Based on the spatial layout and positional relationship of all location points within the surveying path, the local point cloud maps corresponding to all location points are spatially arranged to determine the overlapping point cloud data between any two spatially adjacent local point cloud maps. Redundant point cloud data is removed from the overlapping point cloud data and adjacent point cloud data are stitched together to obtain a global point cloud map formed by all local point cloud maps, thereby constructing a spatial map of the target area where the surveying path is located.

6. A surveying system, characterized in that, The system includes the following modules: A spatial adjustment module is used to adjust the relative spatial relationship of the lidar-IMU according to the lidar-IMU installation deviation; wherein, the lidar-IMU installation deviation is obtained by the vision-assisted lidar-IMU installation deviation calibration method as described in claim 1; The point cloud data preprocessing module is used to construct a one-to-many correspondence between IMU data and point cloud data frames from lidar based on the dynamic characteristics of the mapping path. The local point cloud map construction module is used to filter the point cloud data frames that best match the corresponding location points of the mapping path from all point cloud data corresponding to the same IMU data, so as to construct the local point cloud map corresponding to the location points. The spatial map construction module is used to integrate the local point cloud maps corresponding to all location points according to the spatial layout of all location points within the surveying path, and construct a spatial map of the target area where the surveying path is located.

7. The surveying system according to claim 6, characterized in that, The spatial adjustment module is used to adjust the relative spatial relationship of the lidar-IMU according to the installation deviation of the lidar-IMU, specifically as follows: The installation correction path of the lidar-IMU is determined based on the adjustable parameters of the single relative motion between the lidar and the IMU, the pitch installation deviation angle and the heading installation deviation angle between the lidar and the IMU. According to the lidar-IMU installation correction path, adjust the spatial position of at least one of the lidar and IMU to change the relative spatial relationship between the lidar and IMU until the pitch installation deviation angle and the heading installation deviation angle are eliminated.

8. The surveying system according to claim 6, characterized in that, The point cloud data preprocessing module is used to construct a one-to-many correspondence between IMU data and point cloud data frames from lidar based on the dynamic characteristics of the mapping path, specifically: The motion dynamics of the moving body along the surveying path in the target area are obtained; wherein, the motion dynamics include motion speed variation characteristics and motion trajectory curvature variation characteristics; based on the motion dynamics, several position points are marked on the surveying path; Based on the timing of the movement reaching the several location points, a one-to-many correspondence is constructed between IMU data and point cloud data frames from the lidar; wherein, whenever the movement reaches a location point, the IMU generates IMU data and the lidar forms multiple point cloud data frames in different azimuth ranges relative to the location point.

9. The surveying system according to claim 8, characterized in that, The local point cloud map construction module is used to filter point cloud data frames that best match the corresponding location points of the mapping path from all point cloud data corresponding to the same IMU data, thereby constructing a local point cloud map corresponding to the location points. Specifically: The layout features of face points and corner points of all point cloud data frames corresponding to the same IMU data are obtained. Based on the layout features, the spatial positioning error of each point cloud data frame is estimated. The point cloud data frame with the smallest spatial positioning error is selected as the point cloud data frame that best matches the location point where the IMU data is generated, thereby constructing a local point cloud map corresponding to the location point.

10. The surveying system according to claim 6, characterized in that, The spatial map construction module is used to integrate the local point cloud maps corresponding to all location points according to the spatial layout of all location points within the surveying path, and construct a spatial map of the target area where the surveying path is located, specifically: Based on the spatial layout and positional relationship of all location points within the surveying path, the local point cloud maps corresponding to all location points are spatially arranged to determine the overlapping point cloud data between any two spatially adjacent local point cloud maps. Redundant point cloud data is removed from the overlapping point cloud data and adjacent point cloud data are stitched together to obtain a global point cloud map formed by all local point cloud maps, thereby constructing a spatial map of the target area where the surveying path is located.

Images