Engineering Structure Reconstruction Method Integrating LiDAR, Inertial Navigation Unit and Stereo Vision

Abstract

The present invention relates to the technical field of three-dimensional reconstruction of engineering structures, and particularly to a method and system for reconstructing engineering structures that integrates lidar, inertial navigation unit, and stereo vision. The method includes: a drone carrying a lidar, an inertial navigation unit, and a stereo vision device; subscribing to sensor topics based on the ROS system of the drone to obtain a set of sensor calibration parameters; according to the set of sensor calibration parameters, three-dimensional point cloud data, and visual image data, constructing a model through a stereo matching algorithm to obtain an initial dense texture point cloud model; based on a tightly coupled three-dimensional reconstruction algorithm, fusing data through a geometric consistency algorithm to obtain a final fused point cloud model; performing dynamic parameter compensation based on a stereo vision dynamic compensation algorithm of the inertial navigation unit and a motion compensation method of the lidar to obtain a multi-source model of the engineering structure. The present invention is a method for three-dimensional reconstruction of engineering structures that combines lidar, inertial navigation unit, and stereo vision with high accuracy and rich texture.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction of engineering structures, and particularly to a method and system for reconstructing engineering structures by integrating lidar, inertial navigation unit and stereo vision. Background Art

[0002] There are numerous existing engineering structures in our country. With the continuous development of transportation and residential infrastructure, the quantity and scale of engineering structures are increasing year by year. However, the maintenance and management of engineering structures face many challenges. Currently, three-dimensional reconstruction of engineering structures plays an important role in the detection, evaluation and maintenance of engineering structures. However, the existing three-dimensional reconstruction technologies of engineering structures mainly rely on manual measurement and traditional modeling methods.

[0003] Traditional manual measurement methods require professional personnel to conduct on-site surveys. The measurement process is cumbersome, time-consuming, and inefficient. Especially when dealing with large-scale or complex engineering structures, the problems are more prominent. Due to the limitations of manual measurement methods, the accuracy of data acquisition is limited, resulting in a low-precision three-dimensional model, which is difficult to meet the requirements of high-precision detection and evaluation. In addition, existing modeling methods often perform inadequately when dealing with the complex geometric structures and details of engineering structures, and cannot truly reflect the actual situation of engineering structures. During the three-dimensional reconstruction process of engineering structures, it is necessary to integrate data from different sensors (such as lidar, inertial navigation unit and stereo vision) to obtain complete engineering structure and texture information. However, the spatio-temporal synchronization and external parameter calibration of data between different sensors are technical problems, and existing methods have deficiencies in terms of efficiency and accuracy.

[0004] In the prior art, there is a lack of a method for three-dimensional reconstruction of engineering structures that combines lidar, inertial navigation unit and stereo vision with high accuracy and rich texture. Summary of the Invention

[0005] In order to solve the technical problem of the low accuracy of the reconstructed three-dimensional model in the prior art, an embodiment of the present invention provides a method and system for reconstructing engineering structures by integrating lidar, inertial navigation unit and stereo vision. The technical solution is as follows:

[0006] On the one hand, a method for reconstructing engineering structures by integrating lidar, inertial navigation unit and stereo vision is provided. This method is implemented by an engineering structure reconstruction system, and the method includes:

[0007] An unmanned aerial vehicle is equipped with lidar, an inertial navigation unit and stereo vision equipment; based on the ROS system of the unmanned aerial vehicle, sensor topics are subscribed to obtain a set of sensor calibration parameters; according to the set of sensor calibration parameters, joint calibration is performed using a method based on the method without calibration objects and a method based on automatic corner grabbing to obtain joint calibration parameters;

[0008] According to the joint calibration parameters, the UAV takes pictures of the project to be reconstructed according to a preset close-range path strategy; the lidar is used for data acquisition to obtain three-dimensional point cloud data; the inertial navigation unit is used for data acquisition to obtain inertial test data; the stereo vision device is used for data acquisition to obtain visual image data;

[0009] According to the sensor calibration parameter set, the three-dimensional point cloud data, and the visual image data, a model is constructed through a stereo matching algorithm to obtain an initial dense texture point cloud model;

[0010] Based on a tightly coupled three-dimensional reconstruction algorithm, according to the initial dense texture point cloud model, the three-dimensional point cloud data, and the visual image data, data fusion is performed through a geometric consistency algorithm to obtain a final fused point cloud model;

[0011] Based on a stereo vision dynamic compensation algorithm of the inertial navigation unit and a motion compensation method of the lidar, dynamic parameter compensation is performed on the final fused point cloud model according to the inertial test data to obtain a multi-source model of the engineering structure.

[0012] On the other hand, an engineering structure reconstruction system integrating a lidar, an inertial navigation unit, and stereo vision is provided. This system is applied to an engineering structure reconstruction method integrating a lidar, an inertial navigation unit, and stereo vision. The system includes:

[0013] The UAV is used to carry a lidar, an inertial navigation unit, and a stereo vision device; the ROS system of the UAV subscribes to sensor topics to obtain a sensor calibration parameter set; according to the joint calibration parameters, the UAV takes pictures of the project to be reconstructed according to a preset close-range path strategy;

[0014] The lidar is used for data acquisition through the lidar to obtain three-dimensional point cloud data;

[0015] The inertial navigation unit is used for data acquisition through the inertial navigation unit to obtain inertial test data;

[0016] The stereo vision device is used for data acquisition according to the stereo vision device to obtain visual image data;

[0017] The electronic device is configured to perform joint calibration using a calibration-free method and a fully automatic corner-based grabbing method according to the sensor calibration parameter set to obtain joint calibration parameters; perform model construction through a stereo matching algorithm according to the sensor calibration parameter set, the three-dimensional point cloud data, and the visual image data to obtain an initial dense texture point cloud model; perform data fusion through a geometric consistency algorithm based on a tightly coupled three-dimensional reconstruction algorithm according to the initial dense texture point cloud model, the three-dimensional point cloud data, and the visual image data to obtain a final fused point cloud model; perform dynamic parameter compensation on the final fused point cloud model according to the inertial test data based on a stereo vision dynamic compensation algorithm of an inertial navigation unit and a motion compensation method of a lidar to obtain a multi-source model of an engineering structure.

[0018] On the other hand, an engineering structure reconstruction system is provided. The engineering structure reconstruction system includes: a processor; a memory storing computer-readable instructions thereon, and when the computer-readable instructions are executed by the processor, any one of the engineering structure reconstruction methods integrating a lidar, an inertial navigation unit, and stereo vision as described above is implemented.

[0019] On the other hand, a computer-readable storage medium is provided, and at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the engineering structure reconstruction methods integrating a lidar, an inertial navigation unit, and stereo vision as described above.

[0020] The beneficial effects brought by the technical solutions provided in the embodiments of the present invention at least include:

[0021] The present invention proposes an engineering structure reconstruction method integrating a lidar, an inertial navigation unit, and stereo vision, which uses inertial navigation unit nodes to assist lidar nodes to construct a high-precision geometric structure of an engineering structure; endows the engineering structure with real color texture information through inertial navigation unit nodes assisting stereo vision, providing technical support for the three-dimensional reconstruction of the engineering structure. In terms of the three-dimensional reconstruction of the engineering structure, the accuracy of the reconstructed model is improved, facilitating the management and operation and maintenance of the engineering structure. In a complex environment of the engineering structure, the three-dimensional reconstruction of the engineering structure is realized more accurately and in more detail. The present invention is a three-dimensional reconstruction method of an engineering structure that combines a lidar, an inertial navigation unit, and stereo vision with high accuracy and rich texture. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings without creative efforts based on these drawings.

[0023] Figure 1 It is a flowchart of an engineering structure reconstruction method that integrates lidar, inertial navigation unit, and stereo vision provided by an embodiment of the present invention;

[0024] Figure 2 It is a block diagram of an engineering structure reconstruction system that integrates lidar, inertial navigation unit, and stereo vision provided by an embodiment of the present invention;

[0025] Figure 3 It is a schematic structural diagram of an engineering structure reconstruction system provided by an embodiment of the present invention. Specific embodiments

[0026] The following describes the technical solutions in the present invention with reference to the accompanying drawings.

[0027] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as an "example" in the present invention should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of the word "example" is intended to present concepts in a specific manner. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one of the two.

[0028] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meanings they express are the same. "(of)", "corresponding", and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meanings they express are the same.

[0029] In the embodiments of the present invention, sometimes subscripts such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meanings they express are the same.

[0030] To make the technical problems, technical solutions, and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0031] The embodiments of the present invention provide an engineering structure reconstruction method that integrates lidar, inertial navigation unit, and stereo vision. This method can be implemented by an engineering structure reconstruction system, which can be a terminal or a server. As Figure 1 shown in the flowchart of the engineering structure reconstruction method that integrates lidar, inertial navigation unit, and stereo vision, the processing flow of this method can include the following steps:

[0032] S1. A drone is equipped with a lidar, an inertial navigation unit, and a stereo vision device. The ROS system based on the drone subscribes to the sensor topic to obtain a set of sensor calibration parameters. According to the set of sensor calibration parameters, a joint calibration is performed using the method based on the calibration-free object and the fully automatic corner grabbing method to obtain the joint calibration parameters.

[0033] In a feasible implementation, the present invention utilizes an unmanned device equipped with devices such as a lidar LiDAR, an inertial navigation unit IMU, and stereo vision. The ROS system of the drone subscribes to the calibration parameter topics of sensors such as LiDAR, cameras, and IMU. The obtained set of sensor calibration parameters includes the camera internal parameters (focal length, principal point coordinates, distortion coefficients), external parameters (relative pose between the camera and the IMU), and the attitude information of the IMU.

[0034] The device used is a Velodyne 16 lidar LiDAR. The inertial navigation unit IMU of the HI229 model is selected to cooperate with the LiDAR to reconstruct the geometric structure of the bridge model and collaborate with the binocular camera to render the texture of the bridge model.

[0035] The IMU is external or built-in in the LiDAR or stereo vision. The LiDAR is a three-dimensional type with more than 16 lines and planar local scanning or omnidirectional scanning, and the model is Velodyne 16. The stereo vision device is a high-resolution stereo vision device with a measurement distance of 0.5 - 10m.

[0036] Optionally, according to the set of sensor calibration parameters, a joint calibration is performed using the method based on the calibration-free object and the fully automatic corner grabbing method to obtain the joint calibration parameters, including:

[0037] According to the set of sensor calibration parameters, the method based on the calibration-free object is used to perform a joint calibration on the lidar and the inertial navigation unit to obtain the first joint calibration parameters;

[0038] According to the set of sensor calibration parameters, the fully automatic corner grabbing method is used to perform a joint calibration on the lidar and the stereo vision device to obtain the second joint calibration parameters.

[0039] In a feasible implementation, the present invention uses a Velodyne VLP-16 LiDAR and a HI229 IMU, and uses a ROS package to record the data of both. The two devices need to be firmly fixed and sufficient dynamic excitation, including but not limited to rotation, tilt, and acceleration changes, etc., should be carried out during the recording process.

[0040] Make a preliminary estimate of the extrinsic parameters between the LiDAR and the IMU; determine the matching relationship between the LiDAR data and the IMU data, and use the established data association for preliminary optimization to finely adjust the relative position and attitude between the sensors; based on the initial optimization, perform multiple rounds of iterative optimization until the extrinsic parameter converges; finally, output the combined calibration result file, that is, the first combined calibration parameter is in.yaml format.

[0041] Based on the calibration-free method and ROS, excluding specific target environment data, but it should be ensured that the two devices are firmly fixed, and data recording should be carried out in an environment with many feature points and rich planes, such as a general office environment, to ensure sufficient movement and rotation in the x, y, and z axis directions, while avoiding violent impact motions caused by excessive acceleration.

[0042] Use Autoware to perform combined calibration on the Velodyne VLP-16 LiDAR and stereo vision. Specifically, first perform coordinate transformation on the LiDAR coordinates with the left and right cameras of the stereo vision respectively to unify the coordinate systems. After completing the individual transformations, then perform a combined transformation on these two results to obtain the second combined calibration parameter.

[0043] In the fully automatic corner-based grasping method, the data collected based on the calibration toolbox and the calibration board must include the point cloud and image ROS topic data packets where the LiDAR and stereo vision in the environment are simultaneously mapped on the calibration board, and the data outside the calibration board should be discarded.

[0044] S2. According to the combined calibration parameters, use the drone to take pictures of the project to be reconstructed according to the preset close-range path strategy; collect data through the lidar to obtain three-dimensional point cloud data; collect data through the inertial navigation unit to obtain inertial test data; collect data according to the stereo vision device to obtain visual image data.

[0045] In a feasible real-time method, the preset "three-sided enclosure" close-range path strategy of the present invention includes using unmanned equipment to plan the acquisition paths above and around the engineering structure, and the close-range distance should be greater than 0.5 meters and less than 10 meters.

[0046] The "three-sided enclosure" close-range path strategy needs to be planned through the structure engineering facade path terrain-following flight path algorithm or related software, involving adjusting the path according to the surface shape of the structure engineering, and finally ensuring that the path flight line overlap rate is greater than 60%, and the path flight lines on the side and directly above are generated integrally.

[0047] The "three-sided encirclement" data acquisition path strategy comprehensively considers various principles to ensure that the flight path of the UAV can fully cover all parts of the engineering structure, including directly above, on both sides, and around the engineering structure; ensure a safety distance of more than 0.5 m; and ensure a data acquisition method with multiple angles and multiple heights as much as possible. Data acquisition is carried out in an engineering structure environment with sufficient light, low wind speed, few people and vehicles, and no obstruction.

[0048] Based on the described path plan, LiDAR, IMU, and stereo vision node data of the engineering structure to be reconstructed are collected closely and synchronously. The data acquisition technology uses the method of subscribing to sensor topics based on ROS to obtain multi-source data of the real engineering structure.

[0049] The unmanned device computing unit belongs to a portable airborne computer; using the computing unit, it communicates with each sensor module through the ROS platform, subscribes to relevant ROS topics to obtain necessary multi-sensor data, including LiDAR point cloud, stereo vision images of binocular cameras, and IMU data, and at the same time records an engineering structure multi-source data packet of no less than 150 s.

[0050] S3. According to the sensor calibration parameter set, three-dimensional point cloud data, and visual image data, a model is constructed through a stereo matching algorithm to obtain an initial dense texture point cloud model.

[0051] Optionally, according to the sensor calibration parameter set, three-dimensional point cloud data, and visual image data, a model is constructed through a stereo matching algorithm to obtain an initial dense texture point cloud model, including:

[0052] Based on the sensor calibration parameter set, according to the visual image data, a stereo matching algorithm is used to match the structural points of the left and right images to obtain a disparity map;

[0053] Based on the three-dimensional point cloud data, the pixel points in the disparity map are converted into three-dimensional coordinates to obtain dense point cloud data;

[0054] A model is constructed according to the dense point cloud data to obtain a dense point cloud model;

[0055] According to the visual image data, texture filling is performed on the dense point cloud model to obtain an initial dense texture point cloud model.

[0056] In a feasible implementation manner, the present invention uses a stereo matching algorithm to track the camera pose by minimizing the radiance difference between the points in the map and the sparse pixel set in the current image.

[0057] The present invention uses a classic stereo matching algorithm to process the left and right images. According to the disparity principle, this algorithm generates a disparity map by finding the matching point pairs of the same structural points in the left and right images.

[0058] A dense point cloud is generated by converting each pixel point in the disparity map into three-dimensional coordinates. Texture information extracted from the original image is attached to the generated point cloud model, thereby generating a dense textured point cloud model with high resolution and fine texture. Through the above steps, an initial dense textured point cloud model is generated.

[0059] S4. Based on the tightly coupled three-dimensional reconstruction algorithm, according to the initial dense textured point cloud model, three-dimensional point cloud data, and visual image data, data fusion is performed through a geometric consistency algorithm to obtain a final fused point cloud model.

[0060] Optionally, based on the tightly coupled three-dimensional reconstruction algorithm, according to the initial dense textured point cloud model, three-dimensional point cloud data, and visual image data, data fusion is performed through a geometric consistency algorithm to obtain a final fused point cloud model, including:

[0061] According to the three-dimensional point cloud data, a sparse point cloud model is constructed through the tightly coupled three-dimensional reconstruction algorithm.

[0062] Based on the dense textured point cloud model, the sparse point cloud model is spatially aligned to obtain an aligned sparse point cloud model.

[0063] According to the initial dense textured point cloud model and the aligned sparse point cloud model, matching fusion is performed through a geometric consistency algorithm to obtain a first fused point cloud model.

[0064] According to the three-dimensional point cloud data, geometric structure reconstruction is performed on the first fused point cloud model to obtain a second fused point cloud model.

[0065] According to the visual image data, color rendering is performed on the second fused point cloud model to obtain the final fused point cloud model.

[0066] In a feasible implementation manner, the present invention adopts a geometric consistency algorithm to initially match the point cloud data of different sensors, perform local geometric consistency checks and global optimization based on the nearest neighbor search matching method, exclude mis-matched points and fuse the point clouds to generate a first fused point cloud model.

[0067] The three-dimensional point cloud data is registered to the first fused point cloud model to reconstruct the geometric structure, and combined with stereo vision to realize rendering the pixel colors in the image to the points in the second fused point cloud model to restore the radiometric information.

[0068] S5. Based on the stereo vision dynamic compensation algorithm of the inertial navigation unit and the motion compensation method of the lidar, according to the inertial test data, dynamic parameter compensation is performed on the final fused point cloud model to obtain an engineering structure multi-source model.

[0069] Optionally, based on the stereo vision dynamic compensation algorithm of the inertial navigation unit and the motion compensation method of the lidar, according to the inertial test data, dynamic parameter compensation is performed on the final fused point cloud model to obtain a multi-source model of the engineering structure, including:

[0070] Based on the stereo vision dynamic compensation algorithm, motion compensation is performed according to the inertial test data to obtain the dynamic compensation of the inertial navigation unit;

[0071] According to the dynamic compensation of the inertial navigation unit, through the motion compensation method of the lidar, dynamic parameter compensation is performed on the final fused point cloud model to obtain the calibrated external parameter set;

[0072] According to the calibrated external parameter set, parameter setting is performed on the final fused point cloud model to obtain a preliminary multi-source model of the engineering structure;

[0073] Perform multi-sensor spatial pose alignment on the preliminary multi-source model of the engineering structure to obtain the multi-source model of the engineering structure.

[0074] In a feasible implementation manner, in the Ubuntu18.04 development environment, for the input of the point cloud scan data of the multi-line LiDAR Velodyne VLP-16, motion compensation is performed using the backpropagation of the state data of the IMU. The accumulated three-dimensional points in the three-dimensional model of the engineering structure form a geometric structure, realizing the motion compensation of the LiDAR assisted by the IMU data.

[0075] Based on the Rviz visualization interface in ROS, based on the final fused point cloud model, play the data packet of the previously obtained multi-source data of the engineering structure to be reconstructed; set the obtained external parameter calibration file of the three in the three-dimensional reconstruction deployment environment of the engineering structure; after the data packet playback is completed, save the multi-source model of the engineering structure in the.pcd format.

[0076] Among them, the stereo vision dynamic compensation algorithm is as follows in formulas (1), (2), and (3):

[0077] (1);

[0078] (2);

[0079] (3);

[0080] Among them, the point P represents each point of the final fused point cloud model, and W is the final state of the stereo vision; is the attitude and position of the IMU; is the global linear velocity; is the bias of the gyroscope and accelerometer in the IMU; is the global gravitational acceleration; is the external parameter of the camera; is the reciprocal of the camera exposure time; is the time offset; are the camera intrinsics; ( is the camera focal length, ( is the offset of the principal point from the top left corner of the image plane.

[0081] In a feasible implementation, and operations are mainly used to map state variables between the manifold and the tangent space. These two operations are usually used to handle optimization problems in non-Euclidean spaces, such as pose estimation.

[0082] Among them, the motion compensation method is as shown in the following formulas (4), (5):

[0083] (4);

[0084] (5);

[0085] Among them, MAP is the Maximum A-Posteriori estimate; is the squared Mahalanobis distance of the covariance; is the state estimate propagated by the IMU; is the state covariance propagated by the IMU; is the LiDAR measurement residual; is the residual Jacobian of; is the total noise; is the mapping operation to transform the state variable from the manifold to the tangent space; is the mapping operation to transform the state variable from the tangent space to the manifold.

[0086] In a feasible implementation, for the inertial navigation unit-based stereo vision dynamic compensation algorithm, a certain number of tracked points are selected from the 3D model of the engineering structure, and these points are projected into the current image data of the stereo vision. The photometric error is calculated by minimizing these points and the state estimate of the stereo vision-IMU is iteratively updated within the filter framework. In this process, the texture color of a single engineering structure model point is used to calculate the photometric error. The model points are tracked based on the frame-to-frame optical flow method, and the system state is estimated and updated through the photometric error method. After the state estimate is stable, texture rendering is performed to update the color of the points in the engineering structure model.

[0087] The present invention proposes an engineering structure reconstruction method that integrates lidar, inertial navigation unit, and stereo vision. The inertial navigation unit nodes are used to assist the lidar nodes in constructing a high-precision geometric structure of the engineering structure; the inertial navigation unit nodes are used to assist stereo vision in endowing the engineering structure with real color texture information, providing technical support for the three-dimensional reconstruction of the engineering structure. In terms of the three-dimensional reconstruction of the engineering structure, the accuracy of the reconstructed model is improved, facilitating the management and operation and maintenance of the engineering structure. In a complex environment of the engineering structure, the three-dimensional reconstruction of the engineering structure can be realized more accurately and in more detail. The present invention is a three-dimensional reconstruction method for an engineering structure that combines lidar, inertial navigation unit, and stereo vision with high accuracy and rich texture.

[0088] Figure 2 FIG. is a block diagram of an engineering structure reconstruction system that integrates lidar, inertial navigation unit, and stereo vision according to an exemplary embodiment. This system is used for the engineering structure reconstruction method that integrates lidar, inertial navigation unit, and stereo vision. Referring to Figure 2 , the system includes an unmanned aerial vehicle 210, a lidar 220, an inertial navigation unit 230, a stereo vision device 240, and an electronic device 250. Among them:

[0089] The unmanned aerial vehicle 210 is used to carry a lidar, an inertial navigation unit, and a stereo vision device on the unmanned aerial vehicle; subscribe to sensor topics based on the ROS system of the unmanned aerial vehicle to obtain a set of sensor calibration parameters; according to the joint calibration parameters, the unmanned aerial vehicle takes pictures of the engineering structure to be reconstructed according to a preset close-range path strategy;

[0090] The lidar 220 is used to collect data through the lidar to obtain three-dimensional point cloud data;

[0091] The inertial navigation unit 230 is used to collect data through the inertial navigation unit to obtain inertial test data;

[0092] The stereo vision device 240 is used to collect data according to the stereo vision device to obtain visual image data;

[0093] The electronic device 250 is used to perform joint calibration using a calibration-free method and a fully automatic corner grabbing method based on the set of sensor calibration parameters to obtain joint calibration parameters; according to the set of sensor calibration parameters, three-dimensional point cloud data, and visual image data, a model is constructed through a stereo matching algorithm to obtain an initial dense texture point cloud model; based on a tightly coupled three-dimensional reconstruction algorithm, data fusion is performed through a geometric consistency algorithm according to the initial dense texture point cloud model, three-dimensional point cloud data, and visual image data to obtain a final fused point cloud model; based on a stereo vision dynamic compensation algorithm of the inertial navigation unit and a motion compensation method of the lidar, dynamic parameter compensation is performed on the final fused point cloud model according to the inertial test data to obtain a multi-source model of the engineering structure.

[0094] Optionally, the electronic device 250 is further configured to:

[0095] According to the sensor calibration parameter set, using a calibration-free method, jointly calibrate the lidar and the inertial navigation unit to obtain the first joint calibration parameters;

[0096] According to the sensor calibration parameter set, using a fully automatic corner grabbing method, jointly calibrate the lidar and the stereo vision device to obtain the second joint calibration parameters.

[0097] Optionally, the electronic device 250 is further configured to:

[0098] Based on the sensor calibration parameter set, according to the visual image data, use a stereo matching algorithm to perform left and right image structure point matching to obtain a disparity map;

[0099] Based on the three-dimensional point cloud data, convert the pixel points in the disparity map into three-dimensional coordinates to obtain dense point cloud data;

[0100] Construct a model according to the dense point cloud data to obtain a dense point cloud model;

[0101] According to the visual image data, perform texture filling on the dense point cloud model to obtain an initial dense texture point cloud model.

[0102] Optionally, the electronic device 250 is further configured to:

[0103] According to the three-dimensional point cloud data, construct a model through a tightly coupled three-dimensional reconstruction algorithm to obtain a sparse point cloud model;

[0104] Based on the dense texture point cloud model, spatially align the sparse point cloud model to obtain an aligned sparse point cloud model;

[0105] According to the initial dense texture point cloud model and the aligned sparse point cloud model, perform matching and fusion through a geometric consistency algorithm to obtain a first fused point cloud model;

[0106] According to the three-dimensional point cloud data, perform geometric structure reconstruction on the first fused point cloud model to obtain a second fused point cloud model;

[0107] According to the visual image data, perform color rendering on the second fused point cloud model to obtain a final fused point cloud model.

[0108] Optionally, the electronic device 250 is further configured to:

[0109] Based on the stereo vision dynamic compensation algorithm, perform motion compensation according to the inertial test data to obtain inertial navigation unit dynamic compensation;

[0110] According to the dynamic compensation of the inertial navigation unit, through the motion compensation method of the lidar, dynamic parameter compensation is performed on the final fused point cloud model to obtain the calibrated external parameter set;

[0111] According to the calibrated external parameter set, parameter setting is performed on the final fused point cloud model to obtain the preliminary multi-source model of the engineering structure;

[0112] Perform multi-sensor spatial pose alignment on the preliminary multi-source model of the engineering structure to obtain the multi-source model of the engineering structure.

[0113] Among them, the stereo vision dynamic compensation algorithm is as follows in equations (1), (2), and (3):

[0114] (1);

[0115] (2);

[0116] (3);

[0117] Among them, point P represents each point of the final fused point cloud model, and W is the final state of the stereo vision; is the attitude and position of the IMU; is the global linear velocity; is the bias of the gyroscope and accelerometer in the IMU; is the global gravitational acceleration; is the external parameter of the camera; is the reciprocal of the camera exposure time; is the time offset; is the internal parameter of the camera; ([[]]END]] ) is the camera focal length, ([[]]END]] ) is the offset of the principal point from the upper left corner of the image plane.

[0118] Among them, the motion compensation method is as follows in equations (4) and (5):

[0119] (4);

[0120] (5);

[0121] Among them, MAP is the Maximum A-Posteriori estimate; is the squared Mahalanobis distance of the covariance; is the state estimate propagated by the IMU; is the state covariance propagated by the IMU; is the LiDAR measurement residual; is the residual of the Jacobian; is the total noise; A mapping operation for converting state variables from a manifold to a tangent space; A mapping operation for converting state variables from a tangent space to a manifold

[0122] The present invention provides an engineering structure reconstruction method that integrates lidar, an inertial navigation unit, and stereo vision. The inertial navigation unit nodes are used to assist the lidar nodes in constructing a high-precision geometric structure of the engineering structure; the inertial navigation unit nodes are used to assist stereo vision in endowing the engineering structure with real color texture information, providing technical support for the three-dimensional reconstruction of the engineering structure. In terms of the three-dimensional reconstruction of the engineering structure, the accuracy of the reconstructed model is improved, facilitating the management and operation and maintenance of the engineering structure. In a complex environment of the engineering structure, the three-dimensional reconstruction of the engineering structure can be realized more accurately and in more detail. The present invention is a three-dimensional reconstruction method for an engineering structure that combines lidar, an inertial navigation unit, and stereo vision and has high accuracy and rich texture.

[0123] Figure 3 It is a schematic structural diagram of an engineering structure reconstruction system provided by an embodiment of the present invention. As Figure 3 shown, the engineering structure reconstruction system may include the above Figure 2 shown engineering structure reconstruction system that integrates lidar, an inertial navigation unit, and stereo vision. Optionally, the engineering structure reconstruction system 310 may include a first processor 2001.

[0124] Optionally, the engineering structure reconstruction system 310 may further include a memory 2002 and a transceiver 2003.

[0125] Among them, the first processor 2001 is connected to the memory 2002 and the transceiver 2003, such as through a communication bus.

[0126] Next, in combination with Figure 3 each component of the engineering structure reconstruction system 310 will be specifically introduced:

[0127] Among them, the first processor 2001 is the control center of the engineering structure reconstruction system 310, which can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or can be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as: one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0128] Optionally, the first processor 2001 may execute various functions of the engineering structure reconstruction system 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0129] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 the CPU0 and CPU1 shown in

[0130] In a specific implementation, as an embodiment, the engineering structure reconstruction system 310 may also include multiple processors, such as Figure 3 the first processor 2001 and the second processor 2004 shown in

[0131] Among them, the memory 2002 is used to store software programs for executing the solution of the present invention and is controlled by the first processor 2001 for execution. The specific implementation manner may refer to the above method embodiments and will not be elaborated here.

[0132] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage systems that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage systems that can store information and instructions, or may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage systems, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be coupled to the first processor 2001 through an interface circuit ( Figure 3 not shown in

[0133] The transceiver 2003 is used to communicate with a network system or a terminal system.

[0134] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 not shown separately). Among them, the receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0135] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and is coupled to the first processor 2001 through an interface circuit ( Figure 3 not shown) of the engineering structure reconstruction system 310. The embodiments of the present invention do not make specific limitations on this.

[0136] It should be noted that Figure 3 the structure of the engineering structure reconstruction system 310 shown in

[0137] does not constitute a limitation on the router. The actual knowledge structure recognition system may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0138] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0139] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0140] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage system such as a server or data center that contains one or more collections of available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0141] It should be understood that the term "and / or" in this document is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship, which can be specifically understood by referring to the context before and after.

[0142] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0143] It should be understood that in various embodiments of the present invention, the magnitudes of the serial numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0144] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0145] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, systems, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0146] In several embodiments provided by the present invention, it should be understood that the disclosed systems, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of systems or units can be in electrical, mechanical, or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, the functional units in each embodiment of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0149] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer system (which can be a personal computer, a server, or a network system, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0150] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. An engineering structure reconstruction method integrating lidar, inertial navigation unit and stereo vision, characterized in that, The method includes: A drone is equipped with a lidar, an inertial navigation unit, and a stereo vision device; the ROS system of the drone subscribes to sensor topics to obtain a set of sensor calibration parameters; according to the set of sensor calibration parameters, a joint calibration is performed using a calibration-free method and a fully automatic corner grabbing method to obtain joint calibration parameters; According to the joint calibration parameters, the drone takes pictures of the project to be reconstructed according to a preset close-range path strategy; data is collected by the lidar to obtain three-dimensional point cloud data; data is collected by the inertial navigation unit to obtain inertial test data; data is collected according to the stereo vision device to obtain visual image data; According to the set of sensor calibration parameters, the three-dimensional point cloud data, and the visual image data, a model is constructed through a stereo matching algorithm to obtain an initial dense texture point cloud model; Based on a tightly coupled three-dimensional reconstruction algorithm, according to the initial dense texture point cloud model, the three-dimensional point cloud data, and the visual image data, data fusion is performed through a geometric consistency algorithm to obtain a final fused point cloud model; Among them, the step of performing data fusion through a geometric consistency algorithm according to the initial dense texture point cloud model, the three-dimensional point cloud data, and the visual image data based on the tightly coupled three-dimensional reconstruction algorithm to obtain a final fused point cloud model includes: According to the three-dimensional point cloud data, a model is constructed through a tightly coupled three-dimensional reconstruction algorithm to obtain a sparse point cloud model; Based on the dense texture point cloud model, the sparse point cloud model is spatially aligned to obtain an aligned sparse point cloud model; According to the initial dense texture point cloud model and the aligned sparse point cloud model, matching fusion is performed through a geometric consistency algorithm to obtain a first fused point cloud model; According to the three-dimensional point cloud data, geometric structure reconstruction is performed on the first fused point cloud model to obtain a second fused point cloud model; According to the visual image data, color rendering is performed on the second fused point cloud model to obtain a final fused point cloud model; Based on the stereo vision dynamic compensation algorithm of the inertial navigation unit and the motion compensation method of the lidar, dynamic parameter compensation is performed on the final fused point cloud model according to the inertial test data to obtain a multi-source model of the engineering structure.

2. The engineering structure reconstruction method integrating lidar, inertial navigation unit and stereo vision according to claim 1, wherein, The step of performing a joint calibration using a calibration-free method and a fully automatic corner grabbing method according to the set of sensor calibration parameters to obtain joint calibration parameters includes: According to the set of sensor calibration parameters, a joint calibration of the lidar and the inertial navigation unit is performed using a calibration-free method to obtain first joint calibration parameters; According to the set of sensor calibration parameters, a joint calibration of the lidar and the stereo vision device is performed using a fully automatic corner grabbing method to obtain second joint calibration parameters.

3. The engineering structure reconstruction method integrating lidar, inertial navigation unit and stereo vision according to claim 1, characterized in that, The step of constructing a model through a stereo matching algorithm according to the set of sensor calibration parameters, the three-dimensional point cloud data, and the visual image data to obtain an initial dense texture point cloud model includes: Based on the sensor calibration parameter set, according to the visual image data, use the stereo matching algorithm to match the structural points of the left and right images to obtain a disparity map; Based on the three-dimensional point cloud data, convert the pixel points in the disparity map into three-dimensional coordinates to obtain dense point cloud data; Construct a model according to the dense point cloud data to obtain a dense point cloud model; According to the visual image data, perform texture filling on the dense point cloud model to obtain an initial dense texture point cloud model.

4. The engineering structure reconstruction method integrating lidar, inertial navigation unit and stereo vision according to claim 1, characterized in that, The stereo vision dynamic compensation algorithm based on the inertial navigation unit and the motion compensation method of the lidar, according to the inertial test data, perform dynamic parameter compensation on the final fused point cloud model to obtain an engineering structure multi-source model, including: Based on the stereo vision dynamic compensation algorithm, perform motion compensation according to the inertial test data to obtain inertial navigation unit dynamic compensation; According to the inertial navigation unit dynamic compensation, through the motion compensation method of the lidar, perform dynamic parameter compensation on the final fused point cloud model to obtain a calibrated external parameter set; According to the calibrated external parameter set, perform parameter setting on the final fused point cloud model to obtain a preliminary engineering structure multi-source model; Perform multi-sensor spatial pose alignment on the preliminary engineering structure multi-source model to obtain an engineering structure multi-source model.

5. The engineering structure reconstruction method integrating lidar, inertial navigation unit and stereo vision according to claim 4, characterized in that The stereo vision dynamic compensation algorithm is as follows in equations (1), (2), and (3): ω = [f x , f y , c x , c y T (3);​ Among them, point P represents each point of the final fused point cloud model, and W is the final state of stereo vision; is the attitude and position of the IMU; v G is the global linear velocity; b g , b a is the bias of the gyroscope and accelerometer in the IMU; g G is the global gravitational acceleration; is the extrinsic camera parameter; ∈ is the reciprocal of the camera exposure time; is the time offset; ω is the intrinsic camera parameter; (f x , f y ) is the camera focal length, (c x , c y ) is the offset of the principal point from the upper left corner of the image plane.

6. The engineering structure reconstruction method integrating lidar, inertial navigation unit and stereo vision according to claim 4, characterized in that, The motion compensation method is as follows in equations (4) and (5): Among them, MAP is the Maximum A-Posteriori estimate; is the squared Mahalanobis distance of the covariance; is the state estimate propagated by the IMU; is the state covariance propagated by the IMU; is the LiDAR measurement residual; is the residual Jacobian of; ∑α s is the total noise; is the mapping operation to transform the state variable from the manifold to the tangent space; is the mapping operation to transform the state variable from the tangent space to the manifold.

7. An engineering structure reconstruction system integrating lidar, inertial navigation unit and stereo vision, which is used to implement the engineering structure reconstruction method integrating lidar, inertial navigation unit and stereo vision according to any one of claims 1-6, characterized in that, The system includes a drone, a lidar, an inertial navigation unit, a stereo vision device, and an electronic device, where: The drone is used to carry the lidar, inertial navigation unit, and stereo vision device; subscribe to the sensor topic based on the ROS system of the drone to obtain the sensor calibration parameter set; according to the joint calibration parameters, the drone takes pictures of the project to be reconstructed according to the preset close-range path strategy; The lidar is used to collect data through the lidar to obtain three-dimensional point cloud data; The inertial navigation unit is used to collect data through the inertial navigation unit to obtain inertial test data; The stereo vision device is used to collect data according to the stereo vision device to obtain visual image data; The electronic device is used to perform joint calibration using the method based on the non-calibration object and the fully automatic corner grabbing method according to the sensor calibration parameter set to obtain joint calibration parameters; according to the sensor calibration parameter set, the three-dimensional point cloud data, and the visual image data, construct a model through the stereo matching algorithm to obtain an initial dense texture point cloud model; based on the tightly coupled three-dimensional reconstruction algorithm, according to the initial dense texture point cloud model, the three-dimensional point cloud data, and the visual image data, perform data fusion through the geometric consistency algorithm to obtain the final fused point cloud model; based on the stereo vision dynamic compensation algorithm of the inertial navigation unit and the motion compensation method of the lidar, according to the inertial test data, perform dynamic parameter compensation on the final fused point cloud model to obtain an engineering structure multi-source model; Among them, the electronic device is further used for: Based on the three-dimensional point cloud data, a sparse point cloud model is obtained through a tightly coupled three-dimensional reconstruction algorithm for model construction; Based on the dense texture point cloud model, the sparse point cloud model is spatially aligned to obtain an aligned sparse point cloud model; According to the initial dense texture point cloud model and the aligned sparse point cloud model, a first fused point cloud model is obtained through a geometric consistency algorithm for matching and fusion; According to the three-dimensional point cloud data, geometric structure reconstruction is performed on the first fused point cloud model to obtain a second fused point cloud model; According to the visual image data, color rendering is performed on the second fused point cloud model to obtain a final fused point cloud model.

8. An engineering structure reconstruction system, characterized in that, The engineering structure reconstruction system includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that Program code is stored in the computer-readable storage medium, and the program code can be called by the processor to execute the method described in any one of claims 1 to 6.

Images