Engineering structure reconstruction method fusing laser radar, inertial navigation unit and stereoscopic vision

Abstract

The invention relates to the technical field of engineering structure three-dimensional reconstruction, in particular to an engineering structure reconstruction method and system fusing a laser radar, an inertial navigation unit and stereoscopic vision. The method comprises the following steps: the unmanned aerial vehicle carries a laser radar, an inertial navigation unit and stereoscopic vision equipment; the ROS system based on the unmanned aerial vehicle subscribes sensor topics to obtain a sensor calibration parameter set; according to the sensor calibration parameter set, the three-dimensional point cloud data and the visual image data, performing model construction through a stereo matching algorithm to obtain an initial dense texture point cloud model; based on a tight coupling three-dimensional reconstruction algorithm, fusing data through a geometric consistency algorithm to obtain a final fused point cloud model; and performing dynamic parameter compensation based on a stereoscopic vision dynamic compensation algorithm of an inertial navigation unit and a motion compensation method of the laser radar to obtain an engineering structure multi-source model. The engineering structure three-dimensional reconstruction method combines the laser radar, the inertial navigation unit and stereoscopic vision, and is high in precision and rich in texture.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction of engineering structures, and in particular to an engineering structure reconstruction method and system integrating laser radar, inertial navigation unit and stereo vision. Background Art

[0002] There are many existing engineering structures in my country. With the continuous development of transportation and residential infrastructure, the number and scale of engineering structures are also increasing year by year. However, the maintenance and management of engineering structures face many challenges. At present, 3D reconstruction of engineering structures plays an important role in the detection, evaluation and maintenance of engineering structures, but the existing 3D reconstruction technology of engineering structures mainly relies on manual measurement and traditional modeling methods.

[0003] Traditional manual measurement methods require professionals to conduct on-site surveys. The measurement process is cumbersome, time-consuming, and inefficient, especially when facing large-scale engineering structures or complex structures. The problem is more prominent. Due to the limitations of manual measurement methods, the accuracy of data acquisition is limited, resulting in low accuracy of the generated three-dimensional model, which is difficult to meet the requirements of high-precision detection and evaluation. In addition, the existing modeling methods often perform poorly when dealing with complex geometric structures and details of engineering structures, and cannot truly reflect the actual situation of engineering structures. In the process of three-dimensional reconstruction of engineering structures, it is necessary to integrate data from different sensors (such as lidar, inertial navigation unit, and stereo vision) to obtain complete engineering structure and texture information. However, the spatiotemporal synchronization of data and external parameter calibration between different sensors is a technical problem, and the existing methods are insufficient in terms of efficiency and accuracy.

[0004] In the prior art, there is a lack of a high-precision and texture-rich three-dimensional reconstruction method for engineering structures that combines lidar, inertial navigation unit and stereo vision. Summary of the invention

[0005] In order to solve the technical problem of low accuracy of reconstructed three-dimensional models in the prior art, an embodiment of the present invention provides a method and system for reconstructing an engineering structure by integrating a laser radar, an inertial navigation unit and stereo vision. The technical solution is as follows:

[0006] On the one hand, a method for reconstructing an engineering structure by integrating a laser radar, an inertial navigation unit, and a stereoscopic vision is provided. The method is implemented by an engineering structure reconstruction system, and the method includes:

[0007] The drone is equipped with a laser radar, an inertial navigation unit and a stereo vision device; based on the ROS system of the drone, a sensor topic is subscribed to obtain a sensor calibration parameter set; according to the sensor calibration parameter set, a joint calibration is performed using a method based on no calibration object and a method based on fully automatic corner point capture to obtain a joint calibration parameter;

[0008] According to the joint calibration parameters, the drone is used to shoot the reconstruction project according to a preset close-up path strategy; the laser radar is used to collect data to obtain three-dimensional point cloud data; the inertial navigation unit is used to collect data to obtain inertial test data; and the stereoscopic vision device is used to collect data to obtain visual image data;

[0009] According to the sensor calibration parameter set, the three-dimensional point cloud data and the visual image data, a model is constructed by a stereo matching algorithm to obtain an initial dense texture point cloud model;

[0010] Based on a tightly coupled 3D reconstruction algorithm, data fusion is performed using a geometric consistency algorithm according to the initial dense texture point cloud model, the 3D point cloud data and the visual image data to obtain a final fused point cloud model;

[0011] Based on the stereoscopic vision dynamic compensation algorithm of the inertial navigation unit and the motion compensation method of the laser radar, dynamic parameter compensation is performed on the final fused point cloud model according to the inertial test data to obtain a multi-source model of the engineering structure.

[0012] On the other hand, an engineering structure reconstruction system integrating laser radar, inertial navigation unit and stereo vision is provided, and the system is applied to an engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision, and the system comprises:

[0013] The drone is used to carry a laser radar, an inertial navigation unit and a stereo vision device; based on the ROS system of the drone, subscribe to the sensor topic to obtain a sensor calibration parameter set; according to the joint calibration parameters, the drone is used to shoot the reconstruction project according to a preset close-up path strategy;

[0014] The laser radar is used to collect data through the laser radar to obtain three-dimensional point cloud data;

[0015] The inertial navigation unit is used to collect data through the inertial navigation unit to obtain inertial test data;

[0016] The stereoscopic vision device is used to collect data according to the stereoscopic vision device to obtain visual image data;

[0017] The electronic device is used to perform joint calibration based on the sensor calibration parameter set, using a method based on no calibration objects and a method based on fully automatic corner point capture to obtain joint calibration parameters; to construct a model based on the sensor calibration parameter set, the three-dimensional point cloud data and the visual image data through a stereo matching algorithm to obtain an initial dense texture point cloud model; based on a tightly coupled three-dimensional reconstruction algorithm, to perform data fusion through a geometric consistency algorithm based on the initial dense texture point cloud model, the three-dimensional point cloud data and the visual image data to obtain a final fused point cloud model; based on the stereo vision dynamic compensation algorithm of the inertial navigation unit and the motion compensation method of the laser radar, to perform dynamic parameter compensation on the final fused point cloud model according to the inertial test data to obtain a multi-source model of the engineering structure.

[0018] On the other hand, an engineering structure reconstruction system is provided, which includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned engineering structure reconstruction methods that integrate laser radar, inertial navigation unit and stereo vision is implemented.

[0019] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned engineering structure reconstruction methods integrating laser radar, inertial navigation unit and stereo vision.

[0020] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0021] The present invention proposes a method for reconstructing an engineering structure by integrating laser radar, inertial navigation unit and stereo vision, using inertial navigation unit nodes to assist laser radar nodes in constructing a high-precision geometric structure of the engineering structure; using inertial navigation unit nodes to assist stereo vision to give the engineering structure real color and texture information, providing technical support for the three-dimensional reconstruction of the engineering structure. In terms of three-dimensional reconstruction of engineering structures, the accuracy of the reconstructed model is improved, which facilitates the management and operation of engineering structures. In a complex environment of engineering structures, the three-dimensional reconstruction of engineering structures can be achieved more accurately and in more detail. The present invention is a method for three-dimensional reconstruction of engineering structures that combines laser radar, inertial navigation unit and stereo vision with high accuracy and rich texture. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a flow chart of an engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision provided by an embodiment of the present invention;

[0024] Figure 2 It is a block diagram of an engineering structure reconstruction system integrating a laser radar, an inertial navigation unit and stereo vision provided by an embodiment of the present invention;

[0025] Figure 3 It is a structural schematic diagram of an engineering structure reconstruction system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0027] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0028] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0029] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0030] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0031] The embodiment of the present invention provides a method for reconstructing an engineering structure by integrating a laser radar, an inertial navigation unit and stereo vision. The method can be implemented by an engineering structure reconstruction system, which can be a terminal or a server. Figure 1 The flowchart of the engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision is shown. The processing flow of the method may include the following steps:

[0032] S1. The UAV is equipped with a laser radar, an inertial navigation unit and a stereo vision device. The UAV-based ROS system subscribes to the sensor topic to obtain the sensor calibration parameter set. According to the sensor calibration parameter set, a joint calibration is performed using a method based on no calibration objects and a method based on fully automatic corner point capture to obtain the joint calibration parameters.

[0033] In a feasible implementation, the present invention uses unmanned equipment equipped with LiDAR, IMU, and stereo vision equipment. The drone ROS system subscribes to the calibration parameter topics of sensors such as LiDAR, camera, and IMU. The obtained sensor calibration parameter set includes camera intrinsic parameters (focal length, principal point coordinates, distortion coefficient), extrinsic parameters (relative position between camera and IMU), and IMU attitude information.

[0034] The equipment used is Velodyne 16 LiDAR, and the HI229 inertial navigation unit IMU was selected. It is used in conjunction with the LiDAR to reconstruct the geometric structure of the bridge model and collaborate with the binocular camera to render the texture of the bridge model.

[0035] The IMU is external or built-in in the LiDAR or stereo vision. The LiDAR is a three-dimensional type with more than 16 lines and plane local scanning or omnidirectional scanning, and its model is Velodyne 16. The stereo vision device is a high-resolution stereo vision device with a measuring distance of 0.5~10m.

[0036] Optionally, according to the sensor calibration parameter set, joint calibration is performed using a method based on no calibration object and a method based on fully automatic corner point capture to obtain joint calibration parameters, including:

[0037] According to the sensor calibration parameter set, the laser radar and the inertial navigation unit are jointly calibrated using a method based on no calibration object, the first joint calibration parameter;

[0038] According to the sensor calibration parameter set, the laser radar and stereo vision equipment are jointly calibrated using a fully automatic corner point capture method to obtain the second joint calibration parameters.

[0039] In a feasible implementation, the present invention uses Velodyne VLP-16 LiDAR and HI229 IMU, and uses ROS package to record the data of both. The two devices need to be fixed firmly and fully dynamically stimulated during the recording process, including but not limited to rotation, tilt and acceleration changes.

[0040] Make a preliminary estimate of the external parameters between LiDAR and IMU; determine the matching relationship between LiDAR data and IMU data, use the established data association for preliminary optimization, and fine-tune the relative position and attitude between sensors; based on the initial optimization, perform multiple rounds of iterative optimization until the external parameter parameters converge; the final output joint calibration result file, that is, the first joint calibration parameter is in .yaml format.

[0041] Based on the calibration-free method and ROS, specific target environment data is not included, but the two devices should be firmly fixed. Data recording should be carried out in an environment with many feature points and rich planes, such as a general office environment, to ensure that there is sufficient movement and rotation in the x, y, and z axes, while avoiding violent impact movements caused by excessive acceleration.

[0042] Autoware was used to calibrate the Velodyne VLP-16 LiDAR and stereo vision together. Specifically, the LiDAR coordinates were first transformed with the left and right cameras of the stereo vision to unify the coordinate system. After the individual transformations were completed, the two results were then jointly transformed to obtain the second joint calibration parameters.

[0043] In the fully automatic corner point grabbing method, the data collected based on the calibration toolbox and the calibration board must include the point cloud and image ROS topic data packets mapped on the calibration board by the LiDAR and stereo vision in the environment. Data outside the calibration board should be discarded.

[0044] S2. According to the joint calibration parameters, the UAV is used to shoot the reconstruction project according to the preset close-up path strategy; the laser radar is used to collect data to obtain three-dimensional point cloud data; the inertial navigation unit is used to collect data to obtain inertial test data; the stereo vision equipment is used to collect data to obtain visual image data.

[0045] In a feasible real-time method, the preset "three-sided surround" close-up path strategy of the present invention includes using unmanned equipment to plan the collection path directly above and around the engineering structure, and the close-up distance should be greater than 0.5 meters and less than 10 meters.

[0046] The “three-sided surround” close-up path strategy needs to be planned through the structural engineering facade path terrain-mimicking route algorithm or related software, which involves making relevant adjustments to the path based on the surface shape of the structural engineering, and ultimately ensuring that the path route overlap rate is greater than 60%, and the side and top paths and routes are generated in an integrated manner.

[0047] The "three-sided encirclement" data collection path strategy takes into account multiple principles to ensure that the UAV's flight path can fully cover all parts of the engineering structure, including directly above, on both sides and around the engineering structure; ensure that the safety distance is above 0.5m; ensure multi-angle and multi-height data collection methods as much as possible; and acquire data in an engineering structure environment with sufficient light, low wind speed, few people and vehicles, and no obstructions.

[0048] Based on the path scheme, the LiDAR, IMU and stereo vision node data of the engineering structure to be reconstructed are collected at close range and synchronously. The data collection technology adopts the ROS-based sensor topic subscription method to obtain multi-source data of the real engineering structure.

[0049] The computing unit of the unmanned equipment is a lightweight airborne computer. It uses the computing unit to communicate with each sensor module through the ROS platform and subscribe to related ROS topics to obtain the necessary multi-sensor data, including LiDAR point cloud, stereo vision images of the binocular camera and IMU data, while recording engineering structure multi-source data packets of no less than 150s.

[0050] S3. According to the sensor calibration parameter set, three-dimensional point cloud data and visual image data, a model is constructed through a stereo matching algorithm to obtain an initial dense texture point cloud model.

[0051] Optionally, a model is constructed by a stereo matching algorithm according to a sensor calibration parameter set, three-dimensional point cloud data and visual image data to obtain an initial dense texture point cloud model, including:

[0052] Based on the sensor calibration parameter set and visual image data, a stereo matching algorithm is used to match the structural points of the left and right images to obtain a disparity map.

[0053] Based on the three-dimensional point cloud data, the pixel points in the disparity map are converted into three-dimensional coordinates to obtain dense point cloud data;

[0054] Construct a model according to the dense point cloud data to obtain a dense point cloud model;

[0055] According to the visual image data, the dense point cloud model is texture filled to obtain an initial dense texture point cloud model.

[0056] In one possible implementation, the present invention uses a stereo matching algorithm to track the camera pose by minimizing the radiometric difference between a point in the map and a sparse set of pixels in the current image.

[0057] The present invention uses a classic stereo matching algorithm to process the left and right images. The algorithm generates a disparity map by finding matching point pairs of the same structural points in the left and right images based on the disparity principle.

[0058] By converting each pixel in the disparity map into a three-dimensional coordinate, a dense point cloud is generated. Texture information extracted from the original image is added to the generated point cloud model, thereby generating a dense textured point cloud model with high resolution and fine texture. Through the above steps, the initial dense textured point cloud model is generated.

[0059] S4. Based on the tightly coupled 3D reconstruction algorithm, the initial dense texture point cloud model, 3D point cloud data and visual image data are fused through the geometric consistency algorithm to obtain the final fused point cloud model.

[0060] Optionally, based on a tightly coupled 3D reconstruction algorithm, data fusion is performed using a geometric consistency algorithm according to the initial dense texture point cloud model, the 3D point cloud data and the visual image data to obtain a final fused point cloud model, including:

[0061] Based on the 3D point cloud data, the model is constructed by tightly coupling the 3D reconstruction algorithm to obtain a sparse point cloud model;

[0062] Based on the dense texture point cloud model, the sparse point cloud model is spatially aligned to obtain the aligned sparse point cloud model;

[0063] According to the initial dense texture point cloud model and the aligned sparse point cloud model, matching and fusion are performed through a geometric consistency algorithm to obtain a first fused point cloud model;

[0064] Reconstructing the geometric structure of the first fused point cloud model according to the three-dimensional point cloud data to obtain a second fused point cloud model;

[0065] According to the visual image data, the second fused point cloud model is rendered in color to obtain the final fused point cloud model.

[0066] In a feasible implementation, the present invention uses a geometric consistency algorithm to initially match the point cloud data of different sensors, performs local geometric consistency checks and global optimization based on the nearest neighbor search matching method, eliminates mismatched points and fuses the point clouds to generate a first fused point cloud model.

[0067] The 3D point cloud data is registered to the first fused point cloud model to reconstruct the geometric structure, and the pixel colors in the image are rendered to the points in the second fused point cloud model in combination with stereo vision to restore the radiation information.

[0068] S5, based on the stereo vision dynamic compensation algorithm of the inertial navigation unit and the motion compensation method of the laser radar, performs dynamic parameter compensation on the final fused point cloud model according to the inertial test data to obtain a multi-source model of the engineering structure.

[0069] Optionally, based on the stereoscopic vision dynamic compensation algorithm of the inertial navigation unit and the motion compensation method of the laser radar, dynamic parameter compensation is performed on the final fused point cloud model according to the inertial test data to obtain a multi-source model of the engineering structure, including:

[0070] Based on the stereoscopic vision dynamic compensation algorithm, motion compensation is performed according to the inertial test data to obtain the dynamic compensation of the inertial navigation unit;

[0071] According to the dynamic compensation of the inertial navigation unit, the final fused point cloud model is dynamically compensated by the motion compensation method of the laser radar to obtain the calibration external parameter set;

[0072] According to the calibration external parameter set, the parameters of the final fused point cloud model are set to obtain a preliminary multi-source model of the engineering structure;

[0073] The multi-sensor spatial pose alignment is performed on the preliminary engineering structure multi-source model to obtain the engineering structure multi-source model.

[0074] In a feasible implementation, the present invention uses the IMU state data back propagation to perform motion compensation in the point cloud scanning data input of the multi-line LiDAR Velodyne VLP-16 under the Ubuntu 18.04 development environment. The three-dimensional points accumulated in the three-dimensional model of the engineering structure form a geometric structure, and the IMU data assists the LiDAR in motion compensation.

[0075] Based on the Rviz visualization interface in ROS and the final fused point cloud model, the previously acquired multi-source data of the engineering structure to be reconstructed are played as data packets; the obtained three external parameter calibration files are set in the engineering structure three-dimensional reconstruction deployment environment; after the data packet is played, the multi-source model of the engineering structure is saved in .pcd format.

[0076] The stereoscopic vision motion compensation algorithm is as follows (1), (2), (3):

[0077] (1);

[0078] (2);

[0079] (3);

[0080] Among them, point P represents each point of the final fused point cloud model, and W is the final state of stereo vision; is the attitude and position of the IMU; is the global linear velocity; is the bias of the gyroscope and accelerometer in the IMU; is the global gravitational acceleration; is the camera extrinsic parameter; It is the reciprocal of the camera exposure time; is the time offset; is the camera internal parameter; ( ) is the camera focal length, ( ) is the offset of the principal point from the upper left corner of the image plane.

[0081] In one possible implementation, and The operation representation will be mainly used to map state variables between manifold and tangent space. These two operations are often used to deal with optimization problems in non-Euclidean space, such as pose estimation.

[0082] The motion compensation method is as follows (4) and (5):

[0083] (4);

[0084] (5);

[0085] Among them, MAP is the Maximum A-Posteriori estimate; is the squared Mahalanobis distance of the covariance; is the state estimate propagated by the IMU; is the state covariance propagated by the IMU; is the LiDAR measurement residual; is the residual The Jacobi; is the total noise; A mapping operation to transform state variables from the manifold to the tangent space; It is a mapping operation that transforms the state variables from the tangent space to the manifold.

[0086] In a feasible implementation, a stereoscopic vision dynamic compensation algorithm based on an inertial navigation unit selects a certain number of tracked points from the three-dimensional model of the engineering structure and projects these points into the current image data of the stereoscopic vision. The photometric error of these points is minimized, and the state estimate of the stereoscopic vision-IMU is iteratively updated within the filter framework. In this process, the texture color of a single engineering structure model point is used to calculate the photometric error. The model points are tracked based on the frame-to-frame optical flow method, and the system state is estimated and updated through the photometric error method. After the state estimate is stable, texture rendering is performed to update the color of the points in the engineering structure model.

[0087] The present invention proposes a method for reconstructing an engineering structure by integrating laser radar, inertial navigation unit and stereo vision, using inertial navigation unit nodes to assist laser radar nodes in constructing a high-precision geometric structure of the engineering structure; using inertial navigation unit nodes to assist stereo vision to give the engineering structure real color and texture information, providing technical support for the three-dimensional reconstruction of the engineering structure. In terms of three-dimensional reconstruction of engineering structures, the accuracy of the reconstructed model is improved, which facilitates the management and operation of engineering structures. In a complex environment of engineering structures, the three-dimensional reconstruction of engineering structures can be achieved more accurately and in more detail. The present invention is a method for three-dimensional reconstruction of engineering structures that combines laser radar, inertial navigation unit and stereo vision with high accuracy and rich texture.

[0088] Figure 2 The block diagram of an engineering structure reconstruction system integrating laser radar, inertial navigation unit and stereo vision is shown according to an exemplary embodiment. The system is used for an engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision. Figure 2 The system includes a drone 210, a laser radar 220, an inertial navigation unit 230, a stereoscopic vision device 240, and an electronic device 250. Among them:

[0089] The drone 210 is used to carry a laser radar, an inertial navigation unit, and a stereo vision device on the drone; the drone subscribes to a sensor topic based on the ROS system to obtain a sensor calibration parameter set; and according to the joint calibration parameters, the drone is used to shoot the reconstruction project according to a preset close-up path strategy;

[0090] A laser radar 220 is used to collect data through a laser radar to obtain three-dimensional point cloud data;

[0091] An inertial navigation unit 230 is used to collect data through the inertial navigation unit to obtain inertial test data;

[0092] A stereoscopic vision device 240, used for collecting data according to the stereoscopic vision device to obtain visual image data;

[0093] The electronic device 250 is used to perform joint calibration based on a calibration-free method and a corner point fully automatic grasping method according to a sensor calibration parameter set to obtain joint calibration parameters; to construct a model through a stereo matching algorithm based on the sensor calibration parameter set, three-dimensional point cloud data and visual image data to obtain an initial dense texture point cloud model; to perform data fusion through a geometric consistency algorithm based on the initial dense texture point cloud model, three-dimensional point cloud data and visual image data based on a tightly coupled three-dimensional reconstruction algorithm to obtain a final fused point cloud model; to perform dynamic parameter compensation on the final fused point cloud model based on the stereo vision dynamic compensation algorithm of the inertial navigation unit and the motion compensation method of the laser radar according to the inertial test data to obtain a multi-source model of the engineering structure.

[0094] Optionally, the electronic device 250 is further configured to:

[0095] According to the sensor calibration parameter set, the laser radar and the inertial navigation unit are jointly calibrated using a method based on no calibration object, the first joint calibration parameter;

[0096] According to the sensor calibration parameter set, the laser radar and stereo vision equipment are jointly calibrated using a fully automatic corner point capture method to obtain the second joint calibration parameters.

[0097] Optionally, the electronic device 250 is further configured to:

[0098] Based on the sensor calibration parameter set and visual image data, a stereo matching algorithm is used to match the structural points of the left and right images to obtain a disparity map.

[0099] Based on the three-dimensional point cloud data, the pixel points in the disparity map are converted into three-dimensional coordinates to obtain dense point cloud data;

[0100] Construct a model according to the dense point cloud data to obtain a dense point cloud model;

[0101] According to the visual image data, the dense point cloud model is texture filled to obtain an initial dense texture point cloud model.

[0102] Optionally, the electronic device 250 is further configured to:

[0103] Based on the 3D point cloud data, the model is constructed by tightly coupling the 3D reconstruction algorithm to obtain a sparse point cloud model;

[0104] Based on the dense texture point cloud model, the sparse point cloud model is spatially aligned to obtain the aligned sparse point cloud model;

[0105] According to the initial dense texture point cloud model and the aligned sparse point cloud model, matching and fusion are performed through a geometric consistency algorithm to obtain a first fused point cloud model;

[0106] Reconstructing the geometric structure of the first fused point cloud model according to the three-dimensional point cloud data to obtain a second fused point cloud model;

[0107] According to the visual image data, the second fused point cloud model is rendered in color to obtain the final fused point cloud model.

[0108] Optionally, the electronic device 250 is further configured to:

[0109] Based on the stereoscopic vision dynamic compensation algorithm, motion compensation is performed according to the inertial test data to obtain the dynamic compensation of the inertial navigation unit;

[0110] According to the dynamic compensation of the inertial navigation unit, the final fused point cloud model is dynamically compensated by the motion compensation method of the laser radar to obtain the calibration external parameter set;

[0111] According to the calibration external parameter set, the parameters of the final fused point cloud model are set to obtain a preliminary multi-source model of the engineering structure;

[0112] The multi-sensor spatial pose alignment is performed on the preliminary engineering structure multi-source model to obtain the engineering structure multi-source model.

[0113] The stereoscopic vision motion compensation algorithm is as follows (1), (2), (3):

[0114] (1);

[0115] (2);

[0116] (3);

[0117] Among them, point P represents each point of the final fused point cloud model, and W is the final state of stereo vision; is the attitude and position of the IMU; is the global linear velocity; is the bias of the gyroscope and accelerometer in the IMU; is the global gravitational acceleration; is the camera extrinsic parameter; It is the reciprocal of the camera exposure time; is the time offset; is the camera internal parameter; ( ) is the camera focal length, ( ) is the offset of the principal point from the upper left corner of the image plane.

[0118] The motion compensation method is as follows (4) and (5):

[0119] (4);

[0120] (5);

[0121] Among them, MAP is the Maximum A-Posteriori estimate; is the squared Mahalanobis distance of the covariance; is the state estimate propagated by the IMU; is the state covariance propagated by the IMU; is the LiDAR measurement residual; is the residual The Jacobi; is the total noise; A mapping operation to transform state variables from the manifold to the tangent space; A mapping operation to transform state variables from tangent space to manifold

[0122] The present invention proposes a method for reconstructing an engineering structure by integrating laser radar, inertial navigation unit and stereo vision, using inertial navigation unit nodes to assist laser radar nodes in constructing a high-precision geometric structure of the engineering structure; using inertial navigation unit nodes to assist stereo vision to give the engineering structure real color and texture information, providing technical support for the three-dimensional reconstruction of the engineering structure. In terms of three-dimensional reconstruction of engineering structures, the accuracy of the reconstructed model is improved, which facilitates the management and operation of engineering structures. In a complex environment of engineering structures, the three-dimensional reconstruction of engineering structures can be achieved more accurately and in more detail. The present invention is a method for three-dimensional reconstruction of engineering structures that combines laser radar, inertial navigation unit and stereo vision with high accuracy and rich texture.

[0123] Figure 3 is a schematic diagram of a structural reconstruction system for an engineering structure provided by an embodiment of the present invention. Figure 3 As shown, the engineering structure reconstruction system may include the above Figure 2 The engineering structure reconstruction system shown integrates laser radar, inertial navigation unit and stereo vision. Optionally, the engineering structure reconstruction system 310 may include a first processor 2001.

[0124] Optionally, the engineering structure reconstruction system 310 may further include a memory 2002 and a transceiver 2003 .

[0125] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0126] Combine the following Figure 3 The components of the engineering structure reconstruction system 310 are introduced in detail:

[0127] The first processor 2001 is the control center of the engineering structure reconstruction system 310, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0128] Optionally, the first processor 2001 may execute various functions of the engineering structure reconstruction system 310 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0129] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0130] In a specific implementation, as an embodiment, the engineering structure reconstruction system 310 may also include multiple processors, such as Figure 3 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more systems, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0131] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0132] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage systems that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage systems that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage system, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently and accessed through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0133] The transceiver 2003 is used to communicate with the network system or the terminal system.

[0134] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0135] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and be reconstructed through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0136] It should be noted that Figure 3 The structure of the engineering structure reconstruction system 310 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition system may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0137] In addition, the technical effects of the engineering structure reconstruction system 310 can refer to the technical effects of the engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision described in the above method embodiment, which will not be repeated here.

[0138] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0139] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0140] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage system such as a server, data center, etc. that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0141] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0142] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0143] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0144] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0146] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0149] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer system (which can be a personal computer, server, or network system, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0150] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for reconstructing an engineering structure by integrating laser radar, inertial navigation unit and stereo vision, characterized in that: The method comprises: The drone is equipped with a laser radar, an inertial navigation unit and a stereo vision device; based on the ROS system of the drone, a sensor topic is subscribed to obtain a sensor calibration parameter set; according to the sensor calibration parameter set, a joint calibration is performed using a method based on no calibration object and a method based on fully automatic corner point capture to obtain a joint calibration parameter; According to the joint calibration parameters, the drone is used to shoot the reconstruction project according to a preset close-up path strategy; the laser radar is used to collect data to obtain three-dimensional point cloud data; the inertial navigation unit is used to collect data to obtain inertial test data; and the stereoscopic vision device is used to collect data to obtain visual image data; According to the sensor calibration parameter set, the three-dimensional point cloud data and the visual image data, a model is constructed by a stereo matching algorithm to obtain an initial dense texture point cloud model; Based on a tightly coupled 3D reconstruction algorithm, data fusion is performed using a geometric consistency algorithm according to the initial dense texture point cloud model, the 3D point cloud data and the visual image data to obtain a final fused point cloud model; Based on the stereoscopic vision dynamic compensation algorithm of the inertial navigation unit and the motion compensation method of the laser radar, dynamic parameter compensation is performed on the final fused point cloud model according to the inertial test data to obtain a multi-source model of the engineering structure.

2. The engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision according to claim 1 is characterized in that: The method of performing joint calibration based on the sensor calibration parameter set and using a method based on no calibration object and a method based on fully automatic corner point capture to obtain joint calibration parameters includes: According to the sensor calibration parameter set, the laser radar and the inertial navigation unit are jointly calibrated using a method based on no calibration object, to obtain a first joint calibration parameter; According to the sensor calibration parameter set, the laser radar and the stereo vision device are jointly calibrated using a corner point-based fully automatic capture method to obtain second joint calibration parameters.

3. The engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision according to claim 1 is characterized in that: The method of constructing a model by a stereo matching algorithm based on the sensor calibration parameter set, the three-dimensional point cloud data and the visual image data to obtain an initial dense texture point cloud model includes: Based on the sensor calibration parameter set, according to the visual image data, using a stereo matching algorithm to match the left and right image structure points to obtain a disparity map; Based on the three-dimensional point cloud data, pixels in the disparity map are converted into three-dimensional coordinates to obtain dense point cloud data; Building a model according to the dense point cloud data to obtain a dense point cloud model; According to the visual image data, the dense point cloud model is texture filled to obtain an initial dense texture point cloud model.

4. The engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision according to claim 1 is characterized in that: The method based on the tightly coupled 3D reconstruction algorithm, performing data fusion according to the initial dense texture point cloud model, the 3D point cloud data and the visual image data through a geometric consistency algorithm to obtain a final fused point cloud model, includes: According to the three-dimensional point cloud data, a model is constructed by a tightly coupled three-dimensional reconstruction algorithm to obtain a sparse point cloud model; Based on the dense texture point cloud model, the sparse point cloud model is spatially aligned to obtain an aligned sparse point cloud model; According to the initial dense texture point cloud model and the aligned sparse point cloud model, matching and fusing are performed through a geometric consistency algorithm to obtain a first fused point cloud model; Reconstructing the geometric structure of the first fused point cloud model according to the three-dimensional point cloud data to obtain a second fused point cloud model; The second fused point cloud model is color rendered according to the visual image data to obtain a final fused point cloud model.

5. The engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision according to claim 1 is characterized in that: The stereoscopic vision dynamic compensation algorithm based on the inertial navigation unit and the motion compensation method of the laser radar perform dynamic parameter compensation on the final fused point cloud model according to the inertial test data to obtain a multi-source model of the engineering structure, including: Based on the stereoscopic vision dynamic compensation algorithm, motion compensation is performed according to the inertial test data to obtain dynamic compensation of the inertial navigation unit; According to the dynamic compensation of the inertial navigation unit, dynamic parameter compensation is performed on the final fused point cloud model by a motion compensation method of a laser radar to obtain a calibration external parameter set; According to the calibrated external parameter set, parameters of the final fused point cloud model are set to obtain a preliminary engineering structure multi-source model; The preliminary engineering structure multi-source model is subjected to multi-sensor spatial pose alignment to obtain an engineering structure multi-source model.

6. The engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision according to claim 5 is characterized in that: The stereoscopic vision motion compensation algorithm is as follows: （1）； （2）； （3）； Among them, point P represents each point of the final fused point cloud model, and W is the final state of stereo vision; is the attitude and position of the IMU; is the global linear velocity; is the bias of the gyroscope and accelerometer in the IMU; is the global gravitational acceleration; is the camera extrinsic parameter; It is the reciprocal of the camera exposure time; is the time offset; is the camera internal parameter; ( ) is the camera focal length, ( ) is the offset of the principal point from the upper left corner of the image plane.

7. The engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision according to claim 5 is characterized in that: The motion compensation method is as follows: （4）； （5）； Among them, MAP is the Maximum A-Posteriori estimate; is the squared Mahalanobis distance of the covariance; is the state estimate propagated by the IMU; is the state covariance propagated by the IMU; is the LiDAR measurement residual; is the residual The Jacobi; is the total noise; A mapping operation to transform state variables from the manifold to the tangent space; It is a mapping operation that transforms the state variables from the tangent space to the manifold.

8. An engineering structure reconstruction system integrating laser radar, inertial navigation unit and stereo vision, wherein the engineering structure reconstruction system integrating laser radar, inertial navigation unit and stereo vision is used to implement the engineering structure reconstruction method integrating laser radar, inertial navigation unit and stereo vision as claimed in any one of claims 1 to 7, characterized in that: The system includes a drone, a laser radar, an inertial navigation unit, a stereo vision device and electronic equipment, wherein: The drone is used to carry a laser radar, an inertial navigation unit and a stereo vision device; based on the ROS system of the drone, subscribe to the sensor topic to obtain a sensor calibration parameter set; according to the joint calibration parameters, the drone is used to shoot the reconstruction project according to a preset close-up path strategy; The laser radar is used to collect data through the laser radar to obtain three-dimensional point cloud data; The inertial navigation unit is used to collect data through the inertial navigation unit to obtain inertial test data; The stereoscopic vision device is used to collect data according to the stereoscopic vision device to obtain visual image data; The electronic device is used to perform joint calibration based on the sensor calibration parameter set, using a method based on no calibration objects and a method based on fully automatic corner point capture to obtain joint calibration parameters; to construct a model based on the sensor calibration parameter set, the three-dimensional point cloud data and the visual image data through a stereo matching algorithm to obtain an initial dense texture point cloud model; based on a tightly coupled three-dimensional reconstruction algorithm, to perform data fusion through a geometric consistency algorithm based on the initial dense texture point cloud model, the three-dimensional point cloud data and the visual image data to obtain a final fused point cloud model; based on the stereo vision dynamic compensation algorithm of the inertial navigation unit and the motion compensation method of the laser radar, to perform dynamic parameter compensation on the final fused point cloud model according to the inertial test data to obtain a multi-source model of the engineering structure.

9. An engineering structure reconstruction system, characterized in that: The engineering structure reconstruction system comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images