Bridge structure unmanned aerial vehicle inspection slam modeling method of multi-modal data fusion

Abstract

The application provides a multi-modal data fusion bridge structure unmanned aerial vehicle inspection SLAM modeling method.The method comprises the following steps: step one, establishing a multi-modal data fusion bridge structure unmanned aerial vehicle inspection SLAM architecture; step two, proposing a multi-modal data fusion sliding window optimization method based on marginalization; and step three, verifying the feasibility of large bridge unmanned aerial vehicle inspection and three-dimensional reconstruction application.The application realizes the functions of photographing inspection of the outer surface of the bridge structure, three-dimensional reconstruction of the bridge structure, spatial perception of the unmanned aerial vehicle and autonomous exploration of the unmanned aerial vehicle.

Description

Technical Field

[0001] This invention relates to the technical fields of bridge UAV inspection, SLAM, multimodal data fusion, autonomous navigation, and 3D reconstruction applications, and in particular to a SLAM modeling method for bridge structure UAV inspection using multimodal data fusion. Background Technology

[0002] Simultaneous Localization and Real-Time Mapping (SLAM) is a key technology for robots and autonomous vehicles. It allows devices to build maps in unknown environments while simultaneously determining their own position. The core of SLAM technology lies in achieving environmental perception and pose estimation through sensor data (such as LiDAR and cameras). As a crucial technology for autonomous navigation systems, achieving real-time environmental perception and mapping of unknown environments using the above methods presents the following challenges in the practical application of SLAM technology for unmanned aerial vehicles (UAVs):

[0003] (1) The classic SLAM model is based on the assumption of a fixed scene, which is sensitive to dynamic targets and sudden changes in the environment, and is prone to feature extraction failure or pose estimation deviation.

[0004] (2) When traditional methods rely on a single sensor in high-speed motion and low-light scenarios, the positioning accuracy decreases significantly;

[0005] (3) Multimodal data fusion was carried out by introducing laser point cloud, vision and IMU data, but the real-time performance of dynamic target recognition and static element differentiation still needs to be improved;

[0006] (4) Existing semantic segmentation mechanisms are not robust enough in complex scenarios.

[0007] In recent years, the field of automated structural inspection has seen a surge in robotics research. In intelligent bridge structure monitoring technology, a drone flight platform equipped with an omnidirectional laser scanning unit has been used to estimate the drone's odometry and reconstruct the target bridge in 3D using SLAM technology. This enables safe, efficient, and accurate assessment of the target, providing an innovative solution for the safety assessment of critical bridge infrastructure and offering a new approach to intelligent drone inspection and health monitoring of bridge structures. However, this method still faces limitations, such as limited validation testing in short-distance bridge scenarios with simple motion trajectories and a primary focus on ground-based mobile platforms. These inherent problems become particularly prominent when dealing with actual large-scale bridge drone inspection and 3D modeling, posing new challenges for achieving SLAM modeling of bridge structure drone inspections.

[0008] (1) Drones face challenges such as turbulent disturbances, structural obstruction, and strong wind interference. Their attitude control accuracy is insufficient in highly dynamic environments, making it difficult to ensure the safe operation of the equipment.

[0009] (2) The shielding effect of the steel-concrete structure at the bottom of the bridge on electromagnetic waves, the signal attenuation of satellite navigation system (including conventional GPS and high-precision RTK) and the drift of inertial navigation system data (INS) have not been completely solved, resulting in the need to improve the positioning reliability in the absence of GPS.

[0010] (3) There are still technical difficulties in the spatiotemporal registration and error calibration of different modal data of laser geometric features and visual semantic features, and complete synergistic optimization has not yet been achieved.

[0011] To address the aforementioned issues, this invention proposes a multimodal data fusion-based SLAM modeling method for UAV inspection of bridge structures, enabling UAVs to photograph and inspect the outer surface of bridge structures, perform 3D reconstruction of bridge structures, and enable UAVs to perform spatial perception and autonomous exploration. Summary of the Invention

[0012] The purpose of this invention is to solve the problem that satellite navigation signals cannot be used in the area due to the shielding effect of the steel-concrete structure at the bottom of the bridge on electromagnetic waves. A multimodal data fusion-based SLAM modeling method for UAV inspection of bridge structures is proposed.

[0013] This invention is achieved through the following technical solution: This invention proposes a multimodal data fusion-based SLAM modeling method for UAV inspection of bridge structures, the method comprising:

[0014] Step 1: Establish a multimodal data fusion-based SLAM architecture for UAV inspection of bridge structures; specifically:

[0015] Step 11: Establish the ESIKF framework based on sequential update error state iterative Kalman filter to tightly couple the UAV's lidar, image, and inertial measurement.

[0016] Steps 1 and 2: Establish a visual measurement model for UAV inspection and achieve iterative updates of visual status;

[0017] Step 2: Propose a sliding window optimization method for edge-based multimodal data fusion;

[0018] Step 3: Verify the feasibility of using UAVs for large bridge inspection and 3D reconstruction; specifically, this includes verifying the feasibility of SLAM loop closure detection and multimodal perception data fusion in UAV-based intelligent structure inspection based on non-loop point cloud SLAM and loop closure point cloud SLAM.

[0019] Furthermore, in step one, using and Operations are used to represent manifolds, for have:

[0020]

[0021] In the formula, The exponential and logarithmic operations represent a two-way mapping between rotation matrices and rotation vectors via the Rodriguez formula;

[0022] In this sequentially updated error state iterative Kalman filter (ESIKF) framework, it is assumed that the time offsets between the three sensors of the UAV are known and can be calibrated and synchronized; the IMU frame is used as the UAV body frame, and the first UAV body frame is used as the UAV global frame; the discrete state transition model for the i-th UAV IMU measurement is:

[0023]

[0024] In the formula, Δt is the IMU sampling period, and the state x, input u, process noise w, and function f are defined as follows:

[0025]

[0026] In the formula, G R I , G p I and G v I These represent the IMU attitude, position, and velocity in the global frame of the UAV, respectively. G g is the gravity vector in the drone's global frame, τ is the reciprocal camera exposure time relative to the drone's first frame, and n τ This involves modeling τ as Gaussian noise from a random walk, and ω... m and a m These are the original IMU measurements from the drone, n g and n a It is ω m and a m The noise measured by the drone, b a and b g This is the IMU bias of the UAV, which is modeled as being composed of Gaussian noise n bg and n ba Driven random walk;

[0027] Next, the UAV IMU data is propagated forward; the UAV IMU propagation status is as follows. Prior and covariance Prior to time t k A prior distribution is imposed on the system state x, as shown below:

[0028]

[0029] Let the prior distribution above be represented as p(x), and let the LiDAR and camera measurement model of the UAV be represented as:

[0030]

[0031] In the formula, v l ~N(0, Σv l ) and v c ~N(0, Σv c The numbers ) represent the measurement noise of the UAV's LiDAR and camera, respectively.

[0032] To fuse LiDAR or image measurements from a UAV into q(x|y)∝q(y|x), for the prior distribution q(x), denoted as q(x) in With the LiDAR update of the drone, The state and covariance are obtained from the forward propagation of the UAV IMU; in the case of visual updates... The convergent state and covariance are obtained from the UAV LiDAR update; to obtain the measurement model distribution q(y|x), the estimated state at the κ-th iteration is represented as... in Through its The first-order Taylor expansion performed at the point approximates the measurement model formula (5), yielding:

[0033]

[0034] In the formula, z κ It is a residual. It is lumped measurement noise, H κ and L κ yes Relative to δx κ The Jacobian matrices of v are evaluated at zero; then, substituting the prior distribution q(x) and the measurement distribution q(y|x) into the posterior distribution q(x|y)∝q(y|x)q(x) and performing maximum likelihood estimation, δx can be obtained from the standard update step in the sequentially updated error state iterative Kalman filter framework. κ and x κ Maximum a posteriori estimate:

[0035]

[0036] Finally, the convergent state and covariance matrix make the mean and covariance of the posterior distribution q(x|y).

[0037] Furthermore, steps one and two specifically include:

[0038] Use extracted drone visual map points G p iTo build a visual measurement model; when mapping drone points G p i Transformed into a current image of the UAV with real-time ground conditions (I) k When (·), the photometric error between the UAV reference plane and the current plane is zero:

[0039]

[0040] In the formula, π(·) represents the projection model of the UAV camera. Cr TG is the UAV global frame reference frame C. r It has been estimated during the reception and fusion of reference frames. It is the affine distortion matrix that transforms the pixel from the i-th current block to the reference block, where Δu is the distance to the center u within the current block. i The relative pixel position, This represents the ground truth pixel values ​​of the UAV reference frame and the current frame, which are measured as the measurement noise v of the UAV. c =(δI) k ,δI r The actual image pixel value I k I r ,therefore:

[0041]

[0042] u in formula (10) i Move to u′ i As shown below:

[0043]

[0044] To estimate the drone's inverse exposure time τ k The initial inverse exposure time of the UAV is fixed at τ0 = 1 to eliminate the measurement state estimation when the inverse exposure time of all UAVs is zero; thus, the inverse exposure time estimated for subsequent UAV frames is relative to the exposure time of the first frame.

[0045] Furthermore, step two includes performing a drone-based sliding window edge detection process, specifically:

[0046] The drone optimization model is represented in the following general form:

[0047] HX=b (12)

[0048] The above general form can be broken down into the following forms:

[0049]

[0050] The purpose of disassembling is to remove the old frame X of the drone. mRemove the state variables while retaining the constraints of the UAV; the decomposed optimization problem is then triangulated using the Schur complement matrix, i.e.:

[0051]

[0052] At this point, Xr can be solved without relying on the drone's Xm:

[0053]

[0054] Furthermore, in the sliding window optimization method for multimodal data fusion, the residual of the sliding window based on UAV map localization consists of four parts:

[0055] a. Residuals of drone map matching pose and optimization variables;

[0056] b. Residuals of relative pose and optimization variables in UAV laser odometry;

[0057] c. Residuals of UAV IMU pre-integration and optimization variables;

[0058] d. Residuals corresponding to prior factors resulting from the marginalization of drones.

[0059] Furthermore, in step three, non-loop-free point cloud SLAM is used to establish a time-series-based incremental point cloud registration framework. Its core is to construct a global point cloud map through frame-by-frame registration; specifically, this includes:

[0060] (1) Point cloud preprocessing: The preprocessing stage includes voxel downsampling and normal estimation;

[0061] (2) Frame-by-frame registration: Using the first frame point cloud as the reference coordinate system, each subsequent frame point cloud is registered with the previous frame in sequence. The specific steps are coarse registration, fine registration, and registration quality evaluation and correction.

[0062] (3) Global point cloud construction: The transformation matrix obtained by registration of each frame of point cloud is transformed to the reference coordinate system, and finally all transformed point clouds are merged to form a globally consistent point cloud map.

[0063] Furthermore, in step three, loop-closure point cloud SLAM is employed to establish a globally consistent framework based on pose graph optimization. By constructing a pose graph that includes temporal constraints and loop closure constraints, the poses of all frames are globally optimized to eliminate accumulated errors. Specifically, this includes:

[0064] (1) Point cloud preprocessing: The preprocessing stage includes voxel downsampling and normal estimation to ensure the processing efficiency and registration reliability of point cloud data;

[0065] (2) Pose graph construction: The pose graph consists of nodes and edges. Nodes represent the pose of each frame of point cloud in the global coordinate system and the relative pose constraints between point clouds, respectively.

[0066] (3) Global optimization: The Levenberg-Marquardt algorithm is used to perform global optimization on the pose graph to minimize the error function of all edges;

[0067] (4) Global point cloud construction: Based on the optimized pose matrix, the point clouds of each frame are transformed into the global coordinate system, and all point clouds are merged to form a globally consistent high-precision map.

[0068] Furthermore, in step three, the bridge model reconstructed from the 3D data of the large bridge UAV inspection is subjected to enhanced contour information processing. Specifically, the edge information of the image is enhanced and the contour features of the image are highlighted through image sharpening, image smoothing and edge detection contour enhancement methods.

[0069] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multimodal data fusion bridge structure UAV inspection SLAM modeling method.

[0070] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the bridge structure UAV inspection SLAM modeling method based on multimodal data fusion.

[0071] The beneficial effects of this invention are:

[0072] (1) This invention establishes a multimodal data fusion SLAM architecture for bridge structure UAV inspection, which efficiently fuses IMU, lidar and image measurement data collected by bridge structure UAV inspection based on sequentially updated error state iterative Kalman filter (ESIKF);

[0073] (2) The present invention establishes a visual measurement model for UAV inspection, which realizes another update across visual states. It starts with a coarse level to update the visual state and achieves a more refined visual state estimation after iterative convergence.

[0074] (3) The present invention performs a sliding window edge-out process based on UAVs, removes old key frames of UAVs from the sequentially updated error state iterative Kalman filter, and retains the constraints about UAVs, thereby achieving a high efficiency improvement.

[0075] (4) This invention proposes a sliding window optimization method for a multimodal data fusion system based on edge computing. As the positioning process progresses, the sliding window establishes the Jacobian matrix of each residual with respect to the UAV state variables, thereby achieving a high efficiency improvement.

[0076] (5) This invention establishes an incremental point cloud registration framework based on time series, constructs a global point cloud map by registering frame by frame, and establishes a global consistency framework based on pose graph optimization on this basis. By constructing a pose graph containing time constraints and closed-loop constraints, the pose of all frames is globally optimized to eliminate accumulated errors and achieve high precision improvement.

[0077] (6) Based on the three-dimensional reconstruction of bridge models using large bridge UAV inspection data, this invention enhances the contour information processing of bridge models, improves the contour clarity of bridge components, suppresses noise interference, balances image sharpness and smoothness, enhances the geometric feature recognizability of the three-dimensional model, and provides a high-quality data foundation for UAV inspection. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0079] Figure 1 This is a schematic diagram of the overall architecture of the SLAM modeling method for UAV inspection of bridge structures based on multimodal data fusion.

[0080] Figure 2 This is a schematic diagram of the error state iterative Kalman filter (ESIKF) framework based on sequential updates.

[0081] Figure 3 This is a flowchart of the sliding window edge detection process based on drones.

[0082] Figure 4 This is a flowchart of the sliding window optimization process for a multimodal data fusion system.

[0083] Figure 5 This is a flowchart of SLAM based on unlooped point cloud data from large bridge drone inspections.

[0084] Figure 6 This is a flowchart of SLAM based on loopback point cloud data from large bridge drone inspections.

[0085] Figure 7 This is a flowchart for enhancing the outline of a 3D reconstruction model based on UAV inspection data of large bridges. Detailed Implementation

[0086] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0087] Combination Figures 1-7 This invention proposes a multimodal data fusion-based SLAM modeling method for UAV inspection of bridge structures, the method comprising:

[0088] Step 1: Establish a multimodal data fusion-based SLAM architecture for UAV inspection of bridge structures; specifically:

[0089] Step 11: Establish the ESIKF framework based on sequential update error state iterative Kalman filter to tightly couple the UAV's lidar, image, and inertial measurement.

[0090] In step one, use and Operations are used to represent manifolds, for have:

[0091]

[0092] In the formula, The exponential and logarithmic operations represent a two-way mapping between rotation matrices and rotation vectors via the Rodriguez formula;

[0093] In this sequentially updated error state iterative Kalman filter (ESIKF) framework, it is assumed that the time offsets between the three sensors of the UAV (LiDAR, IMU, and camera) are known and can be calibrated and synchronized; the IMU frame (denoted as I) is used as the UAV body frame, and the first UAV body frame is used as the UAV global frame (denoted as G); the discrete state transition model for the i-th UAV IMU measurement is:

[0094]

[0095] In the formula, Δt is the IMU sampling period, and the state x, input u, process noise w, and function f are defined as follows:

[0096]

[0097] In the formula, G R I , G p I and G v I These represent the IMU attitude, position, and velocity in the global frame of the UAV, respectively.G g is the gravity vector in the drone's global frame, τ is the reciprocal camera exposure time relative to the drone's first frame, and n τ This involves modeling τ as Gaussian noise from a random walk, and ω... m and a m These are the original IMU measurements from the drone, n g and n a It is ω m and a m The noise measured by the drone, b a and b g This is the IMU bias of the UAV, which is modeled as being composed of Gaussian noise n bg and n ba Driven random walk;

[0098] Next, the UAV IMU data is propagated forward; the UAV IMU propagation status is as follows. Prior and covariance Prior to time t k A prior distribution is imposed on the system state x, as shown below:

[0099]

[0100] Let the prior distribution above be represented as p(x), and let the LiDAR and camera measurement model of the UAV be represented as:

[0101]

[0102] In the formula, v l ~N(0, Σv l ) and v c ~N(0, Σv c The numbers ) represent the measurement noise of the UAV's LiDAR and camera, respectively.

[0103] To fuse LiDAR or image measurements from a UAV into q(x|y)∝q(y|x), for the prior distribution q(x), denoted as q(x) in With the LiDAR update of the drone, The state and covariance are obtained from the forward propagation of the UAV IMU; in the case of visual updates... The convergent state and covariance are obtained from the UAV LiDAR update; to obtain the measurement model distribution q(y|x), the estimated state at the κ-th iteration is represented as... in Through its The first-order Taylor expansion performed at the point approximates the measurement model formula (5), yielding:

[0104]

[0105] In the formula, z κ It is a residual. It is lumped measurement noise, H κ and L κ yes Relative to δx κ The Jacobian matrices of v are evaluated at zero; then, substituting the prior distribution q(x) and the measurement distribution q(y|x) into the posterior distribution q(x|y)∝q(y|x)q(x) and performing maximum likelihood estimation, δx can be obtained from the standard update step in the sequentially updated error state iterative Kalman filter framework. κ and x κ Maximum a posteriori estimate:

[0106]

[0107] Finally, the convergent state and covariance matrix make the mean and covariance of the posterior distribution q(x|y).

[0108] Steps 1 and 2: Establish a visual measurement model for UAV inspection and achieve iterative updates of visual status;

[0109] Steps one and two specifically include:

[0110] Use extracted drone visual map points G p i To build a visual measurement model; when mapping drone points G p i Transformed into a current image of the UAV with real-time ground conditions (I) k When (·), the photometric error between the UAV reference plane and the current plane is zero:

[0111]

[0112] In the formula, π(·) represents the projection model of the UAV camera. Cr TG is the UAV global frame reference frame C. r It has been estimated during the reception and fusion of reference frames. It is the affine distortion matrix that transforms the pixel from the i-th current block to the reference block, where Δu is the distance to the center u within the current block. i The relative pixel position, This represents the ground truth pixel values ​​of the UAV reference frame and the current frame, which are measured as the measurement noise v of the UAV. c =(δI) k ,δI r The actual image pixel value I k Ir ,therefore:

[0113]

[0114] u in formula (10) i Move to u′ i As shown below:

[0115]

[0116] To estimate the drone's inverse exposure time τ k The initial inverse exposure time τ0 = 1 for the UAV is fixed to eliminate measurement state estimation when the inverse exposure time of all UAVs is zero; thus, the inverse exposure time estimated for subsequent UAV frames is relative to the exposure time of the first frame. This process achieves another update across the visual state, starting with a coarse level of visual state update and achieving a more refined visual state estimation after iterative convergence.

[0117] Step 2: Propose a sliding window optimization method for edge-based multimodal data fusion;

[0118] Step two includes a sliding window edge-finding process based on the UAV. The sliding window removes older keyframes from the sequentially updated error state iterative Kalman filter because further optimization wouldn't significantly alter the process, while retaining the constraints related to the UAV. This operation is called "edge-finding" of the sliding window. The specific UAV-based sliding window edge-finding process is as follows: Figure 3 As shown. Specifically:

[0119] The drone optimization model is represented in the following general form:

[0120] HX=b (12)

[0121] The above general form can be broken down into the following forms:

[0122]

[0123] The purpose of disassembling is to remove the old frame X of the drone. m Remove the state variables while retaining the constraints of the UAV; the decomposed optimization problem is then triangulated using the Schur complement matrix, i.e.:

[0124]

[0125] At this point, Xr can be solved without relying on the drone's Xm:

[0126]

[0127] In the sliding window optimization method for multimodal data fusion, the optimization problem is expressed in the following form:

[0128] J T ΣJδx=-J T Σr (16)

[0129] r T =[r T0 r T1 r T2 r L0 r L1 r M0 r M1 (17)

[0130]

[0131]

[0132] In the formula, r is the residual, J is the Jacobian matrix of the residual with respect to the UAV state variables, and ∑ is the information matrix.

[0133] The residual of the sliding window based on UAV map positioning consists of four parts:

[0134] a. Residuals of drone map matching pose and optimization variables;

[0135] b. Residuals of relative pose and optimization variables in UAV laser odometry;

[0136] c. Residuals of UAV IMU pre-integration and optimization variables;

[0137] d. Residuals corresponding to prior factors resulting from the marginalization of drones.

[0138] Matrix multiplication can be written in summation form as follows:

[0139]

[0140] The sliding window optimization structure diagram of the multimodal data fusion system is shown below. Figure 4 As shown.

[0141] With a window length of 3, old frames need to be marginalized before adding new frames. In practice, this is done in two steps. Using factors independent of the amount to be marginalized, a Hessian matrix is ​​constructed corresponding to the remaining drone variables. The superscript 'a' indicates the drone variable from the first step. The complete expression including the Hessian matrix is:

[0142]

[0143] Identify the drone factors related to the quantities to be marginalized, construct the Hessian matrix to be marginalized, and use the Schur complement matrix for marginalization. The superscript 'b' indicates a variable from the second step. The complete expression including the Hessian matrix is:

[0144]

[0145] Ultimately, a two-step superposition is used. The complete formula for superimposing Hessian matrices is:

[0146] H rr δx r =b r (twenty three)

[0147]

[0148] As the positioning process proceeds, the process of "edge-finding old frames and adding new frames" is continuously repeated, thereby maintaining the window length unchanged.

[0149] Step 3: Verify the feasibility of using UAVs for large bridge inspection and 3D reconstruction; specifically, this includes verifying the feasibility of SLAM loop closure detection and multimodal perception data fusion in UAV-based intelligent structure inspection based on non-loop point cloud SLAM and loop closure point cloud SLAM.

[0150] In step three, loop-free point cloud SLAM is used to establish a time-series-based incremental point cloud registration framework. Its core is to construct a global point cloud map through frame-by-frame registration; specifically, it includes:

[0151] (1) Point cloud preprocessing: The preprocessing stage includes voxel downsampling and normal estimation;

[0152] (2) Frame-by-frame registration: Using the first frame point cloud as the reference coordinate system, each subsequent frame point cloud is registered with the previous frame in sequence. The specific steps are coarse registration, fine registration, and registration quality evaluation and correction.

[0153] (3) Global point cloud construction: The transformation matrix obtained by registration of each frame of point cloud is transformed to the reference coordinate system, and finally all transformed point clouds are merged to form a globally consistent point cloud map.

[0154] Voxel downsampling divides the point cloud into different voxel grids and uses the centroid of points within a voxel to represent that voxel, reducing the amount of point cloud data and lowering the computational complexity of subsequent registration, while preserving the overall geometric features of the point cloud.

[0155] Let the original point cloud P = {p i |i=1,2,…,n}, where p i =(x i ,y i ,zi Let be the i-th point in the point cloud. Divide the 3D space into a partition of size v. x ×v y ×v z A voxel grid, for the set of points within each voxel The representative point p of a voxel voxel The calculation formula is:

[0156]

[0157] Normal estimation provides the necessary normal vector information for subsequent point-to-plane ICP registration. The KD tree is used to search for neighborhood points, and then the normal is estimated by the least squares method.

[0158] For each point p in the point cloud i Its neighborhood point set is The normal is estimated by fitting the plane using the least squares method. The covariance matrix C is constructed as follows:

[0159]

[0160] In the formula, It is the centroid of the neighborhood points. Perform eigenvalue decomposition on C: C = UΛU T The eigenvector corresponding to the smallest eigenvalue is the point p. i normal n i .

[0161] In coarse registration, the Euclidean distance between the source point cloud and the corresponding point in the target point cloud after transformation is minimized through point-to-point ICP, which quickly converges to the approximate transformation matrix and alleviates the problem of initial pose deviation.

[0162] For the original point cloud P = {p i |i=1,2,…,n}, target point cloud Q={q j |j=1,2,…,m}, transformation matrix Where R is the rotation matrix and t is the translation vector. The goal of point-to-point ICP is to find T that minimizes the following error function:

[0163]

[0164] In fine registration, normal information is utilized through point-to-plane ICP. By minimizing the perpendicular distance from the point to the plane corresponding to the target point, the transformation matrix is ​​further optimized, thereby improving registration accuracy.

[0165] For the original point cloud P = {p i Each point p in |i=1,2,…,n} i normal n i Target point cloud Q = {q jThe corresponding point in |j=1,2,…,m} is q. j The objective function for the point-to-plane ICP is:

[0166]

[0167] In registration quality assessment, the fitness index is used to evaluate the registration effect; the closer the value is to 1, the better the registration effect. Whether re-registration is needed is determined by whether the fitness is less than 0.3.

[0168]

[0169] In global map construction, each frame of point cloud is transformed into the reference coordinate system through the transformation matrix obtained by registration, and then all transformed point clouds are merged to form a globally consistent point cloud map.

[0170] Let P be the point cloud of the i-th frame. i The transformation matrix obtained by registration is T i The transformed point cloud P i ′=T i P i The global point cloud computing formula is:

[0171]

[0172] In step three, loop-closure point cloud SLAM is used to establish a globally consistent framework based on pose graph optimization. By constructing a pose graph that includes temporal constraints and loop closure constraints, the poses of all frames are globally optimized to eliminate accumulated errors. Specifically, this includes:

[0173] (1) Point cloud preprocessing: The preprocessing stage includes voxel downsampling and normal estimation to ensure the processing efficiency and registration reliability of point cloud data;

[0174] (2) Pose graph construction: The pose graph consists of nodes and edges. Nodes represent the pose of each frame of point cloud in the global coordinate system (initially the cumulative transformation of adjacent registration) and the relative pose constraints between point clouds.

[0175] (3) Global optimization: The Levenberg-Marquardt algorithm is used to perform global optimization on the pose graph, minimizing the error function of all edges (based on the weighted sum of squares of the information matrix);

[0176] (4) Global point cloud construction: Based on the optimized pose matrix, the point clouds of each frame are transformed into the global coordinate system, and all point clouds are merged to form a globally consistent high-precision map.

[0177] In pose graph construction, nodes represent the pose of point clouds in each frame in the global coordinate system, providing the foundation for pose graph construction.

[0178] Let the pose of the point cloud in the i-th frame be T. i The initial cumulative transformation of adjacent registrations yields the pose as shown in T. i =T i-1 T i-1→i T i-1→i It is the relative transformation matrix from frame (i-1) to frame i.

[0179] Edges represent relative pose constraints between point clouds, and are divided into two categories: adjacency constraints and loop closure constraints. Adjacency constraints construct strong constraint edges on the time series, describing the relative pose relationship between point clouds in adjacent frames. Loop closure constraints are used to detect loops between non-adjacent frames, and weak constraint edges are added to suppress accumulated errors.

[0180] Relative transformation matrix T in adjacency constraints i→i+1 The information matrix Ω is obtained through point cloud registration between adjacent frames (point-to-point + point-to-plane ICP). i→i+1 It is the inverse of the error covariance:

[0181]

[0182] This covariance is used to estimate the registration of the point cloud based on the distance distribution of corresponding point pairs, σ. 2 I is the variance of the distance between corresponding point pairs, and I6 is a 6×6 identity matrix.

[0183] Initial transformation in closed-loop constraints The estimated relative pose is T i→j If the fitness is greater than 0.3, then add an edge.

[0184] In the global optimization, the pose graph is globally optimized by minimizing the total error function E based on the Levenberg-Marquardt algorithm, resulting in a globally consistent pose estimate.

[0185] Let the measured relative pose of edge (i,j) be... The estimated relative pose is The information matrix is ​​Ω ij Then the error vector e of edge (i,j) ij for:

[0186]

[0187] The total error function E of the pose graph is:

[0188]

[0189] In the formula, ε is the set of all edges in the pose graph.

[0190] In step three, the bridge model reconstructed from the 3D data of the large bridge UAV inspection undergoes contour information enhancement processing. To improve the visual quality of the image, contour enhancement methods such as image sharpening, image smoothing, and edge detection are used to enhance the edge information and highlight the contour features of the image. This improves the clarity of the bridge components' contours, suppresses noise interference, balances image sharpness and smoothness, and enhances the recognizability of the geometric features of the 3D model, providing a high-quality data foundation for UAV inspection.

[0191] In Laplacian sharpening, the second derivative operator is used to detect abrupt changes in pixel grayscale and enhance edge regions.

[0192] The Laplace operator is defined as: Here, f(x, y) represents the pixel value of the image at coordinates (x, y). By calculating the difference between the second derivative of a pixel and its neighborhood, edge regions with drastic grayscale changes are highlighted.

[0193] The Laplacian operator is superimposed on the original image, using the following formula:

[0194]

[0195] In the formula, k is the sharpening coefficient (usually k>0), which is used to adjust the intensity of edge enhancement.

[0196] Edge contrast is enhanced by using the difference between the original image and the blurred image in the unsharpened mask.

[0197] Applying a Gaussian filter G(σ) to the original image f blurs the image, resulting in a blurred image:

[0198] f blur :f blur =f*G(σ) (35)

[0199] In the formula, G(σ) is the Gaussian kernel with standard deviation σ, and * indicates convolution operation.

[0200] Calculate the difference (mask) between the original image and the blurred image:

[0201] mask=ff blur (36)

[0202] The mask is superimposed onto the original image with a weight α to achieve sharpening:

[0203] f sharpened =f + α·mask = f + α·(ff) blur (37)

[0204] In the formula, α is the enhancement coefficient (α≥1), which controls the degree of edge enhancement.

[0205] In the edge detection operator, edges are accurately extracted through multi-stage processing (filtering, gradient calculation, threshold segmentation).

[0206] In Gaussian filtering for noise reduction, a Gaussian kernel G is used to smooth the image and reduce noise interference.

[0207]

[0208] In calculating the gradient magnitude and direction, the Sobel operator is used to calculate the gradients in the x and y directions:

[0209]

[0210] The gradient magnitude M(x,y) and direction θ(x,y) are:

[0211]

[0212] By comparing adjacent pixels along the gradient direction, local maxima are preserved, and effective edges are filtered using a dual threshold (high threshold TH and low threshold TL) to ensure the continuity of the contour.

[0213] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multimodal data fusion bridge structure UAV inspection SLAM modeling method.

[0214] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the bridge structure UAV inspection SLAM modeling method based on multimodal data fusion.

[0215] The memory in this application embodiment can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM). It should be noted that the memory used in the methods described in this invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0216] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0217] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0218] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0219] The above provides a detailed description of the multimodal data fusion-based SLAM modeling method for UAV inspection of bridge structures proposed in this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A multimodal data fusion-based SLAM modeling method for UAV inspection of bridge structures, characterized in that, The method includes: Step 1: Establish a multimodal data fusion-based SLAM architecture for UAV inspection of bridge structures; specifically: Step 11: Establish the ESIKF framework based on sequential update error state iterative Kalman filter to tightly couple the UAV's lidar, image, and inertial measurement. Steps 1 and 2: Establish a visual measurement model for UAV inspection and achieve iterative updates of visual status; Step 2: Propose a sliding window optimization method for edge-based multimodal data fusion; Step 3: Verify the feasibility of using UAVs for inspection and 3D reconstruction of large bridges; specifically, this includes verifying the feasibility of SLAM loop closure detection and multimodal perception data fusion in UAV-based intelligent structure inspection based on non-loop point cloud SLAM and loop-closure point cloud SLAM. Steps one and two specifically include: Using extracted drone visual map points G p i to construct a visual measurement model; when the drone visual map points G p i are transformed into a drone current image with ground truth state I k (·) the photometric error between the drone reference face and the current face is zero: (10) In the formula, π (·) represents the drone camera projection model. Cr T G It is the global frame G reference frame of the drone. C r It has been estimated during the reception and fusion of reference frames. It is to change the pixel from the first i The affine twist matrix that transforms the current block to the reference block. Δu It is to the center of the current block. u i The relative pixel position, This represents the ground truth pixel values ​​of the UAV reference frame and the current frame, which are measured as UAV measurement noise. v c =(δI k ,δI r ） Actual image pixel values I k ，I r ,therefore: (11) In formula (10) u i Moving to As follows: To estimate the drone inverse exposure time τ k , fix the drone initial inverse exposure time τ 0 = 1 to eliminate the measurement state estimate when all drone inverse exposure times are zero; so that the inverse exposure times of subsequent drone frame estimates are all relative to the exposure time of the first frame.

2. The method according to claim 1, characterized in that, In step one, the operations and are used to represent the manifold, for we have: (1) wherein r ∈ , a , b ∈ , the exponential and logarithmic operations represent the bidirectional mapping of rotation matrices and rotation vectors through the Rodrigues formula; In the sequentially updated error state iterative Kalman filter (ESIKF) framework, it is assumed that the time offsets between the three sensors of the UAV are known and are calibrated and synchronized; the IMU frame is used as the UAV body frame, and the first UAV body frame is used as the UAV global frame; In the first i The discrete state transition model of the second UAV IMU measurement is: (2) where Δ t is the IMU sampling period, the state x , the input u , the process noise w and the function f are defined as follows: ； ； ； (3) In the formula, G R I , G p I and G v I These represent the IMU attitude, position, and velocity in the global frame of the UAV, respectively. G g It is the gravity vector in the global frame of the drone. τ It is the countdown time of the camera exposure relative to the first frame of the drone. n τ It is τ Modeled as Gaussian noise of random walk, ω m and a m These are the original IMU measurements from the drone. n g and n a yes ω m and a m The noise measured by the drone in the middle, b a and b g This is the IMU bias of the drone, which is modeled as being composed of Gaussian noise. n bg and n ba Driven random walk; Next, propagate the drone IMU data forward, the drone IMU propagated state prior and covariance prior at time t k system state x impose a prior distribution as follows: (4) The prior distribution is expressed as p ( x ), and the LiDAR and camera measurement models of the UAV are expressed as: (5) wherein, v l N (0, Σv l ) and v c ~ N (0, N Σv c represent the measurement noise of the drone LiDAR and camera, respectively;​ To perform LiDAR or image measurements on drones The fusion of the prior distributions q ( x ), denoted as ,in In the case of LiDAR updates for drones, ( , ) is the state and covariance obtained from the forward propagation of the UAV IMU; in the case of visual updates, ( , The convergence state and covariance are obtained from the UAV LiDAR update; in order to obtain the measurement model distribution q ( y| x ), will the first κ The estimated state at the next iteration is represented as follows: ,in ; through its in The first-order Taylor expansion performed at the point approximates the measurement model formula (5), yielding: (6) (7) In the formula, , z κ It is a residual. It is lumped measurement noise. H κ and L κ yes Compared to δx κ and v The Jacobian matrix is ​​evaluated at zero; then, the prior distribution is... q ( x ) and measurement distribution q ( y | x Substitute into the posterior distribution In addition, maximum likelihood estimation is performed to obtain the results from the standard update step in the sequentially updated error state iterative Kalman filter framework. δx κ as well as x κ Maximum a posteriori estimate: (8) (9) The final convergent state and covariance matrix make the mean and covariance of the posterior distribution equal to... q ( x| y ).

3. The method according to claim 2, characterized in that, Step two includes performing a drone-based sliding window edge detection process, specifically: The drone optimization model is represented in the following general form: HX=b (12) The above general form can be broken down into the following forms: (13) The purpose of disassembly is to remove the old frames from the drone. X m Remove the state variables while retaining the constraints of the UAV; the decomposed optimization problem is then triangulated using the Schur complement matrix, i.e.: (14) At this point, it is possible to operate without relying on the drone's X. m X can then be solved r : (15)。 4. The method according to claim 3, characterized in that, In the sliding window optimization method for multimodal data fusion, the residual of the sliding window based on UAV map localization consists of four parts: a. Residuals of drone map matching pose and optimization variables; b. Residuals of relative pose and optimization variables in UAV laser odometry; c. Residuals of UAV IMU pre-integration and optimization variables; d. Residuals corresponding to prior factors resulting from the marginalization of drones.

5. The method according to claim 4, characterized in that, In step three, non-loop point cloud SLAM is used to establish an incremental point cloud registration framework based on time series. Its core is to build a global point cloud map by registering frame by frame. Specifically, it includes: (1) Point cloud preprocessing: The preprocessing stage includes voxel downsampling and normal estimation; (2) Frame-by-frame registration: Using the first frame point cloud as the reference coordinate system, each subsequent frame point cloud is registered with the previous frame in sequence. The specific steps are coarse registration, fine registration, and registration quality evaluation and correction. (3) Global point cloud construction: The transformation matrix obtained by registration of each frame of point cloud is transformed to the reference coordinate system, and finally all transformed point clouds are merged to form a globally consistent point cloud map.

6. The method according to claim 5, characterized in that, In step three, loop-closure point cloud SLAM is used to establish a globally consistent framework based on pose graph optimization. By constructing a pose graph that includes temporal constraints and loop closure constraints, the poses of all frames are globally optimized to eliminate accumulated errors. Specifically, this includes: (1) Point cloud preprocessing: The preprocessing stage includes voxel downsampling and normal estimation to ensure the processing efficiency and registration reliability of point cloud data; (2) Pose graph construction: The pose graph consists of nodes and edges. Nodes represent the pose of each frame of point cloud in the global coordinate system and the relative pose constraints between point clouds, respectively. (3) Global optimization: The Levenberg-Marquardt algorithm is used to perform global optimization on the pose graph to minimize the error function of all edges; (4) Global point cloud construction: Based on the optimized pose matrix, the point clouds of each frame are transformed into the global coordinate system, and all point clouds are merged to form a globally consistent high-precision map.

7. The method according to claim 1, characterized in that, In step three, the bridge model reconstructed from the 3D data of the large bridge UAV inspection is processed to enhance its contour information. Specifically, the edge information of the image is enhanced and the contour features of the image are highlighted through image sharpening, image smoothing and edge detection contour enhancement methods.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-7.

Images