A method for inspecting steel box girders based on SLAM technology

Abstract

This invention belongs to the field of bridge engineering monitoring technology, specifically a steel box girder inspection method based on SLAM technology. It includes a mobile robot platform, a multi-sensor array, and a data processing terminal. The multi-sensor array includes a lidar, a depth camera, and an IMU, used to synchronously collect raw data of the internal environment of the steel box girder. The data processing terminal receives the raw data and performs real-time pose calculation and map construction based on a tightly coupled lidar, inertial, and visual odometry framework. It also incorporates structural constraint factors based on the inherent structural characteristics of the steel box girder for backend optimization, generating a high-precision 3D semantic map. The mobile robot platform performs autonomous path planning and navigation based on the 3D semantic map and integrates a deep learning model to identify and locate surface defects. This invention enables high-precision positioning, mapping, and intelligent inspection inside a closed steel box girder, significantly improving inspection efficiency and digitalization level.

Description

Technical Field

[0001] This invention belongs to the field of bridge engineering monitoring, specifically a method for inspecting steel box girders based on SLAM technology. Background Technology

[0002] Steel box girders are the core load-bearing structures of modern long-span bridges. They are subjected to vehicle loads and environmental erosion over a long period of time, and are prone to fatigue cracks, corrosion and other defects. Therefore, regular inspections are crucial. However, the internal environment of steel box girders is characterized by enclosed space, insufficient lighting, highly repetitive structural components such as U-shaped ribs and diaphragms, and large reflection interference from metal surfaces. In addition, it is an environment where global satellite navigation signals are denied.

[0003] Currently, the internal inspection of steel box girders mainly relies on manual entry with equipment, which suffers from low efficiency, poor safety, inconsistent inspection quality, and difficulty in quantification and digital archiving. Traditional inspection robots for steel box girder internal inspection still have significant errors in autonomous navigation and precise positioning within the steel box girder. Traditional SLAM algorithms based on single sensors, such as LiDAR or vision-only, are prone to positioning drift and mismatch in the repetitive structure and weak texture environment of steel box girders, resulting in low accuracy and poor reliability of the constructed map, which cannot provide a stable spatial reference for subsequent refined defect detection.

[0004] Therefore, the present invention provides a steel box girder inspection method based on SLAM technology to solve the problems mentioned in the background art. Summary of the Invention

[0005] The technical solution adopted by the present invention to solve its technical problem is: a steel box girder inspection method based on SLAM technology, comprising a mobile robot platform and a data processing terminal connected to it in communication, wherein the mobile robot platform is equipped with a multi-sensor array;

[0006] The SLAM-based steel box girder inspection method includes the following steps:

[0007] S1: Synchronous acquisition of multi-sensor data. Raw data of the internal environment of the steel box girder are synchronously acquired through a multi-sensor array mounted on a mobile robot platform. The multi-sensor array includes a lidar, a depth camera, and an IMU.

[0008] S2: Tightly coupled front-end odometer calculation, through the data processing terminal based on the tightly coupled laser, inertial and visual odometer framework, fuses and processes the raw data collected by the multi-sensor array in step S1, and calculates the first pose estimation and initial local map of the mobile robot platform inside the steel box girder in real time.

[0009] S3: Backend optimization and semantic map construction. The first pose estimate and the initial local map in step S2 are input into the backend optimizer. Combined with the structural constraint factors constructed from the inherent structural features of the steel box girder extracted from the original data in step S2, joint nonlinear optimization is performed to generate a high-precision, globally consistent three-dimensional semantic map of the steel box girder interior.

[0010] S4: Autonomous inspection and defect identification. Based on the three-dimensional semantic map generated in step S3, the mobile robot platform is used for path planning and navigation control, and defect information on the surface of the steel box girder structure is identified and labeled.

[0011] Preferably, the lidar in step S1 is used to acquire high-density three-dimensional point cloud data inside the steel box girder, the depth camera in step S1 is used to acquire color images and depth information inside the steel box girder, and the IMU in step S1 is used to acquire acceleration and angular velocity data of the mobile robot.

[0012] Preferably, the data processing terminal in step S2 processes the raw data in the following specific way:

[0013] S21: Preprocess the data acquired by the IMU in step S1 to obtain the pose prior estimate of the mobile robot platform;

[0014] S22: Perform motion distortion compensation on the 3D point cloud data acquired by the lidar in step S1, and perform point cloud registration with the historical point cloud, and calculate the residual from the lidar point to the local map plane in step S1.

[0015] S23: Perform feature extraction and optical flow tracking on the color image and depth information acquired by the depth camera in step S1, and calculate the reprojection error of visual feature points;

[0016] S24: Input the pose prior estimate from step S21, the LiDAR point-to-plane residual, and the visual reprojection error from step S23 into the iterative error state Kalman filter for tight coupling optimization, and output the first pose estimate and the initial local map from step S2.

[0017] Preferably, the inherent structural features of the steel box girder in step S3 include the equal spacing and parallelism of the U-shaped stiffening ribs, the perpendicularity of the diaphragms and the top plate of the bridge deck, and the local planar features of the top plate of the bridge deck.

[0018] Preferably, the structural constraint factors in step S3 include:

[0019] U-rib spacing constraint factor: used to constrain the distance between adjacent U-rib feature points to be consistent with the prior design spacing;

[0020] Diaphragm vertical constraint factor: used to constrain the plane normal vector of the diaphragm extracted from the point cloud to be perpendicular to the normal vector of the bridge deck top plate;

[0021] Bridge deck planar constraint factor: used to constrain the positional relationship between points on the bridge deck and the plane;

[0022] Symmetry and repeatability constraints: used to verify the reliability of enhanced loop closure detection.

[0023] Preferably, the joint nonlinear optimization in step S3 is achieved through factor graph optimization.

[0024] Preferably, the implementation method for identifying and labeling surface defect information of steel box girder structure in step S4 is as follows: based on the texture image data collected by the depth camera in step S1, the trained deep learning model is used to automatically identify cracks, corrosion, and coating peeling defects, and the identification results and their spatial coordinates are simultaneously labeled on the three-dimensional semantic map in step S3.

[0025] Preferably, the mobile robot platform adopts a magnetic adsorption mobile chassis, which is used for the adsorption and movement of the mobile robot platform on the vertical wall or top surface of the steel box girder.

[0026] The beneficial effects of this invention are as follows:

[0027] The present invention discloses a steel box girder inspection method based on SLAM technology. Through a tightly coupled front-end design of laser, vision, and IMU, the SLAM system's adaptability and robustness in dim, highly reflective, and weakly textured environments of steel box girders are significantly enhanced. Furthermore, the back-end optimization by introducing structural constraints of the steel box girder cleverly utilizes prior knowledge to fundamentally alleviate the SLAM degradation problem caused by repetitive structures, achieving positioning and mapping accuracy far exceeding that of traditional methods. The high-precision SLAM map is seamlessly integrated with AI-based visual defect detection, realizing full-process automation and intelligence from environmental perception to defect identification, greatly improving the efficiency, safety, and digitalization level of steel box girder inspection. Attached Figure Description

[0028] The invention will now be further described with reference to the accompanying drawings.

[0029] Figure 1 This is an embodiment diagram of the method of the present invention. Detailed Implementation

[0030] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0031] Example 1: As Figure 1As shown, an embodiment of the present invention provides a steel box girder inspection method based on SLAM technology, which includes a mobile robot platform and a data processing terminal connected to it in communication. The mobile robot platform is equipped with a multi-sensor array.

[0032] The steel box girder inspection method based on SLAM technology includes the following steps:

[0033] S1: Multi-sensor data synchronous acquisition. Through a multi-sensor array mounted on a mobile robot platform, raw data of the internal environment of the steel box girder is collected synchronously. The multi-sensor array includes a lidar, a depth camera, and an IMU (inertial measurement unit). The lidar scans at a fixed frequency to acquire 3D point clouds of the surrounding environment. The global shutter camera acquires high-definition texture images with the assistance of a fill light. The IMU outputs acceleration and angular velocity at a high frequency. All sensor data are synchronized through hardware triggering or software timestamps.

[0034] S2: Tightly coupled front-end odometer calculation. Based on the tightly coupled laser, inertial and visual odometer framework, the data processing terminal fuses and processes the raw data collected by the multi-sensor array in step S1, and calculates the first pose estimation and initial local map of the mobile robot platform inside the steel box girder in real time.

[0035] S3: Backend optimization and semantic map construction. The first pose estimate and the initial local map in step S2 are input into the backend optimizer. Combined with the structural constraint factors constructed from the inherent structural features of the steel box girder extracted from the original data in step S2, joint nonlinear optimization is performed to generate a high-precision, globally consistent three-dimensional semantic map of the steel box girder interior.

[0036] S4: Autonomous Inspection and Defect Identification. Based on the 3D semantic map generated in step S3, the mobile robot platform is used for path planning and navigation control, and defect information on the surface of the steel box girder structure is identified and labeled.

[0037] Specifically, in step S2, a tightly coupled laser, inertial, and visual odometry system is used. By fusing IMU pre-integration priors, LiDAR point-to-surface residuals, and visual reprojection errors based on optical flow tracking using an iterative error state Kalman filter, the system effectively improves the accuracy and robustness of pose estimation in dim and reflective environments. In step S3, the inherent prior geometric knowledge of the steel box girder, such as the equal spacing of U-ribs and the verticality of the diaphragms, is innovatively utilized. This knowledge is modeled as structural constraint factors and added to the back-end factor graph optimization. This design can effectively correct the cumulative errors and error loops caused by highly repetitive environments, significantly improving the structural consistency and accuracy of the global map. In step S4, defect identification is based on a deep learning model and is bound to the precise spatial location generated by the SLAM system, realizing the visual map location of defects and providing an intuitive and reliable digital archive for maintenance decisions.

[0038] In step S1, the lidar is used to acquire high-density three-dimensional point cloud data inside the steel box girder, the depth camera is used to acquire color images and depth information inside the steel box girder, and the IMU is used to acquire acceleration and angular velocity data of the mobile robot.

[0039] In step S2, the data processing terminal processes the raw data in the following ways:

[0040] S21: Preprocess the data acquired by the IMU in step S1 to obtain the pose prior estimate of the mobile robot platform;

[0041] S22: Perform motion distortion compensation on the 3D point cloud data acquired by the LiDAR in step S1, and perform point cloud registration with the historical point cloud. Calculate the residual from the LiDAR point to the local map plane in step S1.

[0042] S23: Perform feature extraction and optical flow tracking on the color image and depth information acquired by the depth camera in step S1, and calculate the reprojection error of visual feature points;

[0043] S24: Input the pose prior estimate from step S21, the LiDAR point-to-plane residual, and the visual reprojection error from step S23 into the iterative error state Kalman filter for tight coupling optimization, and output the first pose estimate and the initial local map from step S2.

[0044] The inherent structural features of the steel box girder in step S3 include the equal spacing and parallelism of the U-shaped stiffening ribs, the perpendicularity of the transverse diaphragms and the top plate of the bridge deck, and the local planar features of the top plate of the bridge deck.

[0045] The structural constraint factors in step S3 include:

[0046] U-rib spacing constraint factor: used to constrain the distance between adjacent U-rib feature points to be consistent with the prior design spacing;

[0047] Diaphragm vertical constraint factor: used to constrain the plane normal vector of the diaphragm extracted from the point cloud to be perpendicular to the normal vector of the bridge deck top plate;

[0048] Bridge deck planar constraint factor: used to constrain the positional relationship between points on the bridge deck and the plane;

[0049] Symmetry and repeatability constraints: used to verify the reliability of enhanced loop closure detection.

[0050] In step S3, the joint nonlinear optimization is achieved through factor graph optimization, and the objective function is:

[0051]

[0052] in, The set of variables to be optimized. For odometer constrained residuals, For loop closure detection constraint residuals, The residuals are the U-rib spacing constraint factors. This represents the residual of the vertical constraint factor for the diaphragm.

[0053] The specific implementation method for identifying and labeling surface defect information of steel box girder structure in step S4 is as follows: Based on the texture image data collected by the depth camera in step S1, the trained deep learning model is used to automatically identify cracks, corrosion, and coating peeling defects, and the identification results and their spatial coordinates are synchronously labeled on the three-dimensional semantic map in step S3.

[0054] The mobile robot platform uses a magnetic adsorption mobile chassis, which is used for the adsorption and movement of the mobile robot platform on the vertical wall or top surface of the steel box girder.

[0055] Example 2: Compared with Example 1, another implementation of the present invention is as follows: The specific implementation of the tight-coupled front-end real-time odometer calculation in step S2 is as follows:

[0056] 1. Pre-integrate the IMU data to obtain the relative motion prediction (pose prior) between adjacent lidar / camera frames.

[0057] ① Perform IMU pre-integration to calculate relative motion prediction: at two consecutive frame times and Between, for IMU angular velocity and acceleration Integrating, we obtain the relative rotation. speed change and positional changes The pre-integrals, these pre-integrals constitute the time... and The pose prior constraints, whose residual form is as follows: ,in This converts a rotation matrix into a rotation vector. (Relative rotation) ,in It is a moment angular velocity; velocity change ,in It is a moment acceleration, It is the rotation matrix that transforms acceleration from the body coordinate system to the world coordinate system, and the position change. ,in It is a moment The speed.

[0058] ②. Perform visual feature tracking: track adjacent image frames and Extract and match ORB feature points By utilizing the matched feature point pairs, the epipolar geometry is calculated to satisfy... Essential matrix ,in V is an orthogonal matrix, and for the essential matrix Singular value decomposition yields four possible values. By decomposing and verifying through triangulation, a true inter-frame relative rotation is finally determined. unit vector in translation direction The true relative translation between two adjacent frames is obtained by combining lidar data. The magnitude of this translation is the true motion scale, thus yielding the restored scale factor. This yields a visual relative translation with true scale. Thus, the visual relative pose is obtained. This relative pose provides a good initial pose value for subsequent optimization. Furthermore, the 3D points obtained through feature matching and triangulation... This will be used to calculate the visual reprojection error. As a key visual observation constraint in tightly coupled optimization.

[0059] ③. Integrate IMU pre-integration constraints with visual reprojection error ,in It is the camera projection function. It is the camera pose in the current frame. These are the three-dimensional coordinates of the feature points. The pixel coordinates observed on the image are used as common inputs to the iterative error state Kalman filter for tightly coupled optimization. The Kalman filter does not directly optimize; instead, it optimizes small error states around the current estimate, estimating based on the current state. Calculate the residuals of each sensor and its error state Jacobian matrix Construct and solve linear systems The optimal state correction is obtained, and high-frequency, accurate real-time robot pose and odometry information is output.

[0060] First, define the robot's state vector. Where: p is position, v is velocity, and q is the rotation quaternion. It is a gyroscope with zero bias. It is an accelerometer with zero bias.

[0061] Error state vector ,in is the minimum angular vector of the rotation error.

[0062] Subsequently, IESKF iteratively linearizes and optimally fuses these three heterogeneous observation information sets, outputting a smooth and accurate real-time robot pose (i.e., first pose estimation) and a local map at the current moment, with a frequency higher than that of laser frames. Specifically, the fusion method involves establishing a joint optimization objective function:

[0063]

[0064] in , , Let Jacobian matrix be the error state for each observation pair. For IMU pre-integral covariance; , The covariance of laser and visual observation noise.

[0065] During iterative solution, the equation is constructed as follows:

[0066]

[0067] in

[0068] Solve and update:

[0069]

[0070]

[0071] in For state update operators;

[0072] Repeat steps 2-4 until convergence, and output the converged state. As the optimal estimate for the current moment, the smooth, high-frequency (higher than the laser frame rate) pose estimate output by IESKF is:

[0073] The output local map is an optimized set of point clouds:

[0074] 2. LiDAR point cloud motion distortion compensation: Based on the LiDAR scanning model, a precise timestamp is assigned to each point in the current frame's point cloud. The robot's motion trajectory is obtained using the continuous IMU motion integrated in S21. Acquire each collection moment Corresponding robot pose Select the end time of the current frame. For reference time, its corresponding pose is ,pass Calculate the compensated coordinates of each point to eliminate point cloud deformation caused by robot motion during scanning.

[0075] ①. IMU parameterization and measurement calibration: employing a method including zero bias Scale factor Non-orthogonal error matrix A complete IMU model for the raw measured angular velocity. acceleration Perform correction:

[0076]

[0077] ②. Point cloud motion distortion estimation: For a frame with a scan time of The point cloud, based on the LiDAR rotation angle of each point. Calculate its precise acquisition time Estimating each using IMU integration. Robot pose at any given moment:

[0078]

[0079] ③ Motion distortion correction: Transform each point from the coordinate system at its acquisition time to the reference time of that frame. Coordinate system:

[0080]

[0081] in By correcting the acceleration The corrected point cloud is obtained by double integration. This eliminates the deformation caused by robot movement during scanning, laying the foundation for subsequent accurate matching. In the formula... To correct the coordinates of the points, The original point coordinates, These are the reference time and the data acquisition time of the point, respectively. From arrive The relative rotation matrix, From arrive The relative rotation matrix, For reference time The robot's location, For the time of data collection The robot's location, This represents the relative translation of the robot.

[0082] ④. Pre-integral relative motion prediction: Based on the corrected IMU data, calculate adjacent keyframes. arrive Pre-integral quantity This provides high-frequency relative motion constraints and relative rotation for multi-sensor fusion. ,in It is a moment angular velocity, velocity change ,in It is a moment acceleration, It is the rotation matrix that transforms acceleration from the body coordinate system to the world coordinate system, and the position change. ,in It is a moment The speed.

[0083] 3. Match the current frame point cloud with a local map constructed from historical frames. Perform iterative nearest point (ICP) matching between the distortion-compensated point cloud and a sliding window local map composed of historical keyframes to solve for the precise pose of the current frame relative to the local map. The laser residual is calculated by minimizing the "point-to-surface" distance.

[0084] ① Motion distortion removal: Utilizing the robot's motion priors provided by IMU pre-integration, the distortion of the point cloud caused by the LiDAR's own motion within one frame scan time is compensated to obtain the distortion-free point cloud of the current frame. .

[0085] The main steps and formulas are as follows:

[0086] ⑪ Correction of raw IMU measurements

[0087] Using a complete IMU model that includes zero bias, scaling factor, and non-orthogonal error, the original angular velocity was analyzed. and acceleration Perform correction:

[0088]

[0089]

[0090] in, Zero bias, As a scale factor, It is a non-orthogonal error matrix. The rotation matrix from the world frame to the robot's body frame. This is the acceleration due to gravity.

[0091] ⑫ Precise laser point acquisition time estimation

[0092] For the scanning time period A point cloud within a frame, based on the rotation angle of each laser point. Calculate its precise acquisition time :

[0093]

[0094] in This refers to the angular velocity of the lidar scan.

[0095] ⑬ Robot pose estimation during scanning

[0096] Using the corrected IMU angular velocity data, estimation is performed through integration. Robot rotation matrix at time :

[0097]

[0098] By correcting the acceleration By performing double integration, we obtain Location at any moment .

[0099] ⑭ Point cloud motion distortion correction

[0100] Each original laser point From its collection time The coordinate system is transformed to the reference time set for this frame. Coordinate system:

[0101]

[0102] in, Indicates from arrive The relative rotation matrix.

[0103] ⑮ Pre-integral relative motion prediction

[0104] Based on the corrected IMU data, calculate adjacent keyframes (such as... arrive Pre-integral quantities between ) including relative rotation speed change and positional changes This provides high-frequency and accurate relative motion constraints for subsequent tightly coupled multi-sensor fusion.

[0105] ②. Local map construction and maintenance: Maintain a local map centered on the robot's current position with a radius of... (e.g., 10 meters) sliding window partial map The map is composed of point clouds from historical keyframes: when the robot moves a distance or rotates an angle exceeding a set threshold, the point cloud from the previous keyframe is updated. Based on its optimized pose The coordinates are transformed to the world coordinate system and integrated with the existing local map, which uses a 3D voxel grid or KD-Tree data structure for spatial management to support fast nearest neighbor search.

[0106] ③. Point-to-surface iterative nearest point matching: using pose priors from IMU pre-integration As the initial value for iteration, the distortion-free point cloud of the current frame will be used. With local map Perform matching, for Each point in ,exist Find it (For example,) 5 nearest neighbors are used to fit a local plane through eigenvalue decomposition, thus obtaining the unit normal vector of that plane. and center point .

[0107] In the iterative nearest point (ICP) matching process, it is necessary to fit a local plane through eigenvalue decomposition to obtain the unit normal vector and center point of the plane. This method is based on principal component analysis (PCA), and the specific steps are as follows:

[0108] For a point in the point cloud of the current frame In its corresponding local map Search (Usually, 5 nearest neighbors are selected to form a local point set) .

[0109] ㉜ Calculate the centroid (geometric center) of this point set:

[0110]

[0111] Construct the covariance matrix of the local point set:

[0112]

[0113] ㉞ Pair of covariance matrices Perform eigenvalue decomposition to obtain eigenvalues. and the corresponding unit eigenvector .

[0114] Find the minimum eigenvalue Corresponding feature vector The unit normal vector of the fitting plane This direction, which is the direction in which the point set is most discrete, is perpendicular to the local plane.

[0115] Therefore, each matching point can obtain a result from the center point. and normal vector The defined local plane is used to construct the "point-to-surface" distance residual. Furthermore, by optimizing the residual to solve for the accurate pose, this method can effectively utilize local geometric structures to improve the robustness and accuracy of point cloud matching.

[0116] Constructing "point-to-multipoint" distance residuals:

[0117]

[0118] in, It is the transpose of the local plane normal vector. Given the transformation matrix from the current frame to the local map (world coordinate system), the following objective function is iteratively optimized using the Gauss-Newton method:

[0119]

[0120] in A robust kernel function (such as the Huber kernel) is used to suppress the influence of mismatched points. After iterative convergence, the accurate pose of the current frame is obtained. It also calculates the relative motion between the current frame and the previous keyframe. , as a constraint for lidar odometer.

[0121] ④. Local map update: based on the current frame pose Determine whether to generate a new keyframe.

[0122] Keyframe determination criteria:

[0123] The system is based on the pose of the current frame. pose from the previous keyframe Whether the relative motion changes between frames exceed a preset threshold determines whether the current frame should be inserted as a new keyframe. The decision criteria consider translation, rotation, and time intervals simultaneously.

[0124] ㊶ Translation distance:

[0125]

[0126] like (For example If ), then the translation condition is satisfied.

[0127] ㊷ Rotation angle:

[0128] By relative rotation matrix Calculate the rotation angle:

[0129]

[0130] like (For example If ), then the rotation condition is satisfied.

[0131] ㊸ Time interval.

[0132] ㊹If the time since the last keyframe (For example If ), then the time condition is met.

[0133] If any of the above conditions are met, the current frame will be set as the new keyframe.

[0134] If the keyframe conditions are met, then Add after transformation Remove the oldest keyframe point cloud from the sliding window and update the spatial index.

[0135] The specific steps for local map update under keyframe conditions are as follows:

[0136] i. Keyframe point cloud transformation to world coordinate system

[0137] Let the optimized pose of the current keyframe be... The point cloud after motion distortion correction is Transform each point to the world coordinate system:

[0138]

[0139] Obtain the point cloud set in the world coordinate system:

[0140]

[0141] ii. Point cloud fusion and local map update

[0142] Transformed point cloud Compared with the current local map To merge:

[0143]

[0144] To control map density and scale, voxel grid filtering (voxel resolution) is typically performed simultaneously. For example, 0.05m), that is, retaining a representative point (such as the center of gravity) within each voxel:

[0145]

[0146] iii. Sliding window maintenance

[0147] The system maintains a list containing the most recent (e.g., a sliding window queue of 10 keyframes) If the window length exceeds [a certain value] after inserting a new keyframe, Remove the oldest keyframe. and its corresponding point cloud :

[0148]

[0149] This operation keeps the local map containing only the most recent keyframe observations to adapt to environmental changes and control computational load.

[0150] iv. Spatial Index Update

[0151] To support efficient nearest neighbor search in subsequent point-to-multipoint ICP matching, the spatial index structure of the local map needs to be updated. A common method is to rebuild the KD-Tree.

[0152]

[0153] Alternatively, an incremental update strategy can be adopted, updating only the new point cloud. Insert an existing KD-Tree to improve update efficiency.

[0154] 4. Extract ORB or Shi-Tomasi feature points from the camera image.

[0155] ORB feature point extraction consists of two parts: FAST corner detection and BRIEF descriptor calculation. ORB feature matching is performed using Hamming distance. in Refers to two feature descriptors Hamming distance between them Refers to two binary feature descriptors. The first descriptor A binary value (0 or 1); The XOR operator operates on the following rules: 0 for identical values ​​and 1 for different values.

[0156] Shi-Tomasi detects corner features by calculating the "corner response function" of each pixel in the image. Its corner response function is: Corner response value These refer to the two eigenvalues ​​of the structure tensor matrix of the image at that point.

[0157] 5. Using the LK optical flow method for inter-frame tracking, the reprojection error of feature points is calculated, and the optical flow equation is obtained based on the assumption of constant brightness: ,in These refer to the images in gradient of direction, The gradient of an image over time. Each refers to a feature point The reprojection error for the optical flow in the direction of the projection is: ,in Visual reprojection error refers to the difference between the observed location of a feature point and its predicted location based on the projection of the map point. In the first The pixel coordinates of the feature points actually observed in the frame image are obtained from ORB and Shi-Tomasi feature extraction and matching. This refers to the 3D map points corresponding to feature points and the camera pose in the current frame. Predicted pixel coordinates (two-dimensional vectors) projected onto the image plane.

[0158] 6. The IMU pre-integration prior, laser residual, and visual reprojection error are input together into an iterative error state Kalman filter (IESKF). First, let the robot state vector be set. Where: p is position, v is velocity, and q is the rotation quaternion. It is a gyroscope with zero bias. It is an accelerometer with zero bias.

[0159] Error state vector ,in is the minimum angular vector of the rotation error.

[0160] Subsequently, IESKF iteratively linearizes and optimally fuses these three heterogeneous observation information, outputting a smooth and accurate real-time robot pose (i.e., the first pose estimate) and a local map at the current moment with a frequency higher than that of laser frames. Specifically, the fusion method involves establishing a joint optimization objective function:

[0161]

[0162] in , , Let be the Jacobian matrix for the error state of each observation pair; For IMU pre-integral covariance; , The covariance of laser and visual observation noise.

[0163] During iterative solution, the equation is constructed as follows:

[0164]

[0165] in

[0166] Solve and update:

[0167]

[0168]

[0169] in For state update operators;

[0170] Repeat steps 2-4 until convergence, and output the converged state. As the optimal estimate for the current moment, the smooth, high-frequency (higher than the laser frame rate) pose estimate output by IESKF is: .

[0171] The output local map is an optimized set of point clouds: .

[0172] In step S3, the backend global optimization and semantic map construction are integrated with structural constraints. The pose and map output from the frontend have accumulated errors. This step optimizes these errors through loop closure detection and structural constraints. The robot's pose is used as a node, and the frontend odometry constraints, loop closure detection constraints, and the structural constraints proposed in this invention are used as edges to construct a factor graph. The steps are as follows:

[0173] a) Calculate the structural constraint residuals.

[0174] (1) U-rib spacing constraint

[0175] Let the design spacing between adjacent U-ribs be . The unit vector of the arrangement direction is The first point extracted from the point cloud The centerline position of the U-rib feature point set is Then the constraint residual is:

[0176]

[0177] This constraint forces the spacing between adjacent U-ribs to be consistent with the design value, thus resolving mismatches caused by repetitive structures.

[0178] (2) Vertical constraint of diaphragm

[0179] Let the normal vector of the top slab of the bridge deck be... The first point extracted from the point cloud The normal vector of the plane of each diaphragm is Since the diaphragms should be perpendicular to the bridge deck, the constraint residuals are:

[0180]

[0181] Theoretically, the dot product is 0 when the two are perpendicular.

[0182] (3) Bridge deck planar constraints

[0183] Let the equation of the bridge deck be... ,in For any point on the bridge surface, the constraint residual is:

[0184]

[0185] This constraint forces the bridge surface point cloud to conform to the planar model.

[0186] b) Establish a factor graph optimization model.

[0187] Let the robot pose sequence be ;

[0188]

[0189] in For location, It is a rotation matrix.

[0190] The objective function to be optimized is:

[0191]

[0192] in: The set of variables to be optimized includes pose and map points. The odometry constraint residuals are the relative transformations between poses from the front-end IESKF output. The constraint residual for loop closure detection is established when the robot identifies repeated positions. For structural constraint residuals.

[0193] c) Semantic map construction.

[0194] The optimized pose and point cloud are used to construct a 3D semantic map;

[0195]

[0196] in For the first Point cloud of a frame, This is the optimized pose.

[0197] Semantic segmentation is performed on the point cloud to assign semantic labels to structures such as U-ribs, diaphragms, and bridge decks:

[0198]

[0199] When the system detects that the robot has returned to a historical position (loopback), or periodically, it initiates graph optimization (using the GTSAM or g2o library). The optimization process adjusts all pose nodes simultaneously to minimize the overall residual of all constraints, thereby obtaining a globally consistent optimal pose trajectory. Finally, all optimized laser point clouds and camera images (which can be colored) are stitched together to generate a high-precision 3D semantic map with color and semantic labels (such as U-ribs and diaphragms).

[0200] In step S4, autonomous inspection and intelligent defect identification are implemented. Based on the precise 3D semantic map generated in step S3, an inspection path covering the entire bridge can be pre-planned or planned in real-time. The mobile robot platform navigates autonomously based on real-time positioning information (provided by the front-end odometry and optimized by the back-end) and the planned path. During the inspection, a lightweight deep learning model (such as a YOLO or U-Net variant) runs simultaneously to analyze the real-time acquired images online, automatically identifying defects such as cracks and rust spots. Once a defect is identified, the system immediately obtains the robot's precise pose at that moment and permanently labels the defect type, size (estimated through depth information), and precise 3D spatial coordinates as attributes on the 3D semantic map, forming a complete digital inspection report.

[0201] Specifically, in traditional technologies, the U-ribs, diaphragms, and other components inside steel box girders exhibit highly similar and periodic arrangement characteristics. This leads to erroneous associations during feature matching in traditional SLAM algorithms based on a single sensor, resulting in positioning drift and map misalignment. Furthermore, insufficient lighting and strong surface reflection within the steel box girder make visual feature extraction difficult and matching success rates low. Single sensors relying solely on vision or LiDAR struggle to provide stable pose estimation. Additionally, during long-distance inspections, small positioning errors accumulate rapidly in repetitive structural environments, causing traditional loop closure detection to frequently fail due to a lack of effective discriminative features and an inability to promptly correct trajectory deviations. Existing SLAM systems also suffer from localized distortions in maps constructed within steel box girder environments. Problems such as inconsistencies in curvature and scale make it impossible to provide a stable reference framework for the accurate location and quantitative analysis of defects such as cracks and corrosion. However, this invention significantly enhances the adaptability and robustness of the SLAM system in the dim, highly reflective, and weakly textured environment of steel box girders through a tightly coupled front-end design of laser, vision, and IMU. By introducing back-end optimization based on the structural constraints of the steel box girder, it cleverly utilizes prior knowledge to fundamentally alleviate the SLAM degradation problem caused by repetitive structures, achieving a positioning and mapping accuracy far exceeding that of traditional methods. It seamlessly integrates high-precision SLAM maps with AI-based visual defect detection, realizing full-process automation and intelligence from "environmental perception" to "defect identification," greatly improving the efficiency, safety, and digitalization level of steel box girder inspection.

[0202] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for inspecting steel box girders based on SLAM technology, comprising a mobile robot platform and a data processing terminal connected to it in communication, wherein the mobile robot platform is equipped with a multi-sensor array, characterized in that: The SLAM-based steel box girder inspection method includes the following steps: S1: Synchronous acquisition of multi-sensor data. Raw data of the internal environment of the steel box girder are synchronously acquired through a multi-sensor array mounted on a mobile robot platform. The multi-sensor array includes a lidar, a depth camera, and an IMU. S2: Tightly coupled front-end odometer calculation, through the data processing terminal based on the tightly coupled laser, inertial and visual odometer framework, fuses and processes the raw data collected by the multi-sensor array in step S1, and calculates the first pose estimation and initial local map of the mobile robot platform inside the steel box girder in real time. S3: Backend optimization and semantic map construction. The first pose estimate and the initial local map in step S2 are input into the backend optimizer. Combined with the structural constraint factors constructed from the inherent structural features of the steel box girder extracted from the original data in step S2, joint nonlinear optimization is performed to generate a high-precision, globally consistent three-dimensional semantic map of the steel box girder interior. S4: Autonomous inspection and defect identification. Based on the three-dimensional semantic map generated in step S3, the mobile robot platform is used for path planning and navigation control, and defect information on the surface of the steel box girder structure is identified and labeled.

2. The method for inspecting steel box girders based on SLAM technology according to claim 1, characterized in that: The lidar mentioned in step S1 is used to acquire high-density three-dimensional point cloud data inside the steel box girder. The depth camera mentioned in step S1 is used to acquire color images and depth information inside the steel box girder. The IMU mentioned in step S1 is used to acquire acceleration and angular velocity data of the mobile robot.

3. The method for inspecting steel box girders based on SLAM technology according to claim 2, characterized in that: The specific processing method of the data processing terminal for the raw data in step S2 is as follows: S21: Preprocess the data acquired by the IMU in step S1 to obtain the pose prior estimate of the mobile robot platform; S22: Perform motion distortion compensation on the 3D point cloud data acquired by the lidar in step S1, and perform point cloud registration with the historical point cloud, and calculate the residual from the lidar point to the local map plane in step S1. S23: Perform feature extraction and optical flow tracking on the color image and depth information acquired by the depth camera in step S1, and calculate the reprojection error of visual feature points; S24: Input the pose prior estimate from step S21, the LiDAR point-to-plane residual, and the visual reprojection error from step S23 into the iterative error state Kalman filter for tight coupling optimization, and output the first pose estimate and the initial local map from step S2.

4. The method for inspecting steel box girders based on SLAM technology according to claim 1, characterized in that: The inherent structural features of the steel box girder in step S3 include the equal spacing and parallelism of the U-shaped stiffening ribs, the perpendicularity of the transverse diaphragms and the top plate of the bridge deck, and the local planar features of the top plate of the bridge deck.

5. The method for inspecting steel box girders based on SLAM technology according to claim 1, characterized in that: The structural constraint factors in step S3 include: U-rib spacing constraint factor: used to constrain the distance between adjacent U-rib feature points to be consistent with the prior design spacing; Diaphragm vertical constraint factor: used to constrain the plane normal vector of the diaphragm extracted from the point cloud to be perpendicular to the normal vector of the bridge deck top plate; Bridge deck planar constraint factor: used to constrain the positional relationship between points on the bridge deck and the plane; Symmetry and repeatability constraints: used to verify the reliability of enhanced loop closure detection.

6. The method for inspecting steel box girders based on SLAM technology according to claim 1, characterized in that: In step S3, joint nonlinear optimization is achieved through factor graph optimization.

7. The method for inspecting steel box girders based on SLAM technology according to claim 1, characterized in that: The specific implementation method for identifying and labeling surface defect information of steel box girder structure in step S4 is as follows: Based on the texture image data collected by the depth camera in step S1, the trained deep learning model is used to automatically identify cracks, corrosion, and coating peeling defects, and the identification results and their spatial coordinates are synchronously labeled on the three-dimensional semantic map in step S3.

8. The method for inspecting steel box girders based on SLAM technology according to claim 1, characterized in that: The mobile robot platform adopts a magnetic adsorption mobile chassis, which is used for the adsorption and movement of the mobile robot platform on the vertical wall or top surface of the steel box girder.

Images