Bridge wall-climbing robot positioning method based on image recognition

Abstract

A bridge wall-climbing robot positioning method based on image recognition comprises the specific steps that firstly, bridge surface image feature points are extracted, and feature point displacement between adjacent images is calculated; 2, collecting video data, dynamically adjusting a video collection frame rate according to the movement speed of the robot, and storing track data as continuous path points to form a preliminary path model; 3, taking the bridge structure as a reference, establishing a global coordinate system through an image matching algorithm, and positioning the position of the robot; 4, performing path planning based on the coordinate system and the bridge structure model; 5, based on image recognition data and a deep learning algorithm, combining a bridge structure and an environment model, and enabling the robot to plan and adjust a path in real time in a dynamic environment; 6, combining the image recognition data, the video trajectory data and the sensor data, and constructing a high-precision three-dimensional model of the bridge surface; and 7, dynamic obstacle avoidance optimization based on deep reinforcement learning is carried out, and adaptability in a dynamic environment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge detection, and in particular to a positioning method for a bridge wall-climbing robot based on image recognition. Background Art

[0002] Bridge inspection and maintenance are crucial tasks for ensuring bridge safety. Currently, common manual inspection methods are inefficient and dangerous, and robotics is emerging as a key development direction in the field. Existing bridge-climbing robots often rely on magnetic or vacuum devices for attachment and incorporate traditional sensors (such as IMUs, GPS, and lidar) for path planning. However, these existing technologies often face several challenges:

[0003] Insufficient positioning accuracy: Traditional methods often suffer from insufficient positioning accuracy in complex or dynamic environments (such as curved surfaces, bends, obstacles, etc. of bridges). Especially in the absence of clear landmarks or when signal interference is strong, positioning errors may lead to mission failure.

[0004] Poor adaptability: Existing positioning systems often have poor adaptability to environmental changes, especially in complex bridge surfaces and changing terrain. Traditional positioning methods struggle to effectively cope with this. For example, bridge surfaces may have numerous cracks, slopes, or complex geometries, all of which can affect sensor performance and reduce positioning stability.

[0005] Reliance on external sensors: Current positioning methods typically rely on external sensor data (such as IMU, lidar, etc.) to supplement image information, which increases the complexity and cost of the system. In addition, in some special environments, the sensor may experience data loss or interference, further affecting positioning accuracy.

[0006] Therefore, the positioning methods in the existing technology, especially in complex bridge environments, have great limitations, and there is an urgent need to develop a more accurate, adaptable and external sensor-independent positioning method. Summary of the Invention

[0007] In response to the shortcomings of the existing technology, the present invention proposes a positioning method for a bridge wall-climbing robot based on image recognition and video processing technology. By deeply integrating image recognition algorithms and video data, combined with dynamic path planning and real-time error correction, the positioning method completely relies on image data to solve the problems of insufficient positioning accuracy and poor adaptability in the existing technology. In particular, in complex bridge environments such as bends, obstacles, and dynamic changes, the positioning method can provide higher positioning accuracy and adaptability. The specific technical solution is as follows:

[0008] A positioning method for a bridge wall-climbing robot based on image recognition, characterized by the following specific steps:

[0009] Step 1: Collect bridge surface images, extract bridge surface image feature points, and calculate the displacement of feature points between adjacent images;

[0010] Step 2: Collect video data, dynamically adjust the video acquisition frame rate according to the robot's movement speed, store the trajectory data as continuous path points, and form a preliminary path model;

[0011] Step 3: Using the bridge structure as a reference, establish a global coordinate system using an image matching algorithm and locate the robot.

[0012] Step 4: Path planning based on the coordinate system and bridge structure model;

[0013] Step 5: Based on image recognition data and deep learning algorithms, combined with bridge structure and environmental models, the robot can plan and adjust its path in real time in a dynamic environment;

[0014] Step 6: Combine image recognition data, video trajectory data, and sensor data to build a high-precision 3D model of the bridge surface;

[0015] Step 7: Dynamic obstacle avoidance optimization based on deep reinforcement learning to improve adaptability in dynamic environments.

[0016] As an optimization: In step 1, the specific method for extracting the characteristic points of the bridge surface image is:

[0017] W i =α·C i +β·1 / d i +γ·K i

[0018] Among them, W i : weighted matching value of feature point i;

[0019] C i : Confidence score, calculated by the stability of feature points;

[0020] d i : Euclidean distance to neighboring points;

[0021] K i : The importance weight of the feature point is dynamically adjusted according to the importance of the image area;

[0022] α, β, γ: weight coefficients, optimized through experiments and support dynamic adjustment.

[0023] As an optimization: the step three is specifically as follows:

[0024] Combining image recognition data with sensor data, the Kalman filter algorithm is used for real-time error correction. The relative positioning information provided by image recognition is combined with sensor data to dynamically adjust the noise covariance and improve positioning accuracy in dynamic environments.

[0025] State update equation: X t =A·X t-1 +B·U t +w t

[0026] Observation equation: Z t =H·X t +v t

[0027] Among them, X t : System state vector, including robot position, velocity and posture;

[0028] A: state transfer matrix;

[0029] B: control input matrix;

[0030] U t : control input vector;

[0031] w t ,v t : process noise and measurement noise;

[0032] H: Observation matrix.

[0033] As an optimization: In step 5, to achieve precise positioning of the robot in the complex environment of the bridge structure, a deep learning algorithm combined with a dynamic time warping (DTW) algorithm was used. Through image recognition and video processing technology, the robot's current actual motion trajectory Q was obtained and a reference trajectory C was extracted from the bridge environment model. The two are out of sync in the time dimension due to differences in sampling frequency or path.

[0034] The DTW algorithm is used to perform nonlinear alignment on the two trajectories to obtain the optimal matching path and improve positioning accuracy. The calculation formula of the DTW distance is as follows:

[0035]

[0036] Where: Q: actual motion trajectory of the robot;

[0037] C: Bridge environment model trajectory;

[0038] φ(i): mapping relationship at time point i;

[0039] qi: represents the position or eigenvalue corresponding to the i-th time point in the actual motion trajectory Q of the robot;

[0040] cφ(i): represents the optimal corresponding point in the reference trajectory C obtained by DTW matching with qi.

[0041] As an optimization: In step 6, a sparse matrix-based 3D reconstruction optimization algorithm is used to optimize point cloud computing and reduce computational complexity:

[0042] Optimization goal:

[0043] Where: A: feature point matching matrix;

[0044] X: target point cloud coordinate;

[0045] b: observation value vector;

[0046] λ: Regularization parameter used to control sparsity.

[0047] As an optimization: In step 7, the robot obstacle avoidance model is constructed through reinforcement learning (DRL), and the reward function is defined as: R t =-α·D t -β·C t +γ·S t

[0048] D t : Minimum distance to obstacles;

[0049] C t : Collision risk, calculated based on depth image;

[0050] S t : target completion degree of path planning;

[0051] α, β, γ: weight parameters, automatically optimized through meta-learning or evolutionary algorithms;

[0052] The DRL model learns the optimal obstacle avoidance strategy during training to improve its adaptability in dynamic environments.

[0053] The beneficial effects of the present invention are: 1. High-precision positioning is achieved through image feature point matching and video analysis, which can be independent of other external sensors and effectively complete positioning tasks in complex and dynamic bridge environments, greatly improving the system's adaptability and positioning accuracy, and solving the problems of strong sensor dependence and poor adaptability in traditional technologies.

[0054] 2. By combining deep learning (CNN) with image data, dynamic path planning and real-time adjustment are performed. In particular, it can identify obstacles in the complex structure of the bridge surface and automatically adjust the path, improving the robot's adaptability.

[0055] 3. By fusing image recognition data, video trajectory data, and sensor data, a high-precision 3D model of the bridge surface is constructed. Unlike existing technologies, this method utilizes multi-source fusion of image data and deep learning models, optimizing positioning accuracy using image data and performing dynamic error correction based on real-time sensor data. This method effectively improves positioning accuracy in dynamic environments and avoids the limitations of sensor reliance. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the process of the present invention.

[0057] Figure 2 A schematic diagram of establishing the coordinate system in the present invention. DETAILED DESCRIPTION

[0058] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more precise definition of the protection scope of the present invention.

[0059] like Figure 1 and Figure 2 Shown: A positioning method for a bridge wall-climbing robot based on image recognition, the specific steps are:

[0060] S1. Image feature point extraction and matching:

[0061] The robot collects images of the bridge surface through the front camera and uses image recognition technology. In this embodiment, the SIFT, ORB, and SuperPoint methods are used to extract key feature points on the bridge surface. The robot also uses a feature point matching algorithm. In this embodiment, the optical flow method is used to calculate the displacement of feature points between adjacent images. The present invention does not rely on external sensors and achieves positioning entirely through image recognition technology.

[0062] W i =α·C i +β·1 / d i +γ·K i

[0063] Among them, W i : weighted matching value of feature point i;

[0064] C i : Confidence score, calculated by feature point stability, such as matching consistency;

[0065] d i : Euclidean distance to neighboring points;

[0066] K i : The importance weight of the feature point is dynamically adjusted according to the importance of the image area;

[0067] α, β, γ: weight coefficients, which are optimized through experiments and support dynamic adjustment, such as adaptive changes based on environmental complexity.

[0068] Dynamically adjusting matching priorities based on feature point confidence, distance, and image region importance weights (Ki) ensures higher matching accuracy in complex environments. This method dynamically optimizes matching priorities based on the environment, addressing the impact of environmental complexity on positioning accuracy.

[0069] S2, video recognition and motion trajectory storage:

[0070] By combining continuous video frames with image recognition technology, the robot can analyze the displacement of feature points in real time and accurately calculate its motion trajectory, ensuring dynamic positioning accuracy in complex environments. Video data complements image recognition's shortcomings in certain rapidly changing or difficult-to-identify situations, improving the continuity and stability of its motion trajectory, including forward, backward, and steering angles.

[0071] Added frame rate adaptive control module: Dynamically adjust the video acquisition frame rate according to the robot's movement speed to ensure the accuracy of displacement calculation.

[0072] The trajectory data is stored as continuous path points, and a sliding window or compression algorithm is used to reduce data redundancy to form a preliminary path model.

[0073] S3. Coordinate system establishment and positioning:

[0074] Taking the bridge structure as a reference, a global coordinate system is established through image matching algorithm to define the origin and main axis directions, such as the length direction and vertical direction of the bridge.

[0075] For complex bridge structures (such as curved surfaces), a conversion strategy between the local coordinate system and the global coordinate system is added to achieve dynamic alignment through matrix transformation.

[0076] Combining sensor data such as the inertial measurement unit (IMU) and wheel speed meter, the Kalman filter algorithm is used to correct the relative positioning error of image recognition:

[0077] Combining image recognition data with sensor data, a Kalman filter algorithm is used for real-time error correction. The relative positioning information provided by image recognition is combined with sensor data to dynamically adjust noise covariance, effectively improving positioning accuracy in dynamic environments.

[0078] State update equation: X t =A·X t-1 +B·U t +w t

[0079] Observation equation: Z t =H·X t +v t

[0080] Among them, X t : System state vector (robot position, velocity, posture, etc.);

[0081] A: State transition matrix

[0082] B: Control input matrix

[0083] U t : Control input vector (from sensor)

[0084] w t ,v t : Process noise and measurement noise

[0085] H: observation matrix

[0086] Real-time error correction is achieved to improve positioning accuracy. The innovation of this invention lies in the application of image recognition technology, which relies entirely on image data for positioning without relying on traditional sensor data. Especially in complex bridge environments (such as curved surfaces, obstacles, etc.), image recognition can provide higher positioning accuracy and adaptability.

[0087] S4. Path planning and real-time adjustment:

[0088] During the path planning and real-time adjustment phase, the robot uses image recognition technology combined with deep learning algorithms to make path decisions based on the established coordinate system and the bridge structure and environmental models. This invention preferably employs a convolutional neural network (CNN) model to process and analyze real-time image data, enabling the robot to plan and adjust its path within the complex environment of the bridge surface.

[0089] The convolutional neural network model uses images of the bridge structure as input and, combined with an established 3D model of the environment, automatically identifies local features such as obstacles, cracks, and inclined surfaces on the bridge surface, thereby performing path planning. During robot operation, when encountering dynamic factors (such as temporary occlusion, lighting changes, and unexpected obstacles), the model adjusts the path in real time based on the image recognition results, effectively improving the robot's adaptability to complex and unstructured environments.

[0090] To achieve precise alignment between the robot's actual motion trajectory and the bridge environment model trajectory, thereby optimizing navigation accuracy, the present invention uses the dynamic time warping (DTW) algorithm, which is defined as follows:

[0091]

[0092] Where: Q: actual motion trajectory of the robot;

[0093] C: Bridge environment model trajectory;

[0094] φ(i): mapping relationship at time point i;

[0095] qi: represents the position or eigenvalue corresponding to the i-th time point in the actual motion trajectory Q of the robot;

[0096] cφ(i): represents the optimal corresponding point in the reference trajectory C obtained by DTW matching with qi;

[0097] Clarify the training data source of the deep learning model: cover different bridge types (suspension bridges, arch bridges) and environments (rainy days, nighttime) to enhance generalization capabilities.

[0098] S5. Data fusion and feedback:

[0099] By fusing image recognition data, video trajectory data, and sensor data, a high-precision 3D model of the bridge surface was constructed. This multi-source fusion of image data and a deep learning model was employed to optimize positioning accuracy using image data and dynamically correct errors based on real-time sensor data. This method effectively improves positioning accuracy in dynamic environments and avoids the limitations of sensor reliance.

[0100] Use sparse matrix-based 3D reconstruction optimization algorithm to optimize point cloud computing and reduce computational complexity:

[0101] Optimization goal:

[0102] Where: A: feature point matching matrix

[0103] X: target point cloud coordinates

[0104] b: observation vector

[0105] λ: Regularization parameter used to control sparsity.

[0106] S6. Dynamic obstacle avoidance optimization based on deep reinforcement learning:

[0107] Build a robot obstacle avoidance model through reinforcement learning (DRL) and define the reward function: R t =-α·Dt -β·C t +γ·S t

[0108] D t : Minimum distance to obstacles;

[0109] C t : Collision risk (calculated based on depth image);

[0110] S t : target completion degree of path planning;

[0111] α, β, γ: weight parameters, automatically optimized through meta-learning or evolutionary algorithms.

[0112] The DRL model learns the optimal obstacle avoidance strategy during training to improve its adaptability in dynamic environments.

Claims

1. A positioning method for a bridge wall-climbing robot based on image recognition, characterized in that: The specific steps are: Step 1: Collect bridge surface images, extract bridge surface image feature points, and calculate the displacement of feature points between adjacent images; Step 2: Collect video data, dynamically adjust the video acquisition frame rate according to the robot's movement speed, store the trajectory data as continuous path points, and form a preliminary path model; Step 3: Using the bridge structure as a reference, establish a global coordinate system using an image matching algorithm and locate the robot. Step 4: Path planning based on the coordinate system and bridge structure model; Step 5: Based on image recognition data and deep learning algorithms, combined with bridge structure and environmental models, the robot can plan and adjust its path in real time in a dynamic environment; Step 6: Combine image recognition data, video trajectory data, and sensor data to build a high-precision 3D model of the bridge surface; Step 7: Dynamic obstacle avoidance optimization based on deep reinforcement learning to improve adaptability in dynamic environments.

2. The bridge wall-climbing robot positioning method based on image recognition according to claim 1 is characterized in that: In step 1, the specific method for extracting the characteristic points of the bridge surface image is: W i =α·C i +β·1 / d i +γ·K i Among them, W i : weighted matching value of feature point i; C i : Confidence score, calculated by the stability of feature points; d i : Euclidean distance to neighboring points; K i : The importance weight of the feature point is dynamically adjusted according to the importance of the image area; α, β, γ: weight coefficients, optimized through experiments and support dynamic adjustment.

3. The bridge wall-climbing robot positioning method based on image recognition according to claim 1 is characterized in that: The step three is specifically as follows: Combining image recognition data with sensor data, the Kalman filter algorithm is used for real-time error correction. The relative positioning information provided by image recognition is combined with sensor data to dynamically adjust the noise covariance and improve positioning accuracy in dynamic environments. State update equation: X t =A·X t-1 +B·U t +w t Observation equation: Z t =H·X t +v t Among them, X t : System state vector, including robot position, velocity and posture; A: state transfer matrix; B: control input matrix; U t : control input vector; w t ,v t : process noise and measurement noise; H: Observation matrix.

4. The method for positioning a bridge wall-climbing robot based on image recognition according to claim 1, characterized in that: In step 5, to achieve precise positioning of the robot within the complex environment of the bridge structure, a deep learning algorithm combined with a dynamic time warping (DTW) algorithm was used. Through image recognition and video processing, the robot's current actual motion trajectory Q was obtained, and a reference trajectory C was extracted from the bridge's environmental model. The two are out of sync in the time dimension due to differences in sampling frequency or paths. The DTW algorithm is used to perform nonlinear alignment on the two trajectories to obtain the optimal matching path and improve positioning accuracy. The calculation formula of the DTW distance is as follows: Where: Q: actual motion trajectory of the robot; C: Bridge environment model trajectory; φ(i): mapping relationship at time point i; qi: represents the position or eigenvalue corresponding to the i-th time point in the actual motion trajectory Q of the robot; cφ(i): represents the optimal corresponding point in the reference trajectory C obtained by DTW matching with qi.

5. The method for positioning a bridge wall-climbing robot based on image recognition according to claim 1, characterized in that: In step 6, a sparse matrix-based 3D reconstruction optimization algorithm is used to optimize point cloud computing and reduce computational complexity: Optimization goal: Where: A: feature point matching matrix; X: target point cloud coordinate; b: observation value vector; λ: Regularization parameter used to control sparsity.

6. The method for positioning a bridge wall-climbing robot based on image recognition according to claim 1, characterized in that: In step 7, a robot obstacle avoidance model is constructed through Deep Reinforcement Learning (DRL), and the reward function is defined as: t =-α·D t -β·C t +γ·S t D t : Minimum distance to obstacles; C t : Collision risk, calculated based on depth image; S t : target completion degree of path planning; α, β, γ: weight parameters, automatically optimized through meta-learning or evolutionary algorithms; The DRL model learns the optimal obstacle avoidance strategy during training to improve its adaptability in dynamic environments.

Images