A Visual Recognition Method for Rebar Binding Points in Robotic Arms Based on Multi-Algorithm Fusion

Abstract

This invention belongs to the field of robotic arm visual recognition technology, specifically relating to a visual recognition method for rebar tying points based on multi-algorithm fusion. The method includes the following steps: S1: Acquiring image data and point cloud data; S2: Data processing; S3: Passing the image data and image features of the rebar to a target detection algorithm, which marks the bounding box for locating the rebar in the image data; S4: Extracting corner points within the bounding box of the rebar using a corner detection algorithm; these corner points are the rebar intersection points; S5: Based on the positional characteristics of the rebar in the three-dimensional space within the rebar construction layer, selecting the rebar intersection points located within the tying construction layer. These rebar intersection points are the points where the robotic arm needs to perform tying. This invention uses a main program to connect multiple algorithms to solve problems related to accuracy, real-time performance, and adaptability to complex environments during the rebar tying process, further improving the efficient and accurate recognition and path optimization of rebar tying points.

Description

Technical Field

[0001] This invention belongs to the field of robotic arm visual recognition technology, specifically relating to a visual recognition method for rebar binding points of robotic arms based on multi-algorithm fusion. Background Technology

[0002] With the continuous improvement of automation in the construction industry, rebar tying robots are gradually being applied to construction sites. Chinese patent application number CN202411604569.9 provides a rebar tying robot that uses a robotic arm to control a rebar tying machine at the end of the robotic arm to tie rebars, saving manpower and improving work efficiency.

[0003] Existing visual recognition systems rely on traditional computer vision and deep learning algorithms, and there is still room for improvement in recognition accuracy and stability in complex scenarios.

[0004] Taking YOLOv5, the most commonly used computer vision algorithm, as an example, although YOLOv5 can provide high recognition efficiency, its performance is unstable when dealing with multi-layered steel bars and occlusion problems. Especially in complex backgrounds, it is prone to recognition failure or misrecognition. Existing deep learning methods usually rely on static image data and lack adaptability to dynamic environments, making it unable to cope with constantly changing construction scenarios in real-time construction.

[0005] Therefore, a new algorithm is needed to solve problems such as accuracy, real-time performance, and adaptability to complex environments in the process of identifying rebar tying points. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a visual recognition method for rebar tying points using a robotic arm based on multi-algorithm fusion. This method uses a main program to connect multiple advanced algorithms to solve problems related to accuracy, real-time performance, and adaptability to complex environments during the rebar tying process. The main program relies on deep learning, multi-task learning, data fusion, and reinforcement learning to further improve the efficient and accurate recognition and path optimization of rebar tying points, enabling each step in the rebar recognition process to achieve high coordination and optimization.

[0007] The specific technical solution adopted in this invention is as follows:

[0008] A visual recognition method for rebar tying points in robotic arms based on multi-algorithm fusion includes the following steps:

[0009] S1: Data Acquisition: Image data and point cloud data of the direction of the binding construction layer where the rebar binding point is located are acquired through sensors;

[0010] S2: Data Processing: Extracting image features of reinforcing bars in the tying construction layer from image data; extracting three-dimensional spatial location features of reinforcing bars in the tying construction layer from point cloud data;

[0011] S3: Pass the image data and image features of the reinforcing bars to the target detection algorithm, and use the target detection algorithm to mark the bounding box for locating the reinforcing bars in the image data;

[0012] S4: Extract corner points within the bounding box of the reinforcing bars using a corner detection algorithm. These corner points are the intersection points of the reinforcing bars.

[0013] S5: Based on the positional characteristics of the reinforcing bars in the three-dimensional space of the reinforcing bar construction layer, the intersection points of the reinforcing bars located within the binding construction layer are selected. These intersection points are the points where the robotic arm needs to perform binding.

[0014] The original image data in step S2 is preprocessed, and image features are extracted from the preprocessed normalized image data. The preprocessing formula is as follows:

[0015]

[0016] Among them, I raw The original image data is given, where μ is the image mean, σ is the image standard deviation, and I is the image standard deviation. norm This is the normalized image data.

[0017] In step S3, the longitudinal width of the reinforcing bar image selected by the bounding box is measured as its diameter; wherein the diameter of the transverse reinforcing bar in the image is φ1, and the diameter of the longitudinal reinforcing bar is φ2; the coordinate transformation of the three-dimensional spatial position features is performed using the upper surface of the binding construction layer as the xoy plane, and the Z-axis range of the binding construction layer is [0, -(φ1, -(φ2 ... 1+ If the positional feature coordinates of the horizontal and vertical reinforcing bars in the image data fall within the range of the Z-axis, then the reinforcing bar is located in the binding construction layer, and the intersection point of the reinforcing bars formed by the two is the point where the robotic arm needs to bind it.

[0018] The corner detection algorithm mentioned is the Shi-Tomasi corner detection algorithm, and its calculation formula is as follows:

[0019] det(M) = λ1λ2

[0020]

[0021] M is the gradient matrix, λ1 and λ2 are the eigenvalues ​​of the matrix, and I x and I y These are the gradients of the image data in its x and y directions, respectively.

[0022] The point cloud data described in step S2 is voxelized using the VoxelNet algorithm to extract location features.

[0023] The target detection algorithm mentioned in step S3 is the YOLOv8 target detection algorithm.

[0024] Step S1: The image data of the rebar tying points acquired by the sensor are dynamically optimized using Kalman filtering.

[0025] The formula for calculating Kalman filtering is:

[0026] P k =(I k -K k H k )P k-1

[0027]

[0028] Where x is the estimate of the current state, and K k For Kalman gain, y k H is a measured value. k Let P be the observation matrix. k Let be the covariance matrix of the states.

[0029] The beneficial effects of this invention are:

[0030] 1. This invention addresses the issues of accuracy, real-time performance, and adaptability to complex environments in the rebar tying process by integrating multiple algorithms. Relying on deep learning, multi-task processing, and multi-algorithm fusion, it further improves the efficient and accurate identification and path optimization of rebar tying points, enabling each step in the rebar identification process to achieve high coordination and optimization.

[0031] Throughout the rebar tying process, the various algorithms ensure the continuity of the data flow and the synergy between them. The output of each step serves as the input for the next step, ultimately achieving accurate identification and path optimization of the rebar tying points. Through this fusion and optimization of multiple algorithms, this invention enables efficient, safe, and precise rebar tying in complex construction environments.

[0032] 2. This invention improves the accuracy and stability of rebar tying point identification. By combining deep learning with traditional image processing algorithms, the accuracy and stability of the rebar identification system are improved under conditions of changing lighting, overlapping rebars, and occlusion. Simultaneously, through a multi-task learning model, rebar detection and tying point identification are performed concurrently, utilizing shared feature information to enhance the robustness and accuracy of the overall identification system.

[0033] 3. When construction sites frequently encounter intersecting, overlapping, and multi-layered rebar structures, traditional algorithms struggle to accurately distinguish the spatial relationships between the rebars, leading to inaccurate identification of tying points. Furthermore, existing technologies are prone to missing tying points when dealing with complex occlusions or overlapping rebars, reducing recognition efficiency and accuracy. This invention combines point cloud data and image data. The 3D information provided by point cloud data solves the problem of rebar occlusion, while the image data ensures accurate identification of target tying points even in complex overlapping situations. Additionally, this invention introduces multimodal data fusion technology, combining data from multiple sensors such as LiDAR and depth cameras to improve the system's recognition capabilities in occlusion, intersections, and multi-layered rebar layouts.

[0034] 4. This invention employs the YOLOv8 algorithm, which significantly optimizes accuracy and speed, better addressing target detection challenges in complex construction environments. It improves target recognition speed while maintaining accuracy, adapting to the real-time feedback requirements of dynamic construction environments. This invention also introduces multi-threaded processing, simultaneously processing point cloud data and image data, reducing algorithm time and improving the response speed of rebar tying point recognition. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating a visual recognition method for rebar tying points in a robotic arm based on multi-algorithm fusion. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0037] Existing visual recognition systems largely rely on traditional computer vision and deep learning algorithms. However, their recognition accuracy and stability remain insufficient in complex scenarios such as varying lighting conditions, overlapping rebars, and double-layer rebar occlusion. Therefore, a new algorithm is needed to address the issues of accuracy, real-time performance, and adaptability to complex environments during rebar tying. This invention provides a visual recognition method for rebar tying points using a robotic arm based on multi-algorithm fusion, comprising the following steps:

[0038] S1: Data Acquisition: Image data and point cloud data of the direction of the binding construction layer where the rebar binding point is located are acquired through sensors;

[0039] S2: Data Processing: Extracting image features of reinforcing bars in the tying construction layer from image data; extracting three-dimensional spatial location features of reinforcing bars in the tying construction layer from point cloud data;

[0040] S3: Pass the image data and image features of the reinforcing bars to the target detection algorithm, and use the target detection algorithm to mark the bounding box for locating the reinforcing bars in the image data;

[0041] S4: Extract corner points within the bounding box of the reinforcing bars using a corner detection algorithm. These corner points are the intersection points of the reinforcing bars.

[0042] S5: Based on the positional characteristics of the reinforcing bars in the three-dimensional space of the reinforcing bar construction layer, the intersection points of the reinforcing bars located within the binding construction layer are selected. These intersection points are the points where the robotic arm needs to perform binding.

[0043] This invention acquires image data using a depth-of-field camera and obtains point cloud data of the rebar assembly using a LiDAR. By projecting the point cloud data onto the image, it facilitates the filtering of rebar intersections. Furthermore, by extracting corner points, the positional information of the four corner points when the rebars intersect horizontally and vertically is obtained. Based on the lines connecting the four corner points, the overlapping area of ​​the rebars can be basically determined, thus obtaining the point cloud group selected by the overlapping area in the point cloud data. This provides a high-quality data source for accurately obtaining the rebar binding points.

[0044] Furthermore, the original image data from step S2 is preprocessed, and image features are extracted from the preprocessed normalized image data. The preprocessing formula is as follows:

[0045]

[0046] Among them, I raw The original image data is given, where μ is the image mean, σ is the image standard deviation, and I is the image standard deviation. norm This is the normalized image data.

[0047] Furthermore, in step S3, the longitudinal width of the reinforcing bar image selected by the bounding box is measured as its diameter; wherein the diameter of the transverse reinforcing bar in the image is φ1 and the diameter of the longitudinal reinforcing bar is φ2; the coordinate transformation of the position features in three-dimensional space is performed with the upper surface of the binding construction layer as the xoy plane, and the Z-axis range of the binding construction layer is [0,-(φ1+φ2)]. If the position feature coordinates of the transverse and longitudinal reinforcing bars in the image data fall within the range of this Z-axis, then the reinforcing bar is located in the binding construction layer, and the intersection point of the reinforcing bars formed by the two is the point where the robotic arm needs to bind.

[0048] Specifically, the sensor of this invention is tilted and set at the front end of the base. Since the steel bar is approximately cylindrical, the obtained diameter is basically consistent with reality or slightly wider due to the blank space of the outline. The highest value of the steel bar close to the xoy plane is selected from the point cloud data, or a point is taken on the center line of the boundary frame along the direction of the steel bar, thereby ensuring the reliability of the above-mentioned Z-axis range. In addition, according to the requirements of steel bar binding, even if there are multiple layers of steel bars, the layer spacing between adjacent steel bar layers should be at least the diameter of the main bar. Therefore, by setting this Z-axis range, the steel bar data of the layer to be bound can be filtered out, avoiding interference caused by multiple overlapping layers.

[0049] The present invention uses a robotic arm supported by a base such as that described in Chinese patent application CN202411604569.9. With the center of the bottom surface of the base as the origin, the X-axis and Y-axis are parallel to the upper surface of the steel bar to which it is attached during construction. Since the base is magnetically attracted and supported by the support plate for alternating forward movement, it can ensure that the X-axis and Y-axis are always parallel and coincident on the upper surface of the steel bar.

[0050] By filtering the acquired point cloud data within the bound construction layer, situations such as complex overlap of multiple layers of reinforcing bars can be overcome, thereby improving the robustness and accuracy of the overall identification system.

[0051] The corner detection algorithm mentioned is the Shi-Tomasi corner detection algorithm, and its calculation formula is as follows:

[0052] det(M) = λ1λ2

[0053]

[0054] M is the gradient matrix, λ1 and λ2 are the eigenvalues ​​of the matrix, and I x and I y These are the gradients of the image data in the x and y directions, respectively. To calculate the gradients of the image data in the x and y directions, a Sobel operator is used to convolve the image, yielding the gradient images in the x and y directions, respectively, resulting in I. x and I y Then, by substituting into formula M, the gradient matrix is ​​obtained. By calculating the eigenvalues ​​λ1 and λ2 of the gradient matrix M, and comparing them with det(M) and a preset threshold, it is determined whether the pixel is a corner point, thus achieving complete corner point recognition.

[0055] The point cloud data described in step S2 is voxelized using the VoxelNet algorithm to extract positional features. By processing the point cloud data with VoxelNet, the positional features of the reinforcing bars in three-dimensional space are extracted. Through this algorithm, the system can effectively fuse image and depth data, improving the accuracy of reinforcing bar recognition, especially in identifying positional features of reinforcing bars at intersections, occlusions, or in complex backgrounds.

[0056] The target detection algorithm described in step S3 is the YOLOv8 target detection algorithm. By using the YOLOv8 target detection algorithm in conjunction with preprocessing such as noise reduction, illumination correction, and contrast enhancement, the recognition capability of this invention under complex environments with varying lighting conditions can be improved.

[0057] Step S1: The image data of the rebar tying points acquired by the sensor is dynamically optimized using Kalman filtering to improve adaptability to dynamic environments.

[0058] The formula for calculating Kalman filtering is:

[0059] Pk =(I k -K k H k )P k-1

[0060]

[0061] Where x is the estimate of the current state, and K k For Kalman gain, y k H is a measured value. k Let P be the observation matrix. k Let be the covariance matrix of the states.

[0062] The YOLOv8 algorithm employs a deep learning framework that utilizes convolutional layers to extract key features such as the edges, textures, and shapes of reinforcing bars from images. Furthermore, a mature reinforcement learning algorithm can be introduced into the post-processing of the YOLOv8 algorithm as a RL agent, dynamically adjusting parameters based on the detection results.

[0063] After obtaining the point cloud data of the rebar tying points, a Deep Q-Network (DQN) is used for path optimization. DQN progressively optimizes the robotic arm's path through interaction with the environment, helping to reduce wasted time and energy. To ensure physical consistency during the rebar tying process, existing physical constraints can be introduced, including energy conservation, load limitations, and operational area safety requirements. Through these physical constraints, the system ensures that each rebar tying point not only meets the visual recognition accuracy requirements but also complies with the physical limitations of the construction process, guaranteeing construction safety.

[0064] To improve the real-time performance and accuracy of the system, this invention can further introduce a multi-task learning framework, namely a fusion framework, to simultaneously conduct joint training for rebar target detection and tying point positioning. Through joint training, the overall efficiency of rebar detection and tying point recognition can be effectively improved, and the high-precision performance of the system in complex construction environments can be ensured.

Claims

1. A visual recognition method for rebar binding points of a robotic arm based on multi-algorithm fusion, characterized in that: Includes the following steps: S1: Data Acquisition: Image data and point cloud data of the direction of the binding construction layer where the rebar binding point is located are acquired through sensors; S2: Data Processing: Extracting image features of reinforcing bars in the tying construction layer from image data; extracting three-dimensional spatial location features of reinforcing bars in the tying construction layer from point cloud data; S3: Pass the image data and image features of the reinforcing bars to the target detection algorithm, and use the target detection algorithm to mark the bounding box for locating the reinforcing bars in the image data; S4: Extract corner points within the bounding box of the reinforcing bars using a corner detection algorithm. These corner points are the intersection points of the reinforcing bars. S5: Based on the positional characteristics of the reinforcing bars in the three-dimensional space of the reinforcing bar construction layer, the intersection points of the reinforcing bars located within the binding construction layer are selected. These intersection points are the points where the robotic arm needs to perform binding. The original image data in step S2 is preprocessed, and image features are extracted from the preprocessed normalized image data. The preprocessing formula is as follows: Among them, I raw The original image data is given, where μ is the image mean, σ is the image standard deviation, and I is the image standard deviation. norm The image data is normalized. In step S3, the longitudinal width of the steel bar image is measured as its diameter based on the steel bar image selected by the bounding box. In the image, the diameter of the horizontal reinforcing bar is φ1, and the diameter of the vertical reinforcing bar is φ2. Taking the upper surface of the binding construction layer as the xoy plane, a coordinate transformation is performed on the three-dimensional spatial position features. Therefore, the Z-axis range of the binding construction layer is [0, -(φ...). 1+ If the positional feature coordinates of the horizontal and vertical reinforcing bars in the image data fall within the range of the Z-axis, then the reinforcing bar is located in the binding construction layer, and the intersection point of the reinforcing bars formed by the two is the point where the robotic arm needs to bind it. The corner detection algorithm mentioned is the Shi-Tomasi corner detection algorithm, and its calculation formula is as follows: M is the gradient matrix, λ1 and λ2 are the eigenvalues ​​of the matrix, and I x and I y These are the gradients of the image data in its x and y directions, respectively.

2. The visual recognition method for rebar binding points of a robotic arm based on multi-algorithm fusion according to claim 1, characterized in that, The point cloud data described in step S2 is voxelized using the VoxelNet algorithm to extract location features.

3. The visual recognition method for rebar binding points of a robotic arm based on multi-algorithm fusion according to claim 1, characterized in that, The target detection algorithm mentioned in step S3 is the YOLOv8 target detection algorithm.

4. The visual recognition method for rebar binding points of a robotic arm based on multi-algorithm fusion according to claim 1, characterized in that, Step S1: The image data of the rebar tying points acquired by the sensor are dynamically optimized using Kalman filtering.

Images