Mechanical arm steel bar binding point visual identification method based on multi-algorithm fusion

Abstract

The invention belongs to the technical field of mechanical arm visual identification, and particularly relates to a mechanical arm steel bar binding point visual identification method based on multi-algorithm fusion, and the method comprises the following steps: S1, collecting image data and point cloud data; s2, data processing; S3, transmitting the image data and the image features of the reinforcing steel bars to a target detection algorithm, and calibrating a bounding box for positioning the reinforcing steel bars in the image data through the target detection algorithm; s4, angular points are extracted in the bounding box range of the reinforcing steel bars through an angular point detection algorithm, and the angular points are reinforcing steel bar intersection points; and S5, according to the position characteristics of the steel bars in the steel bar construction layer in the three-dimensional space, the steel bar intersection points located in the binding construction layer range are screened out, and the steel bar intersection points are the point positions needing to be bound by the mechanical arm. According to the method, the problems of precision, real-time performance, complex environment adaptability and the like in the reinforcing steel bar binding process are solved by connecting one main program with multiple algorithms in series, and efficient and accurate recognition and path optimization of reinforcing steel bar binding points are further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robotic arm visual recognition, and in particular relates to a robotic arm steel bar binding point visual recognition method based on multi-algorithm fusion. Background Art

[0002] With the continuous improvement of the automation level in the construction industry, rebar tying robots are gradually being used on construction sites. The Chinese patent application number CN202411604569.9 provides a rebar tying robot that uses a robotic arm to control the rebar tying machine at the end of the robotic arm to tie the rebar, saving labor and improving work efficiency.

[0003] Existing visual recognition systems rely on traditional computer vision and deep learning algorithms, and there is room for improvement in recognition accuracy and stability in complex scenarios.

[0004] Taking YOLOv5, the most commonly used computer vision algorithm, as an example, although YOLOv5 can provide high recognition efficiency, its performance is unstable when dealing with multi-layer steel bars and occlusion problems, especially in complex backgrounds, and is prone to recognition failure or misidentification. Existing deep learning methods usually rely on static image data and lack adaptability to dynamic environments. They cannot cope with ever-changing construction scenarios during real-time construction.

[0005] Therefore, a new algorithm is needed to solve the problems of accuracy, real-time performance and adaptability to complex environments in the process of steel bar binding point identification. Summary of the Invention

[0006] In order to solve the problems existing in the above-mentioned prior art, the present invention provides a visual recognition method for steel bar binding points of a robotic arm based on multi-algorithm fusion. A main program is used to connect multiple advanced algorithms to solve problems such as accuracy, real-time performance and adaptability to complex environments in the steel bar binding process. The main program relies on deep learning, multi-task learning, data fusion, and reinforcement learning to further improve the efficient and accurate recognition and path optimization of steel bar binding points, so that each step in the steel bar identification process can be highly coordinated and optimized.

[0007] The specific technical solution adopted in the present invention is:

[0008] The visual recognition method of steel bar binding points of a robotic arm based on multi-algorithm fusion includes the following steps:

[0009] S1: Data acquisition: The image data and point cloud data of the steel bar binding points in the binding construction layer direction are acquired through sensors;

[0010] S2: Data processing: Extract the image features of the steel bars in the binding construction layer from the image data; extract the three-dimensional spatial position features of the steel bars in the binding construction layer from the point cloud data;

[0011] S3: The image data and the image features of the steel bars are passed to the object detection algorithm, and the object detection algorithm is used to calibrate the bounding box used to locate the steel bars in the image data;

[0012] S4: Extract corner points within the bounding box of the steel bars using a corner detection algorithm. These corner points are the intersection points of the steel bars.

[0013] S5: Based on the position characteristics of the steel bars in the three-dimensional space in the steel bar construction layer, the steel bar intersection points within the binding construction layer are screened out. The steel bar intersection points are the points where the robotic arm needs to perform binding.

[0014] The original image data in step S2 is preprocessed, and image features are extracted from the normalized image data after preprocessing. The preprocessing formula is:

[0015]

[0016] Among them, I raw is the original image data, μ is the mean of the image, σ is the standard deviation of the image, I norm is the normalized image data.

[0017] In step S3, the longitudinal width of the steel bar image is measured as its diameter according to the steel bar image selected by the bounding box; the transverse steel bar diameter in the image is φ1, and the longitudinal steel bar diameter is φ2; the coordinate transformation of the position feature in the three-dimensional space is performed with the upper surface of the binding construction layer as the xoy plane, and the Z-axis range of the binding construction layer is [0,-(φ 1+ φ2)], if the position feature coordinates of the transverse and longitudinal steel bars in the image data fall within the Z-axis range, the steel bars are located in the binding construction layer, and the intersection of the two steel bars is the point where the robotic arm needs to be bound.

[0018] The corner detection algorithm is the Shi-Tomasi corner detection algorithm, and its calculation formula is:

[0019] det(M)=λ1λ2

[0020]

[0021] M is the gradient matrix, λ1 and λ2 are the eigenvalues of the matrix, I x and I y are the gradients of the image data in the x and y directions, respectively.

[0022] The point cloud data described in step S2 is voxelized using the VoxelNet algorithm to extract position features.

[0023] The target detection algorithm in step S3 is the YOLOv8 target detection algorithm.

[0024] In step S1, the sensor obtains image data of the steel bar binding point and performs dynamic optimization through Kalman filtering.

[0025] The calculation formula of Kalman filter is:

[0026] P k =(I k -K k H k )P k-1

[0027]

[0028] Among them, x is the estimate of the current state, K k is the Kalman gain, y k is the measured value, H k is the observation matrix, P k is the covariance matrix of the state.

[0029] The beneficial effects of the present invention are:

[0030] 1. The present invention solves the problems of accuracy, real-time performance and adaptability to complex environments in the rebar binding process by integrating multiple algorithms. Relying on deep learning, multi-task processing and multi-algorithm fusion, it further improves the efficient and accurate identification of rebar binding points and path optimization, so that each step in the rebar identification process can be highly coordinated and optimized.

[0031] Throughout the binding process, individual algorithms ensure data flow continuity and synergy between algorithms. The output of each step serves as input for the next, ultimately achieving precise identification of rebar binding points and path optimization. Through this multi-algorithm integration and optimization, the present invention enables efficient, safe, and precise rebar binding in complex construction environments.

[0032] 2. This invention improves the accuracy and stability of rebar binding point identification. By combining deep learning with traditional image processing algorithms, the accuracy and stability of the rebar recognition system are improved under varying lighting conditions, overlapping rebars, and occlusion. Furthermore, through a multi-task learning model, both rebar detection and binding point identification are performed simultaneously, leveraging shared feature information to enhance the robustness and accuracy of the overall recognition system.

[0033] 3. When steel bars often cross, overlap, and overlap in multiple layers at construction sites, traditional algorithms find it difficult to accurately distinguish the spatial relationship between the steel bars, resulting in inaccurate identification of binding points. At the same time, existing technologies are prone to miss binding points when dealing with complex occlusions or overlapping steel bars, reducing recognition efficiency and accuracy. The present invention combines point cloud data and image data, using the three-dimensional information provided by point cloud data to solve the problem of steel bar occlusion, and combines image data to ensure that target binding points can be accurately identified even in complex overlapping situations. In addition, the present invention also introduces multimodal data fusion technology, combining data from multiple sensors such as lidar and depth cameras to improve the system's recognition capabilities in occlusion, crossing, and multi-layer steel bar layouts.

[0034] 4. This invention utilizes the YOLOv8 algorithm, which has been significantly optimized in terms of accuracy and speed. This algorithm is better suited to target detection in complex construction environments, improving target recognition speed while maintaining accuracy and meeting the demands for real-time feedback in dynamic construction environments. This invention also incorporates multi-threaded processing, processing both point cloud and image data simultaneously, reducing algorithm time and improving the response speed of rebar tie point identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The figure is a flow chart of the visual recognition method of steel bar binding points of a robotic arm based on multi-algorithm fusion. DETAILED DESCRIPTION

[0036] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0037] Existing visual recognition systems mostly rely on traditional computer vision and deep learning algorithms. However, recognition accuracy and stability are still insufficient in complex scenarios such as changing lighting, overlapping steel bars, and double-layer steel bar occlusion. Therefore, it is necessary to provide a new algorithm to solve problems such as accuracy, real-time performance, and adaptability to complex environments during the steel bar binding process. The present invention provides a method for visual recognition of steel bar binding points using a robotic arm based on multi-algorithm fusion, which includes the following steps:

[0038] S1: Data acquisition: The image data and point cloud data of the steel bar binding points in the binding construction layer direction are acquired through sensors;

[0039] S2: Data processing: Extract the image features of the steel bars in the binding construction layer from the image data; extract the three-dimensional spatial position features of the steel bars in the binding construction layer from the point cloud data;

[0040] S3: The image data and the image features of the steel bars are passed to the object detection algorithm, and the object detection algorithm is used to calibrate the bounding box used to locate the steel bars in the image data;

[0041] S4: Extract corner points within the bounding box of the steel bars using a corner detection algorithm. These corner points are the intersection points of the steel bars.

[0042] S5: Based on the position characteristics of the steel bars in the three-dimensional space in the steel bar construction layer, the steel bar intersection points within the binding construction layer are screened out. The steel bar intersection points are the points where the robotic arm needs to perform binding.

[0043] The present invention obtains image data through a depth of field camera and obtains point cloud data of steel bar components through a laser radar. By projecting the point cloud data onto the image, it is convenient to screen the intersection points of the steel bars, and by extracting the corner points, the position information of the four corner points when the steel bars cross horizontally and vertically is obtained. The area range of the steel bars overlapping can be basically determined based on the connection lines passing through the four corner points, so that the point cloud group selected by the overlapping range is obtained in the point cloud data, thereby providing a high-quality data source for accurately obtaining the steel bar binding points.

[0044] Furthermore, the original image data in step S2 is preprocessed, and image features are extracted from the normalized image data after preprocessing. The preprocessing formula is:

[0045]

[0046] Among them, I raw is the original image data, μ is the mean of the image, σ is the standard deviation of the image, I norm is the normalized image data.

[0047] Furthermore, in step S3, the longitudinal width of the steel bar image is measured as its diameter based on the steel bar image selected by the bounding box; the diameter of the transverse steel bar in the image is φ1, and the diameter of the longitudinal steel bar is φ2; the coordinate transformation of the position feature in the three-dimensional space is performed with the upper surface of the binding construction layer as the xoy plane, and the Z-axis range of the binding construction layer is [0,-(φ1+φ2)]. If the position feature coordinates of the transverse steel bar and the longitudinal steel bar in the image data fall within the Z-axis range, the steel bar is located in the binding construction layer, and the intersection of the steel bars formed by the two is the point where the robotic arm needs to be tied.

[0048] Specifically, the sensor of the present invention is tiltedly arranged at the front end of the base. Since the steel bar is approximately a cylinder, the obtained diameter is basically consistent with the actual diameter or is slightly wider due to the blank space in the tracing frame. The highest value of the steel bar close to the xoy plane is preferably taken from the point cloud data, or a point is taken on the midline of the boundary box along the direction of the steel bar, thereby ensuring the reliability of the above-mentioned Z-axis range. According to the requirements of steel bar binding, even if there are multiple layers of steel bars, the interlayer spacing between adjacent steel bar layers is at least the main bar diameter. Therefore, by setting this Z-axis range, the steel bar data of the layer to be bound can be screened out to avoid interference caused by overlapping of multiple layers.

[0049] The present invention uses a robotic arm to carry the base with the help of a base such as Chinese patent application CN202411604569.9, and takes the center of the bottom surface of the base as the origin. The X-axis and Y-axis are parallel to the upper surface of the steel bar to which they are attached during construction. Since the base is adsorbed by magnetic attraction and is alternately supported and advanced by the support plate, it can ensure that the X-axis and Y-axis are always parallel and overlap on the upper surface of the steel bar.

[0050] The obtained point cloud data can be screened within the range of the binding construction layer to overcome the complex overlap of multiple layers of steel bars, thereby improving the robustness and accuracy of the overall recognition system.

[0051] The corner detection algorithm is the Shi-Tomasi corner detection algorithm, and its calculation formula is:

[0052] det(M)=λ1λ2

[0053]

[0054] M is the gradient matrix, λ1 and λ2 are the eigenvalues of the matrix, I x and I y The gradients of the image data in the x and y directions are calculated, and the Sobel operator is introduced to perform convolution operation with the image to obtain the gradient images in the x and y directions respectively, and obtain I x and I y Then substitute formula M to obtain the gradient matrix. By calculating the eigenvalues λ1 and λ2 of the gradient matrix M and comparing it with det(M) and the preset threshold, we can determine whether the pixel is a corner point, thereby completing the identification of the corner point.

[0055] The point cloud data described in step S2 is voxelized using the VoxelNet algorithm to extract positional features. VoxelNet processes the point cloud data to extract the positional features of the rebars in three-dimensional space. This algorithm effectively fuses image and depth data, improving the accuracy of rebar recognition, particularly when the rebars are crossed, obscured, or in complex backgrounds.

[0056] The target detection algorithm in step S3 is the YOLOv8 target detection algorithm. By using the YOLOv8 target detection algorithm in conjunction with denoising, illumination correction, and contrast enhancement preprocessing, the recognition capability of the present invention can be enhanced in complex environments with variable illumination.

[0057] In step S1, the image data of the steel bar binding point obtained by the sensor is dynamically optimized through Kalman filtering to improve the adaptability to the dynamic environment.

[0058] The calculation formula of Kalman filter is:

[0059] Pk =(I k -K k H k )P k-1

[0060]

[0061] Among them, x is the estimate of the current state, K k is the Kalman gain, y k is the measured value, H k is the observation matrix, P k is the covariance matrix of the state.

[0062] The YOLOv8 algorithm uses a deep learning framework that leverages convolutional layers to extract key features such as the edges, texture, and shape of rebar from images. Furthermore, a sophisticated reinforcement learning algorithm can be incorporated into the YOLOv8 algorithm's post-processing, acting as an RL agent to dynamically adjust parameters based on detection results.

[0063] After obtaining the point cloud data for the tying points, a Deep Q-Network (DQN) is used for path optimization. By interacting with the environment, the DQN gradually optimizes the robot's path, helping to reduce wasted time and energy. To ensure physical consistency during the rebar tying process, existing physical constraints are introduced, including energy conservation, load limits, and operational area safety requirements. Using these physical constraints, the system ensures that each rebar tying point meets not only the visual recognition accuracy requirements but also the physical limitations of the construction process, ensuring construction safety.

[0064] In order to improve the real-time performance and accuracy of the system, the present invention can further introduce a multi-task learning framework, namely a fusion framework, to simultaneously perform joint training of steel bar target detection and binding point positioning. Through joint training, the overall efficiency of steel bar detection and binding point identification can be effectively improved, and the high-precision performance of the system in complex construction environments can be ensured.

Claims

1. A multi-algorithm fusion-based visual recognition method for steel bar binding points using a robotic arm is characterized by: The following steps are involved: S1: Data acquisition: The image data and point cloud data of the steel bar binding points in the binding construction layer direction are acquired through sensors; S2: Data processing: Extract the image features of the steel bars in the binding construction layer from the image data; extract the three-dimensional spatial position features of the steel bars in the binding construction layer from the point cloud data; S3: The image data and the image features of the steel bars are passed to the object detection algorithm, and the object detection algorithm is used to calibrate the bounding box used to locate the steel bars in the image data; S4: Extract corner points within the bounding box of the steel bars using a corner detection algorithm. These corner points are the intersection points of the steel bars. S5: Based on the position characteristics of the steel bars in the three-dimensional space in the steel bar construction layer, the steel bar intersection points within the binding construction layer are screened out. The steel bar intersection points are the points where the robotic arm needs to perform binding.

2. The method for visual recognition of steel bar binding points by a robotic arm based on multi-algorithm fusion according to claim 1 is characterized in that: The original image data in step S2 is preprocessed, and image features are extracted from the normalized image data after preprocessing. The preprocessing formula is: Among them, I raw is the original image data, μ is the mean of the image, σ is the standard deviation of the image, I norm is the normalized image data.

3. The method for visual recognition of steel bar binding points by a robotic arm based on multi-algorithm fusion according to claim 1 is characterized in that: In step S3, the longitudinal width of the steel bar image is measured as its diameter according to the steel bar image selected by the bounding box; The diameter of the transverse reinforcement in the image is φ1, and the diameter of the longitudinal reinforcement is φ2. The coordinate transformation of the position feature in the three-dimensional space is performed with the upper surface of the binding construction layer as the xoy plane. The Z-axis range of the binding construction layer is [0,-(φ 1+ φ2)], if the position feature coordinates of the transverse and longitudinal steel bars in the image data fall within the Z-axis range, the steel bars are located in the binding construction layer, and the intersection of the two steel bars is the point where the robotic arm needs to be bound.

4. The method for visual recognition of steel bar binding points by a robotic arm based on multi-algorithm fusion according to claim 1 is characterized in that: The corner detection algorithm is the Shi-Tomasi corner detection algorithm, and its calculation formula is: det(M)=λ1λ2 M is the gradient matrix, λ1 and λ2 are the eigenvalues of the matrix, I x and I y are the gradients of the image data in the x and y directions, respectively.

5. The method for visual recognition of steel bar binding points by a robotic arm based on multi-algorithm fusion according to claim 1 is characterized in that: The point cloud data described in step S2 is voxelized using the VoxelNet algorithm to extract position features.

6. The method for visual recognition of steel bar binding points by a robotic arm based on multi-algorithm fusion according to claim 1 is characterized in that: The target detection algorithm in step S3 is the YOLOv8 target detection algorithm.

7. The method for visual recognition of steel bar binding points by a robotic arm based on multi-algorithm fusion according to claim 1 is characterized in that: In step S1, the sensor obtains image data of the steel bar binding point and performs dynamic optimization through Kalman filtering.

Images